The formulas shown on page 94 of the MAX98089 datasheet for K2 are incorrect. The GUI for the MAX98089EVKIT is correct and the results were characterized during introduction and proven to work.
The only solution we can provide is to use the GUI to determine the coefficients. If the use case has a set number of filter configurations, then use the EVK to calculate the register values and create profiles to switch between them. If you have a large or unknowable number of possible filters configurations, then the MAX98089EVKIT will not work for your use case.