Assuming i used a 26.2144MHZ MCLK instead of 24.5760MHZ, is there any problems with running the ADAU1761 at 51.2 kHz instead of 48kHz? The filter coeffs would be designed at 51.2 kHz.
Yes this is permissible. The data sheet allows for a MCLK period as short as 37 nS in 512 x fs mode. This corresponds to MCLK as high as 27 MHz, or a sample rate of 52.7 KHz. Thus, 51200 is fine.
You can have SigmaStudio compute filter coefficients for your new sample rate by right-clicking your input cell and entering the rate:
Retrieving data ...