I use ad73311 as analog-to-digital converter. the sampling data coupled severe noise. According to the sampling digital data, the max value of quantization noise is about 210 when the analog input pins connect to ground.
however, I don't know the noise source. So I want to know the how to determine the source of the noise and the method to reduce the noise.
Noise can be coming from many sources so it is difficult to determine from just a little bit of information.
However, I can say that the way you are measuring the noise is not the best way. The inputs to the PGA are biased up to the reference level of the part. So by grounding the pins you are sinking current into ground and introducing noise. To make this measurement you need to reference the pins to the REFOUT pin.
Then the reference pin, REFOUT needs to have ample decoupling. The datasheet shows only 0.1uf which is really too small. I would place a 0.1uf and a larger cap like 10uf on the ref pin. I would say 1uf min and 47 uf max. This will also help the noise floor.
Then other sources of noise will be your power supplies. They need to be well filtered and if you are using switching supplies make sure they are well filtered and have a very high switching frequency.
Grounding is very important. I would use a ground plane in the layer below the top layer and a power plane layer then the bottom layer. The digital signals need to be properly terminated to prevent overshoot and undershoot and need to be routed away from the audio inputs.
Then there is the input conditioning circuits. They need to be low noise and the circuit needs to be biased to the REFOUT unless you are cap coupling to the PGA.
Please see the Analog Input Design Considerations section of the datasheet. Page 24. There is talks about how to use the reference and figure 22 and 23 illustrate what I am discussing.
This should help you find the noise in your design. If you need more assistance then please include your schematics.