what is the ADC saturation signal strength of AD9864 at normal working conditions.?
The AD9864 has a unintended unique feature in that the LNA/Mixer bias setting is such that causes compression before reaching the ADC's full-scale of 0 dBFS. Refer to figure 10 of the datasheet to observe the gain compression characteristics of the LNA mixer as it approaches saturation point when operating with max attenuation. If the VGA gain is set to minimum attenuation, then AGC would need to limit IF input such that it does not exceed the "IF input Clip Point at Miniumum VGA Attenuation" specification (page 3) of -31 dBm to avoid instability of the sigma-delta ADC (approximately -2 dBFS relative to ADC, refer to page 14 for definition of specification).
The minimum signal that ADC can handle depends on the ADC's Noise Figure at Minimum attenuation that sets the noise floor of the ADC (relative to dBm/Hz), the required minimum CNR to demodulate signal, and the BW of the Signal.
For example, if the AD9864 as a NF=7.5 dB at min. attenuation and is demodulating a signal that is 10 KHz in bandwidth with a CNRmin=10 dB than the minimum signal is -126.5 dBm (=-174 dBm+10*log(10^3)+7.5+10 dB).
With regard to AGC setting, the AGC is reacting to rms power of the input signal thus the setting should take into consideration the pk-to-rms of the signal (or blocker) and set the threshold of the AGC accordingly.