Q: The datasheet for the MAX2769 provides information on how to set the GAINREF value for 2-bit outputs, but how should it be set for a 1-bit output?
A: Based on the way the AGC works, it shouldn't be used with a 1-bit output. The ADC quantizes the data into several levels, the number of levels dependent on the number of bits. For sign/magnitude encoding, the first bit is the sign (0 for positive, 1 for negative), and the rest of the bits are a magnitude. So for 4 bit operation, the quantization levels range from 1111 on the negative end to 0111 on the positive end. The GAINREF value tells the AGC what proportion of inputs will give an extreme code count (either 0111 or 1111). The AGC then scales the gain value to achieve this proportion.
From this, it doesn't make sense to use AGC with a 1-bit output. The only outputs are 0 or 1, so every value is an "extreme value." Instead, gain should be adjusted manually to whatever provides the best SNR. The ADC will be acting as a comparator, looking at whether the input is above or below the midpoint, so a high gain value will likely provide the best results.