In attempting to debug the source of noise in the received data we've gone as far as to adjust the jumpers and connect the output of the DAC directly to the ADC. This takes the mod, demod, LOs, and VGA out of the loop and the problem still remains. We've measured the output of the DAC on a signal analyzer and it's good to -100dB so we believe the issue to be in the ADC.
I've attached a series of plots showing the FFT of the received data for three signals:
1) No signal (i.e. zeros)
2) 7.68MHz cosine applied to I and Q
3) 20MHz wide noise
In (1) and (3) you'll see that the are various multiples of 30.72MHz present in the I data. Swapping the IQ from the DAC does not switch which channel has these spurs which leads further evidence pointing to the ADC. Since these frequencies are derivatives of the input clock it seems that somehow this is feeding through into the output.
In (2) these same spurs are present but you also see harmonics of the cosine in there. Again only in the I data.
In all three there are spurs at 69.12MHz. This shows up in *both* I and Q and I don't have a working theory yet as to where that could be coming from.
I've attached JPGs of the plots but I can provide Matlab .fig files if that's preferred.