I have an application where an RF 200 MHz source is delivered by VCO (Minicircuits ZOS-200+) into the RFIN_I input of the ADL5390-EVAL board. The Q inputs are terminated, and the I baseband inputs are P: ground and N: to 0.5V. The RFOP, final output, from the eval board shows the central line with frequency-dependent heavy modulation. As frequency is varied, behavior can range from 100 MHz sidebands to densely spaced sidebands of varying height. Behavior is modified by whether or not the Q inputs are terminated or jumpered to 0.5V. 5V supply is provided by a LM7805 regulator, with 10 uF aluminum polarized capacitors. What are possible sources and solutions to this modulation problem?
Are you using the main output of your VCO (ZOS 200+)? This has a +10 dBm output. You are compressing the ADL5390 for sure and mostly likely the AD8368 (what is the gain of the AD8368?).
Max input/output for the ADL5390: 7/11 dBm (1.42/2.24 Vpp) @ 140 MHz
Max input/output for the AD8368: 19.5*/16 dBm (6*/4 Vpp) @ 140 MHz
*This is with the gain at minimum.
Use the auxiliary output (-11 dBm) on your VCO or put at least a 20 dB pad on the main output and re-test. I'm sure the spectrum will be much cleaner.