Working with the MAX2850, but also applies to MAX2851.
The datasheet has a BYP_VCO pin (Bypass VCO) that is solely intended to provide a decoupling capacitor to ground.
However the datasheet shows it as an "input" pin. Is there a mode where you can provide an external LO on this pin as that suggests?