ADRF6518 filter setup problem ,if the IF is 1M the filter bandwidth set to 2M ,the output will distortation ,if set the bandwidth to 10M the output is ok ,please refer the picture
It is really hard to tell what the distortion components are and what their magnitudes are from an oscilloscope.
Could you remeasure using a spectrum analyzer? Please set the span from a few kHz to at least 2 GHz.
This could be the 1.2GHz oscillation that shows up sometimes when low filter corners are programmed. Please see the datasheet on page 32 for a discussion on this.