I have looped back the receiver to the transmitter on the ad9361 in hardware and am running on a fmcomms2 board connected to a zc706. Things looked pretty good when the signal was above about -35 dbm. When it was lower than that there was some noise but not too bad. I then removed the 40MHz oscillator and connected a 1V p-p 10MHz sine wave to the external clock in to the sma port and added the required capacitor. The radio creates a lot of noise and the signal output is not stable when the input is the same as the LO frequency. Attached is a gif of the problem and what it looks like when the input signal is not directly on the LO.
If you can see in the image, the signal rapidly varies between about -2 and -18 db. When the signal is slightly off the LO freq the signal looks much better as below.
The only thing I can think of is the clock is causing the error but I dont know why the noise only shows up at the lo frequency.