AD8302 Magnitude/Phase Output Saturation

 I see a few potential issues with your circuit that may or may not be causing the problem. I assume that you are seeing fixed dc levels at the magnitude and phase outputs (0.9 on VMAG and 1.8 to 2 V on VPHS. And you were expecting both signals to be moving because they are at different frequencies (that should at least cause the VPHS signal to be changing).

1. The OFS caps are set 1nF. That sets an input high-pass corner frequency of 2 MHz. If your lowest input frequency is 17 MHz, that is probably ok but I might make that cap a bit bigger to move the corner a little further away from 17 MHz.
2. There is no need for the resistor on VREF. It's not doing any harm and is unlikely to be related to the problem you are seeing. But it's unnecessary.
3. The 10 nf caps on MFLT and PFLT set an output low pass corner of 4.8 KHz. If you are looking to see instantaneous changes in phase between the two input signals, I would say that this corner frequency is too low.  

I've not heard back from you. Let me know if you need additional help. Otherwise, I'll close this out. 

It's hard to know what is going wrong here. This device should not be difficult to solder as it has leads and no slug on the underside. I double checked this; while I don't have a device at hand, the documentation suggests that this is a 14- pin  TSS0P without an exposed pad.  If I'm interpreting your layout correctly, you appear to have a large area of ground plane under the device with vias. While this is not necessarily a bad thing, I'm wondering if the ground plane's proximity to the device's pins is causing shorts. If you are putting solder paste on the ground plane area under the device, I would remove that because it has nothing to attach to. This may cause it to bleed on to the pins.