I've designed in the HMC773LC3B as the mixer in this up-converter. LO is +10dBm at 12GHz, IF is -10dBm at 5GHz, but I'm getting <-40dBm of RF at 17GHz. I've looked at the mixer ports on a curve tracer, LO is AC coupled and IF port has the back/back diodes as would be expected from the data sheet, BUT the RF port is showing a resistive 10ohm, not the AC coupling that the data sheet shows. I've checked mixers insitu and unused parts, all show the 'resistive' Rf port. Is this correct, and the data sheet wrong ??
Just for your information the Die version of the HMC773 would appear to have DC coupled ports as can be seen from the table from the data sheet and the spiral inductors on chip in the picture below.
Here is a picture of the die
So maybe the packaged version has a mix of DC and AC coupling which is contrary to the HMC773LC3B version of the datasheet.
Also +10dBm LO drive, whilst probably not providing an optimum drive for spur suppression, this power level would only really give a degradation in conversion loss by a few dB (see picture below) and not cause the mixer to stop frequency conversion completely as is the case in your application.
An evaluation board can be obtained from here. 13 in stock as of 23-03-2016
125042-HMC773LC3B Analog Devices Inc. | RF/IF and RFID | DigiKey