When you change the reference (or more specifically the PFD frequency) when using digital lock detect, you need to "re-train" the lock detect window. This is done by toggling REG7 =1 then REG7 = 0. This can easily be done if through the HMC Evaluation board GUI by selecting the 'Open Detailed GUI' radio button, and selecting the "20: Lock Detect Train req" bit field in the REG 7 frame bit and writing, then unselecting the check box and writing again. Upon returning to the 'Main' GUI you should see that the part is locked. If not you may need to re-enter the frequency, selector unselect 'Frac Mode' as needed and click 'Update Frequency'.
If I understand you it sounds like it locks with a 50MHz REF + 50MHz PFD (RDIV = 1) but it doesn’t lock with 100MHz REF + 100MHz PFD (RDIV1) but will still lock with 100MHz REF + 50MHz PFD (RDIV=2).
Since the onboard reference is a 50MHz TCXO this indicates that you're using an external reference. Be sure that Jumper 'JP1' is removed so that the on-board 50MHz is powered down when you use the external reference.
If you're already doing this and toggling the lock detect window as discussed earlier isn't working then I suspect that the loop filter may be unstable at the 100MHz PFD frequency which is possible however simulation shows that it should be fine.
You can use ADISimPLL (use link below) to design a new loop filter for this REF & PFD frequency if you believe the loop to be unstable. Select the HMC703 for the PLL and you can use the HMC510 for the VCO.
a)it locks with a 50MHz REF + 50MHz PFD (RDIV = 1)
b)it doesn’t lock with 100MHz REF + 100MHz PFD (RDIV1)
c)it doesn’t lock with 100MHz REF + 50MHz PFD (RDIV=2)
d)it locks with a 150MHz REF + 50MHz PFD (RDIV = 3),but i need upgrate twice sometime then it can lock
the loop filter i have changed for 100MHz REF + 100MHz PFD (RDIV1) by using Hittite design software ,and in this case ,i did a)~d)
the board is not the demo board,i made it by following the HMC767 reference databook