Are you using an Evaluation board or a custom board? If you are using a custom board, did you simulate loop filter configuration with ADISimPLL?
The n-Divider output of the Mux will have a very small duty cycle. It is possible to miss pulses using an oscilloscope without the correct settings. A duty cycle of N-Divider output is heavily dependent on N divider value. The n-divider/2 output will have a 50% duty cycle because of dividing by 2.
