If the outputs of the ADF4371 are terminated with 50Ohms Tx line. Would it produce square or sine waves for the output?
ADF4371 outputs should be AC-coupled to SE 50Ω's or differential 100Ω.
It is not true to classify the output as square or sine wave. As you can see from Figure-15, the 3rd and 5th harmonics are significantly high which make the signal more like to square wave (high slew rate). The harmonic levels are decreasing as the frequency is increasing, so the output starts to behave like a sine wave, but that is not a true/real sine wave anyway.