We were able to set the TMDS registers in the AD9880, these are not documented in the datasheets, however we had found the following information from AD to be very helpful
(Note that we found setting Regs 0x53 and 0x59 to 0 instead of what AD recommend, however that is in our setup, your's may be different).
AD9880 TMDS PLL Settings 1-19-06
The recommendations for the AD9880* TMDS PLL settings (register 0x4D - 0x50) are
being updated. These settings will be incorporated into the next revision of the AD9880
data sheet. These settings are a result of more data being compiled through our
Compliance Testing Lab. These settings are outlined in Table 1 below.
Table 1: AD9880 TMDS PLL Settings
Register Old Setting New Setting Description
0x4D 0x36 0x3B Bits[6:4] set the C1 loop filter values (high freq).
Bits [3:0] set the Ch. Pump Current value (high freq)
0x4E 0x36 0x6D Bits [6:4] set the C1 loop filter values (low freq).
Bits [3:0] set the Ch. Pump Current value (low freq)
0x4F 0x33 0x54 Bits [7:5] set the C2 loop filter values.
Bits [4:0] set the R1 loop filter values
0x50 0x20 0x90 This sets the VCO gear manually.
0x53 0x3F 0x3F Phase Recovery Loop control
0x59 0x20 0x20 Controls TMDS clock termination connect
* These settings also apply to the AD9380, AD9381, AD9396, AD9397, and AD9398.