I am trying to bring up ADV7611 on a new board. It is working fine at resolutions up to 800x600, 60fps. When i increase the resolution to 1024x768, 60fps, the output of the device starts to behave in an unexpected manner.
The hs signal seems to be correct. The vs signal however has the correct frame period but occasionally (perhaps one in 30) the vs stays high for the duration of a frame. In addition the DE signal is behaving in an unexpected manner. In some frames DE is not asserted at all (around 1 in 4), in others DE toggles continuously throughout the frame and into the following (i.e. DE is not low through the VSync pulse).
The TMDS Pll and vertical filter are reporting lock in the register interface.
Please could you give me some advice about what might be going wrong and how to debug this further? I am at a loss of what to try next.