The ADV7604 is being fed a signal resolution 1280 x 1600; clearly this ia a non supported video format by the part.
We have an engineering document that details the use of the ADV7604 in a non supported video format. The document is attached here.
One thing to be mindful of is how to determine the HSync frequency, PLL divide ratio. The HSync = (Active video + Horizontal blanking / Pixel clock Frequency). The PLL = Active video + Horizontal blanking.
We hope that the document will be of help to customers.