We re-organized lots of IO lines, and connected more lines to the video parts of the Zynq.
The ADV7511 is now connected using the 24-bit setup, and that works well, I only had to change the mode "id" in the devicetree to "0".
With the ADV7611 capture device, which is now also "fully wired", we didn't have that much luck. It required some changes in logic, guess we could send a patch for that, but I can't get the driver side of things configured properly. All I could find was a datawidth path currently set to "16" which I could set to "24", but nothing really changed with that. I was looking into the adv7604 driver to figure out if there were some devicetree words that I could supply to put the chip in the proper mode, but didn't find it. Is the current driver capable of handling this? And if not, what should be changed?