We have a long-standing product using the ADV7441a, that has previously been incorrectly outputting 'full range' video data (0-255). We are attempting to modify this such that it only outputs limited range (16-235).
We have set bit 4 of register 0x6b in the User Map, and this appears to be correctly scaling the input in all cases except when we put in an HDMI input that uses the YUV colour space. In this case, if we put in a grey scale image with 8 steps, we can see by looking at the Y values that the step size of the top and bottom gray scales are smaller than all the others. This indicates that the gain factor of 0.859 isn't correctly being applied.
By examining the data coming out of the ADV7441a, it is correctly putting out data using the limited range of 16-235. It would appear that the 7441a is therefore deciding that the input is also using limited range, so is just applying a gain factor of 1, rather than the correct 0.859.
PRIM_MODE is set to 6 (HDMI)
OP_656_RANGE bit is set (limited range)
INP_COLOR_SPACE is set to 7 (auto)
RGB_OUT is cleared (YUV out)
If we change the input colour space to 2 or 3 (YCrCb 601 and YCrCb 709) we see the same effect.
When we feed in HDMI using RGB colour space and set INP_COLOR_SPACE to 1 (RGB limited range) and RGB_OUT to 1 (RGB out) the chip behaves correctly and scales the data such that the output data is correctly in the range 16-235 with all grey scale steps the same height.
From this, it would appear that the gain value isn't being applied correctly when we put in YUV data.
Attached are images of oscilloscope traces showing the RGB case (with all step sizes the same) and the YUV case (where the top and bottom steps are smaller than the rest).
Is there any other register we need to modify to tell the 7441a that the input data is full range rather than limited range as it appears to be assuming. Can you suggest anything else we may have missed?