It suggests to use 105ohm resistor for REF pin in datsheet But EVK uses 330ohm resistor, why?
The REF pulldown resistor value is determined by the impedance of the RGB signals and specifically how that impedance is achieved. If the impedance is achieved by routing the trace as 75 ohm, then both 330 ohm or 105 ohm REF resistors will work but the 330 ohm resistor will provide more noise margin for monitor detection. If the impedance was achieved by adding 150 ohm termination resistors to GND on 50 ohm characteristic impedance traces, then the customer must use 105 ohm REF resistor.