LT3751 voltage sense problem

I am trying to create following isolated flyback capacitor charger:

Vin: 12 V

Vout: 510V

N2/N1 = 25

Relatively low current and capacitor value is also not that large so maybe max. 10u.

Circuit works perfectly fine when I have high current ( so low current sense resistor, for example 10mOhm) or a lower output voltage.

However if I limit my current and try to reach 510 Volts the circuit does not stop to charge. I even look at the current flowing through the RBG resistor and when working correctly the charger stops once the voltage over this resistor reaches 0.98 Volts. However when it doesnt work the voltage(and thus also the current) increases past this. I thought this was the criterion for the circuit to stop.

I hope you can help me solve this problem as I am really out of ideas as to why this is happening.