*1 RB sets the DAC's full-scale peak-to-peak swing and the DAC's output bias level
*2 Shunt resistor RS scales down the DAC's peak-to-peak voltage without affecting the bias level
*3 The "DAC Peak Back-Off" refers to how close the peaks of the DAC's output signal come to the DAC's full-scale output voltage
*4 The Crest Factor that is inputted sets the backoff level between the DAC Peak Output Level and the RMS Level of the carrier
Therefore: DAC Fullscale Level = RMS Level + Crest Factor + DAC Peak Back-Off
In general, the signal peaks need to be backed off from the DAC's full-scale level by a few dB. The amount of back-off required depends on the desired/acceptable signal distortion
*5 Modulator Voltage Gain is defined as the dB difference between the voltage level on the I or Q input (generally these are equal) and the RF Output
We can generalize the relationship between V_peak and V_rms for any type of signal with signal CFR. If it is CW(sinewave), CFR is 3 [dB] and V_rms is 1/sqrt(2) * V_peak. Let's drive the equation below.
Signal CFR [dB]= 10*log(P_peak/P_rms) = 10*log(V^2_peak / V^2_rms)=20*log(V_peak /V_rms)
V_peak/V_rms = 10^(signal CFR[dB]/20)
V_rms = V_peak * 10^( - signal CFR[dB]/20) , V_peak = Vp_p/2
= Vp_p/2 * 10^( - signal CFR[dB]/20)
There is typo at sign assuming signal CFR[dB] is positive number. I will put negative sign at original one and correct it. And at the DAC output swing level, when we set IFS at 20mA, IOUT1P+IOUT1N is 20mA(when IOUT1P is 20mA, IOUT1N is 0mA, for example). Voltage swing level at each IOUT1P or IOUT1N is RB*IFS*10^(Digital Bask Off[Dbfs]/20) and Voltage[Vp_p] between IOUT1P and IOUT1N is 2 times to each. So the equation is correct.
I think you have a mistake in calculation.
Modulator Input level [Vrms] = Modulator Input level [Vp_p]/2 x 10^(signal CFR[dB]/20 )
What you calculate here is not RMS value but PEAK value.
If you want RMS value it has to be:
Modulator Input level [Vrms] = (Modulator Input level [Vp_p]/2) x (sqrt(2)/2) x 10^(signal CFR[dB]/20 )
Also calculation with Full scale current you have to be really careful. Because for example from datasheet of AD9125:
The DAC full-scale output current (IOUTFS) is nominally 20 mA. The output currents from the IOUT1P/IOUT2P and IOUT1N/ IOUT2N pins are complementary, meaning that the sum of the two currents always equals the full-scale current of the DAC.
So it means in calculation you have to take into account and
DAC output level[Vp_p]= 2 x RB x IFS X 10^(Digital Back Off[dBFS]/20 )
DAC output level[Vp_p]= 2 x RB x (IFS/2) X 10^(Digital Back Off[dBFS]/20 )
Otherwise yours calculation is "improved" by ~6dB (I'm not 100% sure, if I'm wrong just let me know)
Do you have similar calculations done for ADL5811 mixer driving AD9255 ADC as predistortion receiver?