What is the difference between RTI and RTO as specified in the AD8237 datasheet? Which one should I use in my noise calculations?
The main function of the AD8237 instrumentation amplifier is to amplify small precision signals, so it's important to understand the different noise sources that can affect your measurement. As you have highlighted these are specified as noise referred to the input, RTI, and noise referred to the output, RTO. The input noise is reflected to the output by the in-amp gain G. The output noise is referred to input by dividing by the gain G. The RTI is integrated over the applicable in-amp closed loop bandwidth to enable the rms value to be calculated relative to the sensor input signal. Being an indirect current feedback instrumentation amplifier, the AD8237 only has an input voltage noise component (eni) whereas 3 op-amp In-amps have both an input and output voltage noise (eno) that will change the total RTI noise as the gain changes.
To get greater insight into the key equations used for the calculation of the RTI and RTO then take a look at mini-tutorial MT-065 found here. Also take a look at this engineer zone post that goes through a worked example on how the RTI and RTO are calculated and how they contribute to the error budget, these calculations can easily be adjusted for the AD8237 using the details from the datasheet.