when use the offset calibration, which conversion mode should I choose ? All of the three conversion mode can be use? And if my system has a negtive offset, for example, -5mV as the input signal, can the offset calibration can cancel out this -5mV? To my knowledge, if I choose the single-end converstion mode, the input signal need to be larger than 0mV, so -5mV will cause problem even I use the offset calibration?
Yes, all 3x conversion modes can be used - the effect is the same.
I don't have a board to check this on but, the offset calibration register should be signed so, you can increase/decrease the offset adjustment.
To calibrate, you could apply 10mV to the input and keep adjusting ADCOF until you get the correct ADCDAT value.