This could be a result of leaving the pins floating. Since there is no active pulldown on these pins their value is going to somewhere between ground and ref but it will be meaningless if left floating. Once you connect these pins to an analog signal you will get valid readings.
It is also possible that the period betweens sampling is less than the tACQ as described in the datasheet on page 10. If the source resistance is large enough the time to discharge the capacitors might be larger than the tACQ and so they will still have a voltage across them when taking the next reading.