The criteria to select the sense resistor in a Maxim Fuel Gauge should be to optimize the analog and digital range of the Fuel Gauge.
For the analog range, Maxim Fuel Gauges typically have a +/-51.2mV measurement range to measure the current across the sense resistor. Based on the application maximum current, the sense resistor should be selected to make as much use of this measurement range as possible.
Current LSB: 1.5625µV/ RSENSE
Current Measurement Range: +/-51.2mV/RSENSE
For the digital range, the sense resistor should be considered to determine the LSB of the capacity registers in order to determine if the capacity of the battery will be less than the maximum capacity value.
Capacity LSB: 5.0µVH/ RSENSE
Maximum Capacity Value: 0x8000 * Capacity LSB
The selection criteria in the following table take these items into consideration and give some overlapping suggestions about battery size and sense resistor, but in the end, you will have to take the range/resolution into consideration if you are using very small or very large batteries or have very small or very large currents.
For example, if you have a very large battery with relatively low currents (32 Ah, <5A max current) it may seem acceptable to use a 10mΩ sense resistor on the analog side, because the application will not exceed the 51.2mV range. However, because of the large battery, you will be over the 0x8000 capacity limit, so a smaller sense resistor should be selected. Alternatively, you can use a scale value of CGAIN to keep the capacity value below 0x8000.
On the other hand, if you use a sense resistor that is too small, it may not use the full analog or digital range. The analog performance may not be optimized because the ADC range is not being fully utilized. If the sense resistor is selected such that the capacity LSB is too small, there will be quantization in the SOC calculation. For example, a customer using a 10mΩ sense resistor (0.5mAh LSB) on a 200mAh battery will have 0.25% SOC resolution. This will make internal calculations like mixing and empty compensation more granular, and will make the SOC reporting less smooth.