I have imported your spice model in LTspice and connected it as in Figure 65 of the datasheet. However, the output signal does not behave as expected. I have tried the same circuit in PSpice with the same result.
The circuit is valid but the output response is not correct. It is most likely the internal structure of the model that has a problem which is more likely because it is not model with gain setting pin reference to VCOM. I would check the model and see what we can do but you can still use the model but with gain setting pin not refer to VCC/2.
I tried your suggestion of taking GAIN in reference to ground, which did give a different result, but not a correct result.
Quoting the datasheet:
For a ground referenced design, the VGAIN range is from −0.7 V to +0.7 V with the most linear-in-dB section of the gain control between −0.6 V and +0.6V. In the center 80% of the VGAIN range, the gain error is typically less than ±0.2 dB. The gain control voltage can be increased or decreased to the positive or negative rails without gain foldover.
From the simulation it is very clear that gain foldover does happen. Please fix your model.
Retrieving data ...