Does the TMC5130 with an incremental encoder use closed loop control?
Although the TMC5130 has inputs for an incremental encoder, it does not natively implement closed loop control.
The target position (XTarget register) does not get compared to the encoder's position (X_ENC register), but rather the on-chip motion controller's position, X_ACTUAL. As such, the encoder's position is not acted on directly by this IC.
The encoder interface on this device is intended to be used as a confirmation tool. Your host can compare X_ENC to X_ACTUAL to check if any steps have been missed and if your X_ACTUAL value is correct. If it is not, you can load the value of X_ENC into X_ACTUAL (make sure you set the IC to velocity mode by setting RAMPMODE to 1 before you overwrite X_ACTUAL, otherwise, overwriting the contents of X_ACTUAL will trigger the motor to move).
If direct encoder based closed-loop control is desired, please look into the TMC4361A motion controller IC.