Can you have RoR on one link and Xtal mode on the other link

Yes, it is possible to have one link in crystal mode and other link in RoR mode on the deserializer, refer to device-specific data sheet and user guides for serializers and deserializers that support RoR mode.