At the smallest input, random jitter has the greatest contribution to the total output jitter. 4mVpp is the guaranteed input swing at low end, so it is used to give the RJ spec on the MAX3747 datasheet. It would be a correct assumption that the RJ performance is better than that at 4mVpp, for input signal above 4mVpp (but below 1200mVpp).