On page 27 of the data sheet, there is a note which says the INT value cannot be incremented by more than 256 from the starting value in the section labeled "Number of Steps".

This one is confusing. The INT value is a 12-bit value, most of examples are showing this is rather small at 241 in the worked examples. The number of steps in R6 is a 20-bit value.

What is the relation here? One value can range from 0-4095, the other from 0-1048575.

The ADF4159 sweep must be <=(256*Fpfd). For example if the phase frequency detector frequency (Fpfd) is 1MHz the total sweep must be <256 MHz. This effectively limits the number of steps for any given DEV and DEV_OFFSET.