1- The loss is due to the balun and the matching network. Both will introduce loss.
2.- Figure 17 SFRD measurement is when is in DCC0 mode, that example has 8 decimation (bandwidth of 375 MHz). While in the figure 21 the system is in full bandwidth mode (1.2 GHz bandwidth).
Bigger the bandwidth, SFDR will be larger, but if we reduce bandwidth to 375 MHz will go up because of the filtering done in DCC0 mode.
Sorry, made a mistake. Is not DCC is DDC (Digital Down Converter). This is a feature embedded in the AD6688 chip. That allows to de-mix the signal digitally by using an NCO (numerical controller oscillator). On DDC mode you can select the bandwidth by choosing the filtration, and after applying filter the signal will be decimating depending on the filter factor. If you want to read more about it, please referrer to pages 42 to 62 on the AD6688 spec sheet.