carrier leakage from the transmitter in ad9361

What happens in manual mode? What happens when you loopback to the same board?

-Travis

I use No - os driver in default mode (fmcomms3), and then I set 20000000UL, //reference_clk_rate and set xo_disable_use_ext_refclk_enable to 1 in AD9361_InitParam default_init_param (I use two external 20 MHz MCXO for two fcomms3).
When I loop back to the same board, it's correctly received, and it's not any problem

as well as I connected two board to one common external clock; it's no problem

Then the differences you are seeing are timing differences between the source clocks. They are effectively creating a phase modulation in your waveform. This is a common artifact synchronization algorithms will remove.

-Travis

How can I remove it? I don't have any idea to solve it.  I apologize; I don't understand these sentences " This is a common artifact synchronization algorithms will remove" ?which artifact synchronization algorithms? 

In order to demodulate QAM256, you will need to recover the timing and frequency offsets in the signal (among other things). This is done algorithmically and needs to be done for any wireless comms systems.

-Travis

yes, I did recover the timing and frequency offsets in the signal  and also used a seven tap equalizer but this problem effected on output constellation 

Are you sure this is not a problem from carrier Leakage in TX Side or something else?

If you want to focus on carrier leakage send a tone and inspect the DC frequency directly.

-Travis