I have recently looked at the performance of the TX (& RX) FIR filters in the AD9361 using your No-OS bare metal driver (latest version) and a simple tone output generated from the FPGA. I am using Zeboard with FMCOMMS3.
I created the tone by connecting a simple Xilinx DDS core and routing this to the DAC DMA port of the axi_ad9361 FPGA component (latest version) provided by your HDL No-OS reference design. I have edited "main.c" in the driver so I can program a frequency into the DDS core.
For some reason I am seeing huge spurs (on a Spectrum Analyser) at the Nyquist zones, outside the bandwidth of the filter when I pass the tone through the passband of the filter. At first I was trying my own low pass filter, but after experiencing this problem I decided to use to the default TX band pass filter (64 coefficients, single rate) provided by your bare metal driver. Again, I see the spurs when I program my DDS with a frequency that it tuned to the pass band of the filter. When the DDS is programmed to run out of the bandwidth of the filter, the tone and spurs are almost undetectable, as you would expect. I changed the filter to interpolate by 2 and I see the same problem.
Have you seen anything like this?
Additionally, I have an internal Xilinx Interpolation (x2) low pass FIR filter between my DDS core and the axi_ad9361 component and I have had working with for a few months, which I can bypass, for comparison to the AD9361 FIR filter. When I use my FIR filter and bypass the AD9361 FIR filters, I do not see this problem.
I have attached pictures of the spectrum analyzer screen for comparison.