On our dev setup using FMCOMMS5, the ADI driver successfully initializes both chips. On our custom board, whose external reference mimics FMCOMM5, we get Calibration timeouts for the Baseband PLL (0x5E, 0x80) about 50% of the time. When it DOES lock, other calibrations fail/timeout. With some strategic printfs, we see illegal programming take place on occasion (i.e., BB PLL divider set to 0). We also occasionally see the radio ID come back as 0. Admittedly, this sounds like a SPI issue, but the same driver code works on the dev setup.
You can try using a lower SPI speed for debug.
In custom board what is the reference clock. You can check for stability of reference clock as well.
Hope all power rails have correct voltages and no voltage dips during initialization.
Issue was identified - turned out to be a signal integrity issue on the SPI interface. We were so focused on the clock quality, that we initially didn't pursue this. Sorry for any time wasted, thanks for the attention!