Post Go back to editing

ADRV9009 ARM time out when setting LO frequency

Category: Hardware
Product Number: adrv9009

Hello

We have custom hardware with three ADRV9009 devices. These devices are controlled through the ADI Linux IIO device driver. We are using version 2019_R2. Our design is stable and we have manufactured multiple units.

However, under certain circumstances, we see problems with the ADRV9009 ARM during the setting of the LO frequency. We are setting the LO frequency in a range between 100MHz to 400MHz. Our sample rate is 204.8MSPS (166MHz BW) using a profile generated by the ADR9009 TES and Filter Wizard.

To make the problem happen, we do successive large jumps in the LO frequency - 100MHz per jump. So for example, we would set the LO frequency to 100MHz, 200MHz, 300MHz, 400MHz and then back to 100MHz. If we jump around randomly between these frequencies, and do it often enough, we eventually trigger a fault in the ADRV9009 ARM and dmesg starts spitting out the following messages:

[ 288.518808] adrv9009 spi1.2: ERROR: 247: TALISE_waitArmCmdStatus() failed due to thrown ARM error. ARM time out
[ 288.528998] adrv9009 spi1.2: adrv9009_set_radio_state: failed
[ 288.534782] adrv9009 spi1.2: adrv9009_set_radio_state: failed
[ 290.604279] adrv9009 spi1.2: ERROR: 179: ARM Mailbox Busy. Command not executed in TALISE_sendArmCommand()
[ 292.677449] adrv9009 spi1.2: ERROR: 179: ARM Mailbox Busy. Command not executed in TALISE_sendArmCommand()
[ 294.750656] adrv9009 spi1.2: ERROR: 179: ARM Mailbox Busy. Command not executed in TALISE_sendArmCommand()
[ 294.760313] adrv9009 spi1.2: adrv9009_set_radio_state: failed
[ 294.766096] adrv9009 spi1.2: adrv9009_set_radio_state: failed
[ 294.771873] adrv9009 spi1.2: ERROR: 439: TALISE_setRfPllFrequency() : Invalid rfpllLoFreq, rfPllLoFreq - TxProfileRFBW/2 must be > 0 (DC)
[ 296.877856] adrv9009 spi1.2: ERROR: 179: ARM Mailbox Busy. Command not executed in TALISE_sendArmCommand()
[ 297.888570] adrv9009 spi1.1: ERROR: 247: TALISE_waitArmCmdStatus() failed due to thrown ARM error. ARM time out
[ 297.898754] adrv9009 spi1.1: adrv9009_set_radio_state: failed
[ 297.904530] adrv9009 spi1.1: adrv9009_set_radio_state: failed

Thereafter, the ADRV9009 responds 'slowly' to requests. It seems like other commands thereafter also fail (like the adrv9009_set_radio_state command as you can see in the above). Another side effect of the ARM being in this broken state is that the temperature reading is completely erroneous. The only way to recover the ADRV9009 is to power cycle the complete unit. Rebooting the Linux does not fix the problem.

If we change the LO frequency in smaller steps (say 10MHz jumps), we cannot reproduce the problem. Also, if we don't have any signal connected to the RX, we cannot reproduce the problem. We are only able to reproduce the problem by having the RX connected to an antenna and by doing large 100MHz frequency jumps.

We are only using the RX channels. We have tried disabling the RX tracking corrections but the fault still happens. We only do initial calibrations at the start and those perform fine so this is not a result of a failed initial calibration (as I have seen elsewhere on the forums).

The ADRV9009 ARM code is a black box to us. Please can you provide some guidance or insight as to why the ADRV9009 ARM would be failing in this way. I have posted another question on the Linux forum about retrieving any logging that is available.

Thank you in advance.

Gavin

Parents
  • Another question:

    Is the ADRV9009 ARM reset when the ADRV9009 is reset through the RESETN (pin J4)? I can see that the RESETN pin of the ADRV9009 devices are toggled low during the Linux boot up process. I would expect that if the ADRV9009 ARM locks up for some reason, that resetting the ADRV9009 device would recover the ARM from the locked-up state. However, as I have mentioned before, the only way to recover correct functioning of the hardware is to perform a complete power cycle of the system. I am hoping that this could give another clue as to what could be going wrong with the ADRV9009 ARM.

    Thank you.

    Any advice would be greatly appreciated!

  • can you please share the LO frequency, bandwidth, and frequencies used for hopping for the experiment?

    as for reset, please below the procedure for resetting the adrv9009 device.

  • Hello

    I have attached the profile that I created using the ADRV9009 Filter Wizard. This provides details of the bandwidths and sample rates.

    <profile Talise version=1 name=Tx_BW160_IR204p80_Rx_BW160_OR204p80_ORx_BW160_OR204p80>
     <clocks>
      <deviceClock_kHz=204800>
      <clkPllVcoFreq_kHz=8192000>
      <clkPllHsDiv=2.5>
     </clocks>
    
     <rx name=Rx 160.00MHz, OutputRate 204.80MHz, TotalDecimation 8>
      <rxChannels=TAL_RX1RX2>
      <rxFirDecimation=2>
      <rxDec5Decimation=4>
      <rhb1Decimation=1>
      <rxOutputRate_kHz=204800>
      <rfBandwidth_Hz=160000000>
      <rxBbf3dBCorner_kHz=160000>
      <rxDdcMode=0>
    
      <rxNcoShifterCfg>
       <bandAInputBandWidth_kHz=0>
       <bandAInputCenterFreq_kHz=0>
       <bandANco1Freq_kHz=0>
       <bandANco2Freq_kHz=0>
       <bandBInputBandWidth_kHz=0>
       <bandBInputCenterFreq_kHz=0>
       <bandBNco1Freq_kHz=0>
       <bandBNco2Freq_kHz=0>
      </rxNcoShifterCfg>
    
      <filter FIR gain_dB=-6 numFirCoefs=48>
      -3
      -10
      16
      28
      -42
      -71
      97
      148
      -194
      -281
      353
      493
      -602
      -821
      983
      1327
      -1565
      -2142
      2500
      3590
      -4320
      -7239
      9270
      31252
      31252
      9270
      -7239
      -4320
      3590
      2500
      -2142
      -1565
      1327
      983
      -821
      -602
      493
      353
      -281
      -194
      148
      97
      -71
      -42
      28
      16
      -10
      -3
      </filter>
    
      <rxAdcProfile num=42>
      229
      168
      181
      90
      1280
      893
      1316
      83
      1190
      38
      829
      21
      48
      42
      30
      210
      0
      0
      0
      0
      53
      0
      7
      6
      42
      0
      7
      6
      42
      0
      25
      27
      0
      0
      25
      27
      0
      0
      165
      44
      31
      905
      </rxAdcProfile>
     </rx>
    
     <obsRx name=Rx 160.00MHz, OutputRate 204.80MHz, TotalDecimation 8>
      <obsRxChannelsEnable=TAL_ORX1ORX2>
      <enAdcStitching=0>
      <rxFirDecimation=2>
      <rxDec5Decimation=4>
      <rhb1Decimation=1>
      <orxOutputRate_kHz=204800>
      <rfBandwidth_Hz=160000000>
      <rxBbf3dBCorner_kHz=225000>
      <orxDdcMode=0>
    
      <filter FIR gain_dB=-6 numFirCoefs=48>
      -3
      -8
      15
      25
      -39
      -64
      76
      101
      -206
      -245
      349
      437
      -583
      -733
      943
      1189
      -1497
      -1924
      2398
      3246
      -4149
      -6490
      9504
      30424
      30424
      9504
      -6490
      -4149
      3246
      2398
      -1924
      -1497
      1189
      943
      -733
      -583
      437
      349
      -245
      -206
      101
      76
      -64
      -39
      25
      15
      -8
      -3
      </filter>
    
      <orxLowPassAdcProfile num=42>
      229
      168
      181
      90
      1280
      893
      1316
      83
      1190
      38
      829
      21
      48
      42
      30
      210
      0
      0
      0
      0
      53
      0
      7
      6
      42
      0
      7
      6
      42
      0
      25
      27
      0
      0
      25
      27
      0
      0
      165
      44
      31
      905
      </orxLowPassAdcProfile>
    
      <orxBandPassAdcProfile num=42>
      229
      168
      181
      90
      1280
      893
      1316
      83
      1190
      38
      829
      21
      48
      42
      30
      210
      0
      0
      0
      0
      53
      0
      7
      6
      42
      0
      7
      6
      42
      0
      25
      27
      0
      0
      25
      27
      0
      0
      165
      44
      31
      905
      </orxBandPassAdcProfile>
    
     </obsRx>
    
     <lpbk>
      <rxFirDecimation=2>
      <rhb1Decimation=1>
      <outputRate_kHz=204800>
      <rfBandwidth_Hz=75000000>
      <rxBbf3dBCorner_kHz=225000>
    
      <filter FIR gain_dB=-6 num=48>
      -3
      -8
      15
      25
      -39
      -64
      76
      101
      -206
      -245
      349
      437
      -583
      -733
      943
      1189
      -1497
      -1924
      2398
      3246
      -4149
      -6490
      9504
      30424
      30424
      9504
      -6490
      -4149
      3246
      2398
      -1924
      -1497
      1189
      943
      -733
      -583
      437
      349
      -245
      -206
      101
      76
      -64
      -39
      25
      15
      -8
      -3
      </filter>
    
      <lpbkAdcProfile num=42>
      267
      168
      181
      90
      1280
      623
      1285
      51
      1137
      25
      727
      30
      48
      41
      27
      190
      0
      0
      0
      0
      48
      0
      7
      6
      42
      0
      7
      6
      42
      0
      25
      27
      0
      0
      25
      27
      0
      0
      165
      44
      31
      905
      </lpbkAdcProfile>
     </lpbk>
    
     <tx name=Tx 160.00MHz, InputRate 204.80MHz, TotalInterpolation 8>
      <txChannels=TAL_TX1TX2>
      <dacDiv=1>
      <txFirInterpolation=1>
      <thb1Interpolation=2>
      <thb2Interpolation=2>
      <thb3Interpolation=2>
      <txInt5Interpolation=1>
      <txInputRate_kHz=204800>
      <primarySigBandwidth_Hz=75000000>
      <rfBandwidth_Hz=160000000>
      <txDac3dBCorner_kHz=187000>
      <txBbf3dBCorner_kHz=80000>
    
      <filter FIR gain_dB=0 numFirCoefs=20>
      26
      -73
      131
      -188
      232
      -209
      -31
      946
      -2961
      20082
      -2961
      946
      -31
      -209
      232
      -188
      131
      -73
      26
      0
      </filter>
     </tx>
    </profile>
    

    In terms of the LO frequencies, I have already described the process followed to recreate the problem. I jump the LO frequency randomly by 100MHz jumps between 100MHz and 400MHz. Please see the original question posted for more details.

    Thank you for providing details of the reset. This reset is being asserted by the ADRV9009 IIO Linux device driver during boot up. Please can you answer the following question: 

    - Is the ARM in the ADRV9009 reset when the external RESET pin is toggled?

    Thank you.

Reply
  • Hello

    I have attached the profile that I created using the ADRV9009 Filter Wizard. This provides details of the bandwidths and sample rates.

    <profile Talise version=1 name=Tx_BW160_IR204p80_Rx_BW160_OR204p80_ORx_BW160_OR204p80>
     <clocks>
      <deviceClock_kHz=204800>
      <clkPllVcoFreq_kHz=8192000>
      <clkPllHsDiv=2.5>
     </clocks>
    
     <rx name=Rx 160.00MHz, OutputRate 204.80MHz, TotalDecimation 8>
      <rxChannels=TAL_RX1RX2>
      <rxFirDecimation=2>
      <rxDec5Decimation=4>
      <rhb1Decimation=1>
      <rxOutputRate_kHz=204800>
      <rfBandwidth_Hz=160000000>
      <rxBbf3dBCorner_kHz=160000>
      <rxDdcMode=0>
    
      <rxNcoShifterCfg>
       <bandAInputBandWidth_kHz=0>
       <bandAInputCenterFreq_kHz=0>
       <bandANco1Freq_kHz=0>
       <bandANco2Freq_kHz=0>
       <bandBInputBandWidth_kHz=0>
       <bandBInputCenterFreq_kHz=0>
       <bandBNco1Freq_kHz=0>
       <bandBNco2Freq_kHz=0>
      </rxNcoShifterCfg>
    
      <filter FIR gain_dB=-6 numFirCoefs=48>
      -3
      -10
      16
      28
      -42
      -71
      97
      148
      -194
      -281
      353
      493
      -602
      -821
      983
      1327
      -1565
      -2142
      2500
      3590
      -4320
      -7239
      9270
      31252
      31252
      9270
      -7239
      -4320
      3590
      2500
      -2142
      -1565
      1327
      983
      -821
      -602
      493
      353
      -281
      -194
      148
      97
      -71
      -42
      28
      16
      -10
      -3
      </filter>
    
      <rxAdcProfile num=42>
      229
      168
      181
      90
      1280
      893
      1316
      83
      1190
      38
      829
      21
      48
      42
      30
      210
      0
      0
      0
      0
      53
      0
      7
      6
      42
      0
      7
      6
      42
      0
      25
      27
      0
      0
      25
      27
      0
      0
      165
      44
      31
      905
      </rxAdcProfile>
     </rx>
    
     <obsRx name=Rx 160.00MHz, OutputRate 204.80MHz, TotalDecimation 8>
      <obsRxChannelsEnable=TAL_ORX1ORX2>
      <enAdcStitching=0>
      <rxFirDecimation=2>
      <rxDec5Decimation=4>
      <rhb1Decimation=1>
      <orxOutputRate_kHz=204800>
      <rfBandwidth_Hz=160000000>
      <rxBbf3dBCorner_kHz=225000>
      <orxDdcMode=0>
    
      <filter FIR gain_dB=-6 numFirCoefs=48>
      -3
      -8
      15
      25
      -39
      -64
      76
      101
      -206
      -245
      349
      437
      -583
      -733
      943
      1189
      -1497
      -1924
      2398
      3246
      -4149
      -6490
      9504
      30424
      30424
      9504
      -6490
      -4149
      3246
      2398
      -1924
      -1497
      1189
      943
      -733
      -583
      437
      349
      -245
      -206
      101
      76
      -64
      -39
      25
      15
      -8
      -3
      </filter>
    
      <orxLowPassAdcProfile num=42>
      229
      168
      181
      90
      1280
      893
      1316
      83
      1190
      38
      829
      21
      48
      42
      30
      210
      0
      0
      0
      0
      53
      0
      7
      6
      42
      0
      7
      6
      42
      0
      25
      27
      0
      0
      25
      27
      0
      0
      165
      44
      31
      905
      </orxLowPassAdcProfile>
    
      <orxBandPassAdcProfile num=42>
      229
      168
      181
      90
      1280
      893
      1316
      83
      1190
      38
      829
      21
      48
      42
      30
      210
      0
      0
      0
      0
      53
      0
      7
      6
      42
      0
      7
      6
      42
      0
      25
      27
      0
      0
      25
      27
      0
      0
      165
      44
      31
      905
      </orxBandPassAdcProfile>
    
     </obsRx>
    
     <lpbk>
      <rxFirDecimation=2>
      <rhb1Decimation=1>
      <outputRate_kHz=204800>
      <rfBandwidth_Hz=75000000>
      <rxBbf3dBCorner_kHz=225000>
    
      <filter FIR gain_dB=-6 num=48>
      -3
      -8
      15
      25
      -39
      -64
      76
      101
      -206
      -245
      349
      437
      -583
      -733
      943
      1189
      -1497
      -1924
      2398
      3246
      -4149
      -6490
      9504
      30424
      30424
      9504
      -6490
      -4149
      3246
      2398
      -1924
      -1497
      1189
      943
      -733
      -583
      437
      349
      -245
      -206
      101
      76
      -64
      -39
      25
      15
      -8
      -3
      </filter>
    
      <lpbkAdcProfile num=42>
      267
      168
      181
      90
      1280
      623
      1285
      51
      1137
      25
      727
      30
      48
      41
      27
      190
      0
      0
      0
      0
      48
      0
      7
      6
      42
      0
      7
      6
      42
      0
      25
      27
      0
      0
      25
      27
      0
      0
      165
      44
      31
      905
      </lpbkAdcProfile>
     </lpbk>
    
     <tx name=Tx 160.00MHz, InputRate 204.80MHz, TotalInterpolation 8>
      <txChannels=TAL_TX1TX2>
      <dacDiv=1>
      <txFirInterpolation=1>
      <thb1Interpolation=2>
      <thb2Interpolation=2>
      <thb3Interpolation=2>
      <txInt5Interpolation=1>
      <txInputRate_kHz=204800>
      <primarySigBandwidth_Hz=75000000>
      <rfBandwidth_Hz=160000000>
      <txDac3dBCorner_kHz=187000>
      <txBbf3dBCorner_kHz=80000>
    
      <filter FIR gain_dB=0 numFirCoefs=20>
      26
      -73
      131
      -188
      232
      -209
      -31
      946
      -2961
      20082
      -2961
      946
      -31
      -209
      232
      -188
      131
      -73
      26
      0
      </filter>
     </tx>
    </profile>
    

    In terms of the LO frequencies, I have already described the process followed to recreate the problem. I jump the LO frequency randomly by 100MHz jumps between 100MHz and 400MHz. Please see the original question posted for more details.

    Thank you for providing details of the reset. This reset is being asserted by the ADRV9009 IIO Linux device driver during boot up. Please can you answer the following question: 

    - Is the ARM in the ADRV9009 reset when the external RESET pin is toggled?

    Thank you.

Children
  • Please make sure to maintain rfpllLoFreq, rfPllLoFreq - TxProfileRFBW/2 must be > 0 (DC) when you shift the LO frequency. can you please let us know exactly at what frequency you face an issue?

    yes ARM in the ADRV9009 reset when the RESET applied to the chip.

  • We have tested this and it seems independent of the frequency:

    203.92 -> 103.92

    203.92 -> 155.92

    103.92 -> 203.92

    203.92 -> 303.92

    228.92 -> 128.92

    228.92 -> 298.92

    Where it is 

    FROM LO FREQUENCY (MHz) -> TO LO FREQUENCY (MHz)

  • A correction to a previous post:
    The following statement:

    The only way to recover the ADRV9009 is to power cycle the complete unit. Rebooting the Linux does not fix the problem.

    This statement is incorrect. I have confirmed that rebooting the Linux DOES fix the problem. The ADRV9009 is reset during the Linux boot up. As you have pointed out, this will also reset the ARM. After a Linux reboot, the system returns to a fully functional state.

  • Some more information:
    We adding logging of the IIO commands to dmesg so that we can see exactly which command causes the failure. Here is a copy of the log when the failure happens:

    [ 263.651587] iio_device_find_channel(m_pIioDevAdrv9009[1], "altvoltage0", true)
    [ 264.582678] adrv9009 spi1.1: ERROR: 247: TALISE_waitArmCmdStatus() failed due to thrown ARM error. ARM time out
    [ 264.592860] adrv9009 spi1.1: adrv9009_set_radio_state: failed
    [ 264.598637] adrv9009 spi1.1: adrv9009_set_radio_state: failed
    [ 266.544511] adrv9009 spi1.1: ERROR: 179: ARM Mailbox Busy. Command not executed in TALISE_sendArmCommand()
    [ 268.494084] adrv9009 spi1.1: ERROR: 179: ARM Mailbox Busy. Command not executed in TALISE_sendArmCommand()
    [ 270.443572] adrv9009 spi1.1: ERROR: 179: ARM Mailbox Busy. Command not executed in TALISE_sendArmCommand()
    [ 270.453237] adrv9009 spi1.1: adrv9009_set_radio_state: failed
    [ 270.459015] adrv9009 spi1.1: adrv9009_set_radio_state: failed
    [ 270.464794] adrv9009 spi1.1: ERROR: 439: TALISE_setRfPllFrequency() : Invalid rfpllLoFreq, rfPllLoFreq - TxProfileRFBW/2 must be > 0 (DC)
    [ 272.488053] adrv9009 spi1.1: ERROR: 179: ARM Mailbox Busy. Command not executed in TALISE_sendArmCommand()
    [ 272.497773] iio_channel_attr_write_longlong(pIioChannel 1, "frequency", 128920000)

    The write to dmesg happens after the command has been executed. So iio_device_find_channel completes successfully and then iio_channel_attr_write_longlong triggers the failure.

    The error message Invalid "rfpllLoFreq, rfPllLoFreq - TxProfileRFBW/2 must be > 0 (DC)" doesn't make sense because the LO frequency is being set to 128.92MHz and the TX profile BW is 160MHz, so this requirement is being met.

    Please advise what I should check next.

  • I have added extra debug print-outs to the ARV9009 IIO device driver. The ARM times out when the IIO driver attempts to put the ADRV9009 into the radio off state before setting the LO frequency. I am going to close this thread because the issue is not related to the setting of the LO frequency and open a new thread relating to getting an ARM time out when attempting to put the ADRV9009 into the radio off state.