AD9371: Does Lane Latency Confirm The "Deterministic Latency" Exist Or Not?

Hi All,

I am working subclass-1 concept. I've few doubts which are quite complex :

1)  I am seeing my lane latency is keep on changing b/w "1Multiframe and 49 Octet" To "1Multiframe 62 Octet" on each power-up.

     Does it means that there is no deterministic latency ?

2) How can we calculate LMFC_OFFSET for framer and deframer ?

3)  Do we need to set same LMFC_OFFSET value in FPGA which we are using in AD9371 ?

4)  How can we calculate RBD value ?

Regards,

Manish

Parents
  • +1
    •  Analog Employees 
    on May 21, 2020 4:19 PM 2 months ago

    Hi Manish, 

    the reported latency is the actual delay in the datpath (e.g. DAC serdes/transceiver, physical delays on the route, FPGA transceiver)  per lane. As you can see this varies from power up to power up. The latency is measured from the next LMFC edge after the SYNC is de-asserted to the moment when the lane receives the ILAS sequence  (transitions from CGS to ILAS).

    Once all lanes received the ILAS sequence they are aligned(released) to the next LMFC edge. 

    In you case you have a multiframe of 128 Bytes (K*F).

    This means the CGS/ ILAS transition in your case (1 multiframe 62 Bytes) happens in the middle of the multiframe which gives you the larges margin. 

    In your case the lanes are aligned to the second multiframe so you will have deterministic latency of 2 multiframes.   

    2/3)  I don't see a reason to modify the LMFC offsets on the framer and deframer in the same time, you should adjust the one or the other. I think these are device tree properties for the FPGA link peripheral and for the 9371 device too. 

    You would need to set these only if your lane latency would  be near the the edge of the multiframe (not your case,  but e.g. 1 multiframe 127 bytes)

    or could jump to the next multiframe during another powerup. In this case the release point could jump a whole multiframe, canceling the deterministic latency. In that case you would need to shift one of the LMFC to compensate the  lane latency variations.

    4) The RBD value should be changed only if you want to reduce the latency of the link, but this will reduce the margin. 

    Suppose your lane latency is always around 1 multiframe and 32 bytes and you can guarantee the variation on the latency is smaller than +/-32 bytes, then you can move the release point from multiframe 2 to 1 multiframe and 64 bytes. So in this case the RBD can be set to 64 bytes.   

    Laszlo

  • Hi Sir,

    Thanks so much to unleash the concepts behind this.

    I understood most of the things which you have mentioned but i have still two more queries:

    >> Once all lanes received the ILAS sequence they are aligned(released) to the next LMFC edge

    Does it means that it is aligned based on RBD if RBD = 1 then its is aligned to "next LMFC edge" ?

    In my case, I have set RBD = 31 so will it release (align) on 31st LMFC edge ? 

    >> As you can see power-2-power up variance exists in lane latency and it comes around 1Multiframe 62/49/52/54 Bytes so what will be the RBD value in my case ?

    Looking forward for your reply.

    Regards,

    Manish

  • Yeah. Setting the value is a critical and a necessary task.

  • Hi @Inagy Sir,

    I would like to convey my sincere thanks to you to clarify each of the doubts.

    Regards,

    Manish

  • Hi Sir,

    1st ADI Lane Info

    root@manish:/sys/bus/platform/devices/8000c000.axi_jesd204_rx# cat lane0_info
    Errors: 0
    CGS state: DATA
    Initial Frame Synchronization: Yes
    Lane Latency: 1 Multi-frames and 55 Octets
    Initial Lane Alignment Sequence: Yes
    DID: 0, BID: 0, LID: 0, L: 2, SCR: 1, F: 4
    K: 32, M: 4, N: 16, CS: 0, N': 16, S: 1, HD: 0
    FCHK: 0x4A, CF: 0
    ADJCNT: 0, PHADJ: 0, ADJDIR: 0, JESDV: 1, SUBCLASS: 1
    FC: 4915200


    root@manish:/sys/bus/platform/devices/8000c000.axi_jesd204_rx# cat lane1_info
    Errors: 2
    CGS state: DATA
    Initial Frame Synchronization: Yes
    Lane Latency: 1 Multi-frames and 54 Octets
    Initial Lane Alignment Sequence: Yes
    DID: 0, BID: 0, LID: 1, L: 2, SCR: 1, F: 4
    K: 32, M: 4, N: 16, CS: 0, N': 16, S: 1, HD: 0
    FCHK: 0x4B, CF: 0
    ADJCNT: 0, PHADJ: 0, ADJDIR: 0, JESDV: 1, SUBCLASS: 1
    FC: 4915200


    2nd ADI Lane Info


    root@manish:/sys/bus/platform/devices/8001c000.axi_jesd204_rx# cat lane0_info
    Errors: 0
    CGS state: DATA
    Initial Frame Synchronization: Yes
    Lane Latency: 5 Multi-frames and 102 Octets
    Initial Lane Alignment Sequence: Yes
    DID: 0, BID: 0, LID: 0, L: 2, SCR: 1, F: 4
    K: 32, M: 4, N: 16, CS: 0, N': 16, S: 1, HD: 0
    FCHK: 0x4A, CF: 0
    ADJCNT: 0, PHADJ: 0, ADJDIR: 0, JESDV: 1, SUBCLASS: 1
    FC: 4915200


    root@manish:/sys/bus/platform/devices/8001c000.axi_jesd204_rx# cat lane1_info
    Errors: 0
    CGS state: DATA
    Initial Frame Synchronization: Yes
    Lane Latency: 5 Multi-frames and 105 Octets
    Initial Lane Alignment Sequence: Yes
    DID: 0, BID: 0, LID: 1, L: 2, SCR: 1, F: 4
    K: 32, M: 4, N: 16, CS: 0, N': 16, S: 1, HD: 0
    FCHK: 0x4B, CF: 0
    ADJCNT: 0, PHADJ: 0, ADJDIR: 0, JESDV: 1, SUBCLASS: 1

    SYREF_To_LMFC Delay

    root@manish:/sys/kernel/debug/iio/iio:device2# cat *lmfc*
    17 ---> Tx
    31 ---> Rx
    31 ---> RxOS



    root@manish:/sys/kernel/debug/iio/iio:device3# cat *lmfc*
    17 ---> Tx
    31 ---> Rx
    31 ---> RxOS

    Tx Side Observation:

    Query :

    a) Lane Latency is keep on changing after each ad9371_reinit. Is it right ?

    b) For subclass #1, I don't expect different lane to lane latency on each power up. Am i correct ?

    c) If Tx side obs are fixed ( Above results) then why lane latency does exist ?

    Regards,

    Manish

  • +1
    •  Analog Employees 
    on Jul 6, 2020 9:27 AM 28 days ago in reply to manish3134

    Hi Manish,

    I am trying to understand the latency differences between the two links (8000c000 vs 8001c000)    1 multiframe vs. 5 multiframes. This will cause the the two links to have different deterministic latency. The difference between these two should be less than 1 LMFC period. 

    a) the reported Lane Latency is normal to vary from one boot to another.  This is not the deterministic latency that is obtained after lane alignment, it is the measured latency per lane. 

    b) In subclass 1 you will still see the same variation in the reported latency.

    c) the latency includes also the datapath that includes serialization - deserialization, not just the control path you pointed (SYSREF and SYNC)  

    Laszlo

  • Hi Sir,

    Thanks for addressing the queries. 

    I didn't understand few things:

    >>  The difference between these two should be less than 1 LMFC period

    a) How did we reach to this as JESD204B doc doesn't talk about this. Why cannot be more than        1-LMFC ?

    b) How can we achieve this if it should be less than 1 LMFC period ? 

    >> The reported Lane Latency is normal to vary from one boot to another.  This is not the deterministic latency that is obtained after lane alignment, it is the measured latency per lane

    If lane latency can vary from boot to boot and it should not be DL but a measured latency per lane then how can we achieve DL in the system ? What are the steps of achieving this ? We have done following changes to achieve this:

    1. Sysref to LMFC delay ( Please see above table for this)

    2. Sysref is aligned across all the devices and LMFC are aligned. The same is captured in the          FPGA.

    3. Set RBD  = 0x1F

    4. Set BUFFER_EARLY_RELEASE[16] 

    >> In subclass 1 you will still see the same variation in the reported latency

    a) This means to me that lane latency variation exists in the subclass-1 and it doesn't confirm about DL ? Is it correct ?

    b) However, I am confused here because if lane latency is keep on changing then the overall latency cannot be constant all the time. On each boot, my lane latency is keep on changing which will move the release point to next multi-frame or release point cannot be constant so it will eventually affect the overall latency. Is it correct ? 

    c) How lane latency is different then Total Link Latency ?

    d) Why cannot lane latency impacting overall latency in the system ? 

    e) How can we measure that release point is keeping on changing on each power up ?

    >> the latency includes also the datapath that includes serialization - deserialization, not just the control path you pointed (SYSREF and SYNC) 

    a) How can we achieve this ? 

    b) What are the ways to determine this ? 

    Please see signals which we have captured. Please let us know what are the other signals which we can probe and proof the DL in the system.

    Regards,

    Manish

    RX_MCS_Signals_Capture.PNG

    TX_MCS_Signals_Capture.PNG

Reply
  • Hi Sir,

    Thanks for addressing the queries. 

    I didn't understand few things:

    >>  The difference between these two should be less than 1 LMFC period

    a) How did we reach to this as JESD204B doc doesn't talk about this. Why cannot be more than        1-LMFC ?

    b) How can we achieve this if it should be less than 1 LMFC period ? 

    >> The reported Lane Latency is normal to vary from one boot to another.  This is not the deterministic latency that is obtained after lane alignment, it is the measured latency per lane

    If lane latency can vary from boot to boot and it should not be DL but a measured latency per lane then how can we achieve DL in the system ? What are the steps of achieving this ? We have done following changes to achieve this:

    1. Sysref to LMFC delay ( Please see above table for this)

    2. Sysref is aligned across all the devices and LMFC are aligned. The same is captured in the          FPGA.

    3. Set RBD  = 0x1F

    4. Set BUFFER_EARLY_RELEASE[16] 

    >> In subclass 1 you will still see the same variation in the reported latency

    a) This means to me that lane latency variation exists in the subclass-1 and it doesn't confirm about DL ? Is it correct ?

    b) However, I am confused here because if lane latency is keep on changing then the overall latency cannot be constant all the time. On each boot, my lane latency is keep on changing which will move the release point to next multi-frame or release point cannot be constant so it will eventually affect the overall latency. Is it correct ? 

    c) How lane latency is different then Total Link Latency ?

    d) Why cannot lane latency impacting overall latency in the system ? 

    e) How can we measure that release point is keeping on changing on each power up ?

    >> the latency includes also the datapath that includes serialization - deserialization, not just the control path you pointed (SYSREF and SYNC) 

    a) How can we achieve this ? 

    b) What are the ways to determine this ? 

    Please see signals which we have captured. Please let us know what are the other signals which we can probe and proof the DL in the system.

    Regards,

    Manish

    RX_MCS_Signals_Capture.PNG

    TX_MCS_Signals_Capture.PNG

Children
  • +1
    •  Analog Employees 
    on Jul 13, 2020 8:44 AM 21 days ago in reply to manish3134

    Hi Manish, 

    Your setup is different from a standard multipoint link the standard refers to.  Since you have two link receive peripherals for Rx (and other paths obs,tx), these act as independent links which can result in different deterministic latency values if the skew between links is larger than 1 LMFC(your case).

    The solution would be to use a single link receive peripheral with NUM_LINKS set to 2. Connect all sync and lanes to that.  Do the same also for RxObs and Tx. 

    So you would have a peripheral for Rx, one for RxObs and one for Tx.  (3 instead of 6)

    In this case even if there are different latencies between ADC1-FPGA and ADC2-FPGA paths, this will be evened out by the link receive peripheral, ensuring DL. 

    This would likely solve also the SYSREF alignment error you are seeing. 

    Also the number of transport layer component can be reduced by 2 with the same logic.    Just you need to double the number of converters and number of lanes for each TPL. 

    You can measure the synchronization between channels by injecting the same signal into the two parts and looking at the data inside the FPGA or application level with IIO Scope. 

    Laszlo

  • Hi Sir,

    I am working on RRH application where we have two ad9371 on board sourced with single AD9528 clock distribution module. The task is to use one channel of one AD9371 and one channel of second ADI to perform MIMO. 

    Hopefully, this will make our queries more clear to you. 

    Few Observations So Far:

    1) My Device_Clk is 122.88MHz when i am seeing on AD9528 while what i am seeing by checking JESD status is 122.890 MHz which says that 10KHz variation. Is this problematic ? I am not able to visualize the issue. What i see is that LMFC will be different for example

    LaneRate_1 = (IQ Sample Rate * M * 16 * (10/8))/L

                         = (122.880*4*116*1.25)/2 = 4.9152MHz

    LaneRate_2 = (122.890*4*16*1.25)/2 = 4.9156MHz

    LMFC_1 = (LaneRate_1) / (K*F*10) = 4.9152/(32*2*10) = 7.68MHz

    LMFC_2 = (LaneRate_2) / (K*F*10) = 4.9156/(32*2*10) = 7.680625 MHz

    SYSREF is 3.072MHz which is integer multiple of LMFC_1 but not LMFC_2 ?

    2) We provided common SYNC from ADI-CORE-DAC to AD9371 Deframer. Is it correct ?

    3) We provided common SYNC from AD9371 Framer to ADI-Core-ADC. Is it correct ? 

    4) We didn't implement RF LO Sync on the board. Is it required ? 

    As you have pointed out that my measured lane latency is different on each boot and it is not < 1LMFC w.r.t. each other.

    Kindly put some light on "How can achieve constant lane latency on each boot ?"

    Regards,

    Manish

  • 0
    •  Analog Employees 
    on Jul 15, 2020 9:20 AM 19 days ago in reply to manish3134

    Hi Manish, 

    See below a proposed block diagram: 

    A similar project can be found here: https://github.com/analogdevicesinc/hdl/blob/master/projects/fmcomms8/common/fmcomms8_bd.tcl

    The difference is that uses ADRV9009 , but the architecture, number of lanes and convertes is similar. 

    It uses a different approach connecting the SYNC signal (not using NUM_LINKS = 2) but you will get the same result either way. 

    "We didn't implement RF LO Sync on the board. Is it required ? "

    For deterministic latency, the DEV_CLK/SYSREF pair is used, that should be sourced from the AD9528.

    Thank you,

    Laszlo

  • Hi Sir,

    We have used below approach to implement this where number of lanes and converters are similar.

    1. Is the above approach fine  ?

    2. My measured device clock is mentioned below while i am giving 122.880MHz from AD9528 --> FPGA & AD9528 --> AD9371 so "Is the variation which is mentioned below holds good ?"

    The clock distribution source which is AD9528 is a single alone in the design.

    Measured Link Clock: 122.888 MHz

    Reported Link Clock: 0.000 Mhz

    Lane rate: 4915.200 MHz

    Lane rate / 40: 122.880 MHz

    LMFC rate: 3.840 MHz

    Measured Link Clock: 122.890 MHz

    Reported Link Clock: 0.000 MHz

    Lane rate: 4915.200 MHz

    Lane rate / 40: 122.880 MHz

    I have few observation which i am sharing below.

    Initially, We were testing two AD9371 separately which means the SYNC signals were not ANDED and I wasn't seeing so much variation in the lane latency. It was fluctuating from 1MF + 44 Octets to 1 MF + 90 Octets which gave me enough margin to release on the next LMFC boundary. It was not crossing the next Multi-frame boundary on the multiple reboot.

    Objective:

    Now, We want to use one channel of 1st AD9371 & one channel of 2nd Ad9371 to test MIMO. For this,  We've ANDED the SYNC signal from FPGA and feed it into two AD9371 and vice versa. Now, the lane latency is keep on changing on each AD9371. In this case, its an issues as it affects the total latency of the system.  How can we fix this issue ?

    Regards,

    Manish

  • +1
    •  Analog Employees 
    on Jul 16, 2020 2:41 PM 18 days ago in reply to manish3134

    Hi Manish,

    The way I would do it is with one set of IPs not with two  ( see my previous post).  Now I see why you got the difference in the reported latencies between links. This is a reporting issue and the actual latency on that link is not that big. ( 4 multiframes ... ) it is actually 1 multiframe and few bytes as you measured without merging the SYNC signals.

    AND-ing the two SYNCs will help synchronize the two links, but it has the side effect you see on the reported latencies. 

    The reported latencies are measured from the deassertion of SYNC (0->1) to the start of ILA phase,  in your case this happens in each link receive peripheral.  Since you AND-ed the two syncs the ILA phase will happen only if all cores deassert SYNC, but in case one link is enabled later most certainly will move the SYNC deassertion.   

    By looking again to your waves looks that ila0_rx_valid and ila0_rx_valid_0 happen in the same time, meaning the links are synchronized.

      

    Now, the lane latency is keep on changing on each AD9371. In this case, its an issues as it affects the total latency of the system. 

    The actual latency in this case is the smaller value from the two links. The large value is a reporting problem which I tired to explain above. 

    My measured device clock is mentioned below while i am giving 122.880MHz from AD9528 --> FPGA & AD9528 --> AD9371 so "Is the variation which is mentioned below holds good ?"

    The measured device clock is an approximation, the measurement is done with a 100MHz clock so I would expect some variation there. 

    Thanks,

    Laszlo