AD9371: Does Lane Latency Confirm The "Deterministic Latency" Exist Or Not?

Hi Manish, 

the reported latency is the actual delay in the datpath (e.g. DAC serdes/transceiver, physical delays on the route, FPGA transceiver)  per lane. As you can see this varies from power up to power up. The latency is measured from the next LMFC edge after the SYNC is de-asserted to the moment when the lane receives the ILAS sequence  (transitions from CGS to ILAS).

Once all lanes received the ILAS sequence they are aligned(released) to the next LMFC edge. 

In you case you have a multiframe of 128 Bytes (K*F).

This means the CGS/ ILAS transition in your case (1 multiframe 62 Bytes) happens in the middle of the multiframe which gives you the larges margin. 

In your case the lanes are aligned to the second multiframe so you will have deterministic latency of 2 multiframes.   

2/3)  I don't see a reason to modify the LMFC offsets on the framer and deframer in the same time, you should adjust the one or the other. I think these are device tree properties for the FPGA link peripheral and for the 9371 device too. 

You would need to set these only if your lane latency would  be near the the edge of the multiframe (not your case,  but e.g. 1 multiframe 127 bytes)

or could jump to the next multiframe during another powerup. In this case the release point could jump a whole multiframe, canceling the deterministic latency. In that case you would need to shift one of the LMFC to compensate the  lane latency variations.

4) The RBD value should be changed only if you want to reduce the latency of the link, but this will reduce the margin. 

Suppose your lane latency is always around 1 multiframe and 32 bytes and you can guarantee the variation on the latency is smaller than +/-32 bytes, then you can move the release point from multiframe 2 to 1 multiframe and 64 bytes. So in this case the RBD can be set to 64 bytes.   

Laszlo

Hi Sir,

Thanks so much to unleash the concepts behind this.

I understood most of the things which you have mentioned but i have still two more queries:

>> Once all lanes received the ILAS sequence they are aligned(released) to the next LMFC edge

Does it means that it is aligned based on RBD if RBD = 1 then its is aligned to "next LMFC edge" ?

In my case, I have set RBD = 31 so will it release (align) on 31st LMFC edge ? 

>> As you can see power-2-power up variance exists in lane latency and it comes around 1Multiframe 62/49/52/54 Bytes so what will be the RBD value in my case ?

Looking forward for your reply.

Regards,

Manish

Yeah. Setting the value is a critical and a necessary task.

Hi @Inagy Sir,

I would like to convey my sincere thanks to you to clarify each of the doubts.

Regards,

Manish

Hi Sir,

1st ADI Lane Info

root@manish:/sys/bus/platform/devices/8000c000.axi_jesd204_rx# cat lane0_info
Errors: 0
CGS state: DATA
Initial Frame Synchronization: Yes
Lane Latency: 1 Multi-frames and 55 Octets
Initial Lane Alignment Sequence: Yes
DID: 0, BID: 0, LID: 0, L: 2, SCR: 1, F: 4
K: 32, M: 4, N: 16, CS: 0, N': 16, S: 1, HD: 0
FCHK: 0x4A, CF: 0
ADJCNT: 0, PHADJ: 0, ADJDIR: 0, JESDV: 1, SUBCLASS: 1
FC: 4915200

root@manish:/sys/bus/platform/devices/8000c000.axi_jesd204_rx# cat lane1_info
Errors: 2
CGS state: DATA
Initial Frame Synchronization: Yes
Lane Latency: 1 Multi-frames and 54 Octets
Initial Lane Alignment Sequence: Yes
DID: 0, BID: 0, LID: 1, L: 2, SCR: 1, F: 4
K: 32, M: 4, N: 16, CS: 0, N': 16, S: 1, HD: 0
FCHK: 0x4B, CF: 0
ADJCNT: 0, PHADJ: 0, ADJDIR: 0, JESDV: 1, SUBCLASS: 1
FC: 4915200

2nd ADI Lane Info

root@manish:/sys/bus/platform/devices/8001c000.axi_jesd204_rx# cat lane0_info
Errors: 0
CGS state: DATA
Initial Frame Synchronization: Yes
Lane Latency: 5 Multi-frames and 102 Octets
Initial Lane Alignment Sequence: Yes
DID: 0, BID: 0, LID: 0, L: 2, SCR: 1, F: 4
K: 32, M: 4, N: 16, CS: 0, N': 16, S: 1, HD: 0
FCHK: 0x4A, CF: 0
ADJCNT: 0, PHADJ: 0, ADJDIR: 0, JESDV: 1, SUBCLASS: 1
FC: 4915200

root@manish:/sys/bus/platform/devices/8001c000.axi_jesd204_rx# cat lane1_info
Errors: 0
CGS state: DATA
Initial Frame Synchronization: Yes
Lane Latency: 5 Multi-frames and 105 Octets
Initial Lane Alignment Sequence: Yes
DID: 0, BID: 0, LID: 1, L: 2, SCR: 1, F: 4
K: 32, M: 4, N: 16, CS: 0, N': 16, S: 1, HD: 0
FCHK: 0x4B, CF: 0
ADJCNT: 0, PHADJ: 0, ADJDIR: 0, JESDV: 1, SUBCLASS: 1

SYREF_To_LMFC Delay

root@manish:/sys/kernel/debug/iio/iio:device2# cat *lmfc*
17 ---> Tx
31 ---> Rx
31 ---> RxOS

root@manish:/sys/kernel/debug/iio/iio:device3# cat *lmfc*
17 ---> Tx
31 ---> Rx
31 ---> RxOS

Tx Side Observation:

Query :

a) Lane Latency is keep on changing after each ad9371_reinit. Is it right ?

b) For subclass #1, I don't expect different lane to lane latency on each power up. Am i correct ?

c) If Tx side obs are fixed ( Above results) then why lane latency does exist ?

Regards,

Manish

Hi Manish,

I am trying to understand the latency differences between the two links (8000c000 vs 8001c000)    1 multiframe vs. 5 multiframes. This will cause the the two links to have different deterministic latency. The difference between these two should be less than 1 LMFC period. 

a) the reported Lane Latency is normal to vary from one boot to another.  This is not the deterministic latency that is obtained after lane alignment, it is the measured latency per lane. 

b) In subclass 1 you will still see the same variation in the reported latency.

c) the latency includes also the datapath that includes serialization - deserialization, not just the control path you pointed (SYSREF and SYNC)  

Laszlo

Hi Sir,

Thanks for addressing the queries. 

I didn't understand few things:

>>  The difference between these two should be less than 1 LMFC period

a) How did we reach to this as JESD204B doc doesn't talk about this. Why cannot be more than        1-LMFC ?

b) How can we achieve this if it should be less than 1 LMFC period ? 

>> The reported Lane Latency is normal to vary from one boot to another.  This is not the deterministic latency that is obtained after lane alignment, it is the measured latency per lane

If lane latency can vary from boot to boot and it should not be DL but a measured latency per lane then how can we achieve DL in the system ? What are the steps of achieving this ? We have done following changes to achieve this:

1. Sysref to LMFC delay ( Please see above table for this)

2. Sysref is aligned across all the devices and LMFC are aligned. The same is captured in the          FPGA.

3. Set RBD  = 0x1F

4. Set BUFFER_EARLY_RELEASE[16] 

>> In subclass 1 you will still see the same variation in the reported latency

a) This means to me that lane latency variation exists in the subclass-1 and it doesn't confirm about DL ? Is it correct ?

b) However, I am confused here because if lane latency is keep on changing then the overall latency cannot be constant all the time. On each boot, my lane latency is keep on changing which will move the release point to next multi-frame or release point cannot be constant so it will eventually affect the overall latency. Is it correct ? 

c) How lane latency is different then Total Link Latency ?

d) Why cannot lane latency impacting overall latency in the system ? 

e) How can we measure that release point is keeping on changing on each power up ?

>> the latency includes also the datapath that includes serialization - deserialization, not just the control path you pointed (SYSREF and SYNC) 

a) How can we achieve this ? 

b) What are the ways to determine this ? 

Please see signals which we have captured. Please let us know what are the other signals which we can probe and proof the DL in the system.

Regards,

Manish

Hi Manish, 

Your setup is different from a standard multipoint link the standard refers to.  Since you have two link receive peripherals for Rx (and other paths obs,tx), these act as independent links which can result in different deterministic latency values if the skew between links is larger than 1 LMFC(your case).

The solution would be to use a single link receive peripheral with NUM_LINKS set to 2. Connect all sync and lanes to that.  Do the same also for RxObs and Tx. 

So you would have a peripheral for Rx, one for RxObs and one for Tx.  (3 instead of 6)

In this case even if there are different latencies between ADC1-FPGA and ADC2-FPGA paths, this will be evened out by the link receive peripheral, ensuring DL. 

This would likely solve also the SYSREF alignment error you are seeing. 

Also the number of transport layer component can be reduced by 2 with the same logic.    Just you need to double the number of converters and number of lanes for each TPL. 

You can measure the synchronization between channels by injecting the same signal into the two parts and looking at the data inside the FPGA or application level with IIO Scope. 

Laszlo

Hi Sir,

I am working on RRH application where we have two ad9371 on board sourced with single AD9528 clock distribution module. The task is to use one channel of one AD9371 and one channel of second ADI to perform MIMO. 

Hopefully, this will make our queries more clear to you. 

Few Observations So Far:

1) My Device_Clk is 122.88MHz when i am seeing on AD9528 while what i am seeing by checking JESD status is 122.890 MHz which says that 10KHz variation. Is this problematic ? I am not able to visualize the issue. What i see is that LMFC will be different for example

LaneRate_1 = (IQ Sample Rate * M * 16 * (10/8))/L

                     = (122.880*4*116*1.25)/2 = 4.9152MHz

LaneRate_2 = (122.890*4*16*1.25)/2 = 4.9156MHz

LMFC_1 = (LaneRate_1) / (K*F*10) = 4.9152/(32*2*10) = 7.68MHz

LMFC_2 = (LaneRate_2) / (K*F*10) = 4.9156/(32*2*10) = 7.680625 MHz

SYSREF is 3.072MHz which is integer multiple of LMFC_1 but not LMFC_2 ?

2) We provided common SYNC from ADI-CORE-DAC to AD9371 Deframer. Is it correct ?

3) We provided common SYNC from AD9371 Framer to ADI-Core-ADC. Is it correct ? 

4) We didn't implement RF LO Sync on the board. Is it required ? 

As you have pointed out that my measured lane latency is different on each boot and it is not < 1LMFC w.r.t. each other.

Kindly put some light on "How can achieve constant lane latency on each boot ?"

Regards,

Manish

Hi Manish, 

See below a proposed block diagram: 

A similar project can be found here: https://github.com/analogdevicesinc/hdl/blob/master/projects/fmcomms8/common/fmcomms8_bd.tcl

The difference is that uses ADRV9009 , but the architecture, number of lanes and convertes is similar. 

It uses a different approach connecting the SYNC signal (not using NUM_LINKS = 2) but you will get the same result either way. 

"We didn't implement RF LO Sync on the board. Is it required ? "

For deterministic latency, the DEV_CLK/SYSREF pair is used, that should be sourced from the AD9528.

Thank you,

Laszlo

Hi Sir,

We have used below approach to implement this where number of lanes and converters are similar.

1. Is the above approach fine  ?

2. My measured device clock is mentioned below while i am giving 122.880MHz from AD9528 --> FPGA & AD9528 --> AD9371 so "Is the variation which is mentioned below holds good ?"

The clock distribution source which is AD9528 is a single alone in the design.

Measured Link Clock: 122.888 MHz

Reported Link Clock: 0.000 Mhz

Lane rate: 4915.200 MHz

Lane rate / 40: 122.880 MHz

LMFC rate: 3.840 MHz

Measured Link Clock: 122.890 MHz

Reported Link Clock: 0.000 MHz

Lane rate: 4915.200 MHz

Lane rate / 40: 122.880 MHz

I have few observation which i am sharing below.

Initially, We were testing two AD9371 separately which means the SYNC signals were not ANDED and I wasn't seeing so much variation in the lane latency. It was fluctuating from 1MF + 44 Octets to 1 MF + 90 Octets which gave me enough margin to release on the next LMFC boundary. It was not crossing the next Multi-frame boundary on the multiple reboot.

Objective:

Now, We want to use one channel of 1st AD9371 & one channel of 2nd Ad9371 to test MIMO. For this,  We've ANDED the SYNC signal from FPGA and feed it into two AD9371 and vice versa. Now, the lane latency is keep on changing on each AD9371. In this case, its an issues as it affects the total latency of the system.  How can we fix this issue ?

Regards,

Manish

Hi Manish,

The way I would do it is with one set of IPs not with two  ( see my previous post).  Now I see why you got the difference in the reported latencies between links. This is a reporting issue and the actual latency on that link is not that big. ( 4 multiframes ... ) it is actually 1 multiframe and few bytes as you measured without merging the SYNC signals.

AND-ing the two SYNCs will help synchronize the two links, but it has the side effect you see on the reported latencies. 

The reported latencies are measured from the deassertion of SYNC (0->1) to the start of ILA phase,  in your case this happens in each link receive peripheral.  Since you AND-ed the two syncs the ILA phase will happen only if all cores deassert SYNC, but in case one link is enabled later most certainly will move the SYNC deassertion.   

By looking again to your waves looks that ila0_rx_valid and ila0_rx_valid_0 happen in the same time, meaning the links are synchronized.

MKS3134 said:

Now, the lane latency is keep on changing on each AD9371. In this case, its an issues as it affects the total latency of the system. 

The actual latency in this case is the smaller value from the two links. The large value is a reporting problem which I tired to explain above. 

MKS3134 said:

My measured device clock is mentioned below while i am giving 122.880MHz from AD9528 --> FPGA & AD9528 --> AD9371 so "Is the variation which is mentioned below holds good ?"

The measured device clock is an approximation, the measurement is done with a 100MHz clock so I would expect some variation there. 

Thanks,

Laszlo

Hi Manish,

The way I would do it is with one set of IPs not with two  ( see my previous post).  Now I see why you got the difference in the reported latencies between links. This is a reporting issue and the actual latency on that link is not that big. ( 4 multiframes ... ) it is actually 1 multiframe and few bytes as you measured without merging the SYNC signals.

AND-ing the two SYNCs will help synchronize the two links, but it has the side effect you see on the reported latencies. 

The reported latencies are measured from the deassertion of SYNC (0->1) to the start of ILA phase,  in your case this happens in each link receive peripheral.  Since you AND-ed the two syncs the ILA phase will happen only if all cores deassert SYNC, but in case one link is enabled later most certainly will move the SYNC deassertion.   

By looking again to your waves looks that ila0_rx_valid and ila0_rx_valid_0 happen in the same time, meaning the links are synchronized.

MKS3134 said:

Now, the lane latency is keep on changing on each AD9371. In this case, its an issues as it affects the total latency of the system. 

The actual latency in this case is the smaller value from the two links. The large value is a reporting problem which I tired to explain above. 

MKS3134 said:

My measured device clock is mentioned below while i am giving 122.880MHz from AD9528 --> FPGA & AD9528 --> AD9371 so "Is the variation which is mentioned below holds good ?"

The measured device clock is an approximation, the measurement is done with a 100MHz clock so I would expect some variation there. 

Thanks,

Laszlo

Hi Sir,

Thanks so much Sir for sharing your time to address my questions.

Let me explain about the reason of doing ANDED signal.

"I was assuming that if I perform lane to lane synchronization" or " the ILAS phase on all the transmitters should start @ same time" then, Two AD9371 were supposed to  transmit @ same time.  However, the latency variation is huge like 127Octets to 127MF+127Octet. Is this a design flaw ? Could you please put some more light on this topic as its less understood by me sir ?

Few Queries:

a) Is LaneToLane SYNC one of the key thing in MIMO system ? If yes, Could you please briefly explain ?

b) Is ANDED SYNC a FLAW or DESIGN ISSUE ?

Device Clock Measurement Issue

Sir,  The reason of asking this is that "the system is showing SYSREF ALIGNMENT ERROR" which is because "SYSREF is not integer multiple of LMFC". If there is slight variation in DEVICE CLOCK then SYSREF cannot be integer multiple of LMFC.  Sir, below is the calculation which i did to prove it. Kindly look @ once.

LaneRate_1 = (IQ Sample Rate * M * 16 * (10/8))/L

                     = (122.880 * 4 * 16 * 1.25) / 2 = 4.9152MHz

LaneRate_2 = (122.890 * 4 * 16 * 1.25) / 2 = 4.9156MHz

LMFC_1 = (LaneRate_1) / (K*F*10) = 4.9152/(32*2*10) = 7.68MHz

LMFC_2 = (LaneRate_2) / (K*F*10) = 4.9156/(32*2*10) = 7.680625 MHz

SYSREF is 30.72KHz which is integer multiple of LMFC_1 but not LMFC_2 ?

So slight variation in DEVICE_CLK causes "SYSREF ALIGNMENT ERROR" ?

I read JESD status where "Measured Clock is calculated" ( axi_jesd204_rx.c : LINE :155) .

Sir, I know that this is not the driver forum but if you could address this too, it will be a great help. 

Regards,

Manish

Having two Rx peripherals (one for each 9317) requires the use of AND on the SYNC signal. Other case they behave as two independent links. But this is a workaround, and the side effect is the incorrectly reported long latency on one of the links ( 1 multiframe vs 4 multiframe in your case).

The proper way to do it is through a common Rx peripheral with dual link. 

At the end of the day it works either way, you should have the links synchronized and have deterministic latency even with the current design.

The reported link clock frequency is not related to the sysref alignment error. 

This likely is caused by setup/hold  violation on one of the paths. 

Laszlo

Thanks so much Sir.

I am highly obliged by your help. You put everything in very lucid way and put light on this topic which is very complex for beginners like me.

Thanks Sir.

Regards,

Manish

Hi Laszlo Sir,

With your inputs, I have competed my observations. Please see my observation below and i am seeking your comment on that.

1) lane to lane latency lies in the middle of 1 and 2 MF

2) AD9371 Deframer FIFO Depth comes 7 or 9 while i was expecting 60-64 as  it says that Read/Write pointer should never overlap so if 0<fifio>127 is the range then it must come into the middle to have a good margin so that in worst case also, it will never overlap. I found a link below.

 https://ez.analog.com/wide-band-rf-transceivers/design-support-ad9371/f/q-a/103512/ad9371-deframer-deterministic-fifo-size

3) I am using Analog IP. What will be the value of the LMFC Offset in axi_jesd204_rx.c & axi_jesd204_tx.c ?

   I found a link below where they are talking about LMFC offset but I don't know that what value must be set in these drivers ?

https://wiki.analog.com/resources/fpga/peripherals/jesd204/axi_jesd204_rx

https://wiki.analog.com/resources/fpga/peripherals/jesd204/axi_jesd204_tx

Please see my setting below. 

        3.a) However, In this case , I am seeing "Sysref Alignment Error" and AD9371 Deframer FIFO       

               depth is 11.

      3.b)  However, I didn't see "Sysref Alignment Error" in case below.  AD9371 Deframer FIFO depth is

              7 or 9 .

I am awaiting for your valuable comments.

Regards,

Manish

Hi Manish, 

1. Discussing the FPGA(Tx) -> AD9371(Rx) data path : 

in the axi_jesd204_tx you can adjust the LMFC relative to the SYSREF specified in octets. 

The valid range is 0 .. K*F-1   ;  in your case 0 .. 32*2-1 ; 0..63 octets   

However from your experiment it looks the fifo depth is reported in my assumption PCLK cycles which is usually Lane Rate/40 or 4 octets per clock cycle 

Taking this in consideration a valid range for the fifo depth translates to 0 .. 15 , so having 7 or 9 looks reasonable to me.

I did not find in the documentation the unit of the fifo depth, please post a question on the forum you mentioned just to double check. 

2.  The SYSREF alignment error is more likely a setup/hold problem.

Try shifting the phase of the SYSREF signal from your clock chip,  e.g https://github.com/analogdevicesinc/linux/blob/master/arch/arm/boot/dts/adi-adrv9371.dtsi#L74

Laszlo

Hi Laszlo Sir,

Thanks so much for the reply.

>> Taking this in consideration a valid range for the fifo depth translates to 0 .. 15 , so having 7 or 9 looks reasonable to me

[Manish]: I couldn't understand that how did you translates it to 0 -15 and converge into a conclusion that 7-9 was a reasonable.  I was seeing that uint8_t fifoDepth in ad9371 driver so i was expecting the range 0 -128 so for all the lanes, it must be in the middle to have proper margin between read and write pointer.

>> Try shifting the phase of the SYSREF signal from your clock chip

[Manish]: I did it but no improvement so far. 

Configuration

1. Setting lmfcOffset = 0  in axi_jesd204_rx.c and axi_jesd204_tx.c

2. Setting lmfcOffset = 31 in Framer and lmfcOffset = 17 in Deframer of Ad9371 ( I didn't try other non-zero values)

By above two configurations, I am not seeing "Sysref Alignement Error".   Why doesn't it show the error ? 

How can it be possible that configuration of the lmfcOffset affects "Sysref Alignement" ?

Regards,

Manish

From your previous post I understood that:

for axi_jesd204_tx  LMFC offset = 17      the    AD9371 Deframer FIFO  depth is 11

for axi_jesd204_tx  LMFC offset = 0         the   AD9371 Deframer FIFO depth is   7 or 9 .

so a axi_jesd204_tx difference of 17 octest (is actually 16 since is rounded to multiple of 4)  translates to a difference of 11-9 .. 11-7     2..4 units 

The LMFC offset from axi_jesd204_tx should affect directly the deframer FIFO depth,  so the units of the LMFC offset does not match with the unit of the reported FIFO depth.   

Since the unit of LMFC offset of axi_jesd204_tx is octest, the FIFO depth unit  cannot be octets. I think is octest x 4 ; 

As you can see in the linux driver the depth is calculated from read and write pointers. 

https://github.com/analogdevicesinc/linux/blob/master/drivers/iio/adc/mykonos/mykonos.c#L10710

These pointers unlikely to have their units in octets. 

Please post a question on the following forum to confirm this: https://ez.analog.com/wide-band-rf-transceivers/design-support-ad9371

So the valid range is 0 .. K*F-1   ;  in your case 0 .. 32*2-1 ; 0..63 octets translates to  a range of 0..15    

Laszlo