Lane Latency

The complete MCS is a 12 step sequence, which must be performed at all to sync devices in parallel.

https://github.com/analogdevicesinc/linux_image_ADI-scripts/blob/master/adrv9009_multichip_sync.sh

Tx latency must be read from the device itself, IIRC there is no API for that.

But how can I have deterministic latency with a single device?

In my test setup, I have loop-back from Tx port to the Rx.
I want to decimate my Rx stream with factor of 12. The alignment is done to the sysref puls. This would only be correct if the lane latency is fixed or known.

Could it be fixed? Then the remaining error would be an offset.

If I know what the Tx and Rx latency is, I could compensate this in hdl. On which forum, shall I ask the question about the Tx latency?

In the default ADRV9009 design, we are using for each RX/TX/OBS a device clock generated from the same reference clock through an MMCM. Because of this, the phase between SYSREF and the generated device clock is not deterministic and you can have a +-1 sample variation. We implemented it this way because we only have a single reference clock and three links and we wanted to be able to independently change the lane rates.

You could change the design to a fixed profile and if the single clock and sysref are enough for that profile, then you could constrain correctly the HDL and have full deterministic latency.

Regards,

Adrian

Hdl input pins: "tx_ref_clk0", "rx_ref_clk0","rx_ref_clk2" are in my design all 240MHz.
The output of "axi_adrv9009_rx_clkgen" and "axi_adrv9009_os_clkgen" are 240MHz.
The output of "axi_adrv9009_tx_clkgen" is 120MHz.

To have full deterministic latency I need to remove all mmcm? Shall I remove all mmcm: axi_adrv9009_rx_clkgen,axi_adrv9009_tx_clkgen and axi_adrv9009_os_clkgen.

Which clocks shall I use instead? Bypass those mmcm, are use the hdl input pins instead?

The MMCM should be removed from the paths you want deterministic latency. If all three paths need it, then all three should be removed. 

We are using a single clock, ref_clk1 to drive tx_ref_clk and rx_ref_clk_x. It would be better to use the ODIV2 output as a device clock to be able to properly constrain it. You'll probably need to run all three paths at the 240MHz or find a way to divide it and have deterministic phase on the divided path.

Regarding the lane latency, in subclass 1 it doesn't necessary matter if you have +- octets delay difference between lanes, as the elastic buffer will be released on the next LMFC for both lanes.

When performing the changes, the device tree probably needs to be configured accordingly.

Regards,

Adrian

I have removed the mmcm. And clocks are driven from the odiv2.

Tx clock is 120 MHz (ref_clk_bufg_gt_div2)

Rx clock is 240 MHz (ref_clk_bufg_gt_div1)

like this:

  i_ibufds_ref_clk1 (
    .CEB (1'd0),
    .I (ref_clk1_p),
    .IB (ref_clk1_n),
    .O (ref_clk1),
    .ODIV2 (ref_clk1_odiv)
  );
  
  BUFG_GT i_bufg_div1 (
    .CE (1'd1),
    .CEMASK(1'd0),
    .CLR(1'd0),
    .CLRMASK(1'd0),
    .DIV(3'd0),
    .I(ref_clk1_odiv),
    .O(ref_clk_bufg_gt_div1
    ));

  BUFG_GT i_bufg_div2 (
    .CE (1'd1),
    .CEMASK(1'd0),
    .CLR(1'd0),
    .CLRMASK(1'd0),
    .DIV(3'd1),
    .I(ref_clk1_odiv),
    .O(ref_clk_bufg_gt_div2
    ));

The octets in the lane latency are still variating from 102 to 109 octets. Did I understand correctly that this has noting to do with the deterministic latency? And that this variation will only be a problem if the lane latency has more then 1 multi-frame latency?

In this configuration the Tx is still half the speed of Rx. This will result in ambiguity of one clock in the Tx path? How can I change this? Can I do it in the build script?

Or do I have to change it manualy in dtb:
   adi,jesd204-deframer-a-deserializer-lanes-enabled = <0x0F>;
to :        
   adi,jesd204-deframer-a-deserializer-lanes-enabled = <0x03>;

And ignore the path tx_phy2 and tx_phy3?

The latency would be per link. For having the same link configuration for TX I would change: https://github.com/analogdevicesinc/hdl/blob/master/projects/adrv9009/common/adrv9009_bd.tcl#L3 to 2, and the device tree configuration, but this will limit your TX maximum bandwidth.

I tried to build this whit this setting, and I get this error:

ERROR: [BD 5-106] Arguments to the connect_bd_intf_net command cannot be empty.
ERROR: [Common 17-39] 'connect_bd_intf_net' failed due to earlier errors.

I started with the 2019_r1 release. Or should I use another one?

AdrianC said:

but this will limit your TX maximum bandwidth.

Is this about doubling the sampling rate. And use only one channel instead of two?

I will try the configuration on my side and let you know what other changes need to be done. 

If you change the number of channels you'll have different link configuration which may change the link behavior. 

Hello,

I've added some changes in the HDL at https://github.com/analogdevicesinc/hdl/tree/adrv9009_zcu102_2_tx_lanes but still need to create a devicetree and test on linux.

Regards,
Adrian

I have build the hdl successfully! Thanks for that.

The updates I made in the dtb's so far:

The  adi,sys-clk-select  set to "0" (same as axi_adrv9009_adxcvr_rx)

        axi_adrv9009_adxcvr_tx: axi-adxcvr-tx@84a80000 {
            #address-cells = <1>;
            #size-cells = <0>;
            compatible = "adi,axi-adxcvr-1.0";
            reg = <0x84a80000 0x1000>;

            clocks = <&clk0_ad9528 1>, <&axi_tx_clkgen>;
            clock-names = "conv", "div40";

            #clock-cells = <1>;
            clock-output-names = "tx_gt_clk", "tx_out_clk";

            adi,sys-clk-select = <0>;
            adi,out-clk-select = <3>;
        };

Selected the corresponding lanes: 0x0f to 0x03

    adi,jesd204-deframer-a-deserializer-lanes-enabled = <0x03>;

When I read out my jesd status I observed the following things.

• Link status tx: CGS
• Link status rx: DATA
• I expected for the same lane rates (Tx,Rx) , that the LMFC rate would be the same as well.  But the tx LMFC rate is twice as fast as the rx LMFC rate.

Maybe following condition is violated:

• The CGS phase has lasted for at least the configured minimum CGS duration (1 frame + 9 octets by default). (from: https://wiki.analog.com/resources/fpga/peripherals/jesd204/axi_jesd204_tx)
  

Hello,

In my commit I've activated lane 0 and lane 3 from the ADRV9009 perspective, so the parameter should be:

+ adi,jesd204-deframer-a-deserializer-lanes-enabled = <0x09>;

In arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev10-adrv9009.dts:

For L=2, M=4, the octets per frame changes to 4:

@@ -172,7 +172,7 @@
#clock-cells = <0>;
clock-output-names = "jesd_tx_lane_clk";

- adi,octets-per-frame = <2>;
+ adi,octets-per-frame = <4>;

The buswidth of the TX DMA also changes:

@@ -106,7 +106,7 @@
reg = <0>;
adi,source-bus-width = <128>;
adi,source-bus-type = <0>;
- adi,destination-bus-width = <128>;
+ adi,destination-bus-width = <64>;

You'll probably need to generate a whole new list of profiles for your use case from the ADRV9009 TES.

Regards,

Adrian

Hello,

In my commit I've activated lane 0 and lane 3 from the ADRV9009 perspective, so the parameter should be:

+ adi,jesd204-deframer-a-deserializer-lanes-enabled = <0x09>;

In arch/arm64/boot/dts/xilinx/zynqmp-zcu102-rev10-adrv9009.dts:

For L=2, M=4, the octets per frame changes to 4:

@@ -172,7 +172,7 @@
#clock-cells = <0>;
clock-output-names = "jesd_tx_lane_clk";

- adi,octets-per-frame = <2>;
+ adi,octets-per-frame = <4>;

The buswidth of the TX DMA also changes:

@@ -106,7 +106,7 @@
reg = <0>;
adi,source-bus-width = <128>;
adi,source-bus-type = <0>;
- adi,destination-bus-width = <128>;
+ adi,destination-bus-width = <64>;

You'll probably need to generate a whole new list of profiles for your use case from the ADRV9009 TES.

Regards,

Adrian

Thanks, LMFSC rate is fixed. And the link status went from CGS to DATA.

I am able to transmit and receive data.

I still have to check if, deterministic latency is fixed.

AdrianC said:

Regarding the lane latency, in subclass 1 it doesn't necessary matter if you have +- octets delay difference between lanes, as the elastic buffer will be released on the next LMFC for both lanes.

Is it OK to assume that the lane latency is deterministic if following output does not exceed 1 multiframe:

This would be OK:

  /sys/bus/platform/devices/84aa0000.axi-jesd204-rx/lane0_info:Lane Latency: 1 Multi-frames and 105 Octets

And this would be a problem:

   /sys/bus/platform/devices/84aa0000.axi-jesd204-rx/lane0_info:Lane Latency: 2 Multi-frames and 1 Octets

All MMCM's are removed and all clock network are driven from,

    util_adrv9009_xcvr => tx_out_clk_1

So, I should have full deterministic latency now?

It's a problem if lane latency for different lanes part of the same link cross over the mutiframe boundary. If all have delays within the same mutiframe, it is fine.

Have you constrained the SYSREF to be source synchronous with the device clock for each path ?

We have an example here: https://github.com/analogdevicesinc/hdl/blob/master/projects/ad_fmclidar1_ebz/zc706/system_constr.xdc#L103 and here: https://github.com/analogdevicesinc/hdl/blob/master/projects/adrv9009zu11eg/common/adrv9009zu11eg_constr.xdc#L211

If you properly constrained the path for your desired frequencies, and the design meets timing, you should have deterministic latency,

Regards,

Adrian

No I did not constrained them yet. I will try to add them.

How can I know if the TX lanes did not cross the multiframe boundary?

I don't think there is an API or functionality for that on ADRV9009. 

You can play with the offset parameter to  https://github.com/analogdevicesinc/linux/blob/master/arch/arm64/boot/dts/xilinx/zynqmp-adrv9009-zu11eg-reva.dtsi#L582 

I'm looking for a better answer, will come back if the above is not enough.

I noticed the fluctuation on the Rx path (like earlier discussed), those seems low enough to make sure they not passes the multi frame clock.The problem is, that I don't have a clue how much the tx fluctuation is. And how far they are separated from the multi frame boundary? If the latency passes the boundary every1/1000,then it will be difficult to find this error.

To solve this, I could put an Rf loopback and check if the latency Tx to Rx shift with one multiframe clock when I increase the "adi,jesd204-deframer-a-lmfc-offset". When this occur, then I could use the used offset as a representation of the spare margin with the default lmfc offset.

Will the result of this Rf loopback be correct? To determine the margin?

I think this should work. I've moved the question to ADRV9009 support forum, maybe they can also confirm.

Regards,

Adrian

As a note:

UG-1295:

When establishing a JESD204B link, it is desirable that the data arriving to the deframer does not arrive very close to an LMFC boundary. If this does happen, the deterministic latency can vary from system to system if the data on one system arrives just before an LMFC event, and arrives on another system just after an LMFC event. If this happens, there is an LMFC period difference in the latency between the systems. Furthermore, the architecture in the device does not support a very small delay through the FIFO, and data corruption occurs if the delay is too small. Therefore, it is important that the FIFO depth be checked after the link is established, and the link is adjusted to achieve a FIFO depth that is close to the medium depth. The FIFO depth can be checked in Register 0x15CE for Deframer 0 and in Register 0x161E for Deframer 1. Write to the appropriate register with a value of 0x80 to latch the current FIFO depth, and then read back. The readback value is reported as a signed, twos complement number located in Bits[D5:D0] of the register with valid values from +K2 to −(K/2 − 1). The value reported is the difference of the number of frames between the read and write pointers. If the value is found to be close to 0, for example, −2, −1, 0, +1, or +2, adjust the depth by varying the LMFC offset parameter on either end of the JESD204B link. Note that across multiple system starts, the depth in the FIFO can vary by one or two frames. This variation is expected because sampling phase uncertainties are absorbed by the FIFO to give deterministic latency. Consider this fact when optimizing the JESD204B link and performing multiple system starts to find the worst case depth values for a given LMFC offset.

 You should be able to check how close you are to LMFC offset by checking register 0x1CE of the ADRV9009.

Regards,

Adrian

yes as AdrianC mentioned with LMFC offset you can tune the latency such that the elastic buffers are release at middle of the multi frame boundary to make sure over boot up, temp,board to board variation will not cause to cross another multi frame boundary.