Corrupted Packets Only at 10M When Running 100% Traffic

HI Walter,

Thanks for your message and the detailed information. 

Can you advise what the link partner is? Is the established link HD or FD? How do you configure the link speed (which side, is it forced or auto-neg). I will review with the PHY team, but note our responses may be delayed until next week due to the holidays. 

best regards,

Catherine. 

Link partner is either an IXIA or Xena Packet Generator (do not know the PHY). We had been testing with Windows PCs and Tinker micros and iPerf but have switched to packet generators for more controlled testing.

Always use Auto-Neg, but use Advertisement settings to control resulting link speed (i.e. only advertise 10M FD)

Always Full-Duplex as trying to test at 100% line rate not possible when Half Duplex selected.

I have reviewed history notes (going back over a year) and can add:

Originally project used the AD CN0506 Evaluation board with a FPGA dev platform to evaluate ADIN1300, the notes indicate the inability to obtain 100% at 10M FD existed on this board.

We have built our own daughtercard, very similar to CN0506 and its exhibits same behavior.

After extensive testing of both the CN0506 and our board, we have nearly concluded that it is the TX side of the ADIN1300 at 10M ONLY.

All testing at 1000M FD and 100M FD have achieved 100% line rate with BER=0 and no lost packets over extended test periods.

Here is some more info as we have been performing more detailed testing.

Problem definitely appears at the TX side of the ADIN1300 only at 10M Full Duplex.

The issues is repeatable (i.e. does not appear to be a noise, power supply, etc. issue).

Our testing has focused on 66-byte IPV4/UDP packets.

The ADIN1300 is driven by an FPGA where through ILA (logic analyzer) we can observe exact data going out the RGMII interface to the PHY.

The issues comes and goes with the width of the IFG and is related to packet size.

When we start transmitting, we see the first packet come out of the PHY (ADIN1300) correctly followed by the IFG.

It is approximately mid-way through the second packet that we see the "insertion" of 54-56 bits of ''0" inserted.  The original packet bits (as verified using ILA) are present both at the beginning and end of the packet. In other words, the packet has grown in size by approximately 7 bytes of 0x00.

This repeats for the third, fourth and sometimes fifth packets after which the tester (Xena) receives from 1-2 properly formed packets. Surprisingly, this cycle repeats over and over, 1-2 good packets followed by 4-5 bad/oversized packets.

For 66-byte packets, setting the tester's IPG to 62 bytes (spec minimum is 20 bytes = 8 IFG, 7 Preamble, 1 SFD) results is successful transmission of over 2000 packets.

Setting IFG to minimum of 8 bytes, 96-bits results in almost complete loss of packets.

Hi, thanks for the detailed explanation of the issue.

From what you’ve described, the problem could be related to how the traffic is sent to the PHY at the MAC interface, especially at 10 Mbps. Since it works fine at 100 Mbps and 1000 Mbps, we don’t think the problem is on the MDI side. To dig deeper, could you clarify a few things for us? 

1. What happens with less than 100% traffic? Does the issue still show up, or does it improve?

2. What does the preamble look like? Will you be able to probe the RGMII signals? Have you checked the clock alignment?

3. Is the FPGA generating the packets or relaying them from somewhere else? If it’s relaying, could part of the preamble be getting dropped?

regards,

Mark

1. What happens with less than 100% traffic? Does the issue still show up, or does it improve?

This data shows two things, varies with packet size and problem disappears as rate is lowered, for example any rate below 81% works at 66-byte packets. The Xena tester generates packets at a linear rate, so essentially as rate is lowered the IFG increases.

10M 66 Bytes 81% OK
10M 66 Bytes >82% BAD – Hi Packet Loss/Corruption
10M 759 Bytes 97% OK
10M 759 Bytes >98% BAD – Hi Packet Loss/Corruption
10M 1518 Bytes 98% OK
10M 1518 Bytes >99% BAD – Hi Packet Loss/Corruption
10M Random Bytes 97% OK
10M Random Bytes >98% BAD – Hi Packet Loss/Corruption

It varies with packet size, for example with all 66 byte packets, loss starts at 82% loading while at 1518 bytes starts having problems at 99%.

2. What does the preamble look like? Will you be able to probe the RGMII signals? Have you checked the clock alignment?

RGMII Clock alignment has been verified. ADIN1300 is driven by an FPGA where through ILA (logic analyzer) we can observe exact data going out the RGMII interface to the PHY. We transmit a full 7-byte (0x55) Preamble with SFD. Its been verified that packet is sent to PHY contiguously with no gaps in a packet.

3. Is the FPGA generating the packets or relaying them from somewhere else? If it’s relaying, could part of the preamble be getting dropped?

Yes, the FPGA is rx'ing packets from another ADIN1300, BUT the Preamble and SFD are dropped upon receive.  The Preamble (7 x 0x55) and SFD are re-created at the TX side. We are definitely sending a completed Preamble and SFD to PHY (as verified via logic analyzer (ILA)).

After a lot of testing, it is really as simple as set packet size to 66 and set bandwidth > 81% and packets are corrupted. Change rate to say 70% (this basically increases IFG) and packets pass error free forever.  No changes to clock/data timing, etc.

Hi,

Thank you for your patience. I consulted with our design team, and they provided the following insights:

The information provided is somewhat confusing due to the loose use of terminology. It’s unclear whether the terms refer to IEEE Std 802.3 or the Xena test platform. I assume it’s the latter, which would follow RFC-2544.

Regarding the last two sentences in the message from January 13:

• For 66-byte packets, setting the tester's IPG to 62 bytes (spec minimum is 20 bytes = 8 IFG, 7 Preamble, 1 SFD) results in successful transmission of over 2000 packets.
• Setting IFG to a minimum of 8 bytes, 96-bits results in almost complete loss of packets.

The spec minimum of 20 bytes (8 IFG, 7 Preamble, 1 SFD) is confusing. Which spec is this? According to IEEE Std 802.3 (and RFC-2544), the IPG minimum for Ethernet is 12 bytes. It seems the reference is to IFG, considering the packet header.

• Setting IFG to a minimum of 8 bytes, 96-bits: Is it IFG or IPG? 96-bits is 12 bytes, which is the minimum IPG per IEEE Std 802.3. If the tester equipment sends less than 12 IPG bytes, it would not be IEEE Std 802.3 compliant. The Xena tester should follow RFC-2544.

Additionally, the message from January 15 raises concerns:

• The FPGA is receiving packets from another ADIN1300, but the Preamble and SFD are dropped upon receipt. They are recreated on the TX side. We are sending a complete Preamble and SFD to the PHY (verified via logic analyzer).

It’s unclear to us how your test setup looks. It appears you are testing the ADIN1300 DUT TX performance using an FPGA to receive packets from another ADIN1300, connected to the Xena tester. The FPGA is then connected to the DUT TX through RGMII. This setup tests not just the DUT TX but the ADIN1300 RX -> FPGA -> DUT TX. How do they ensure the IPG sent by the tester is preserved through the FPGA?

We believe the issue lies in the ADIN1300 RX -> FPGA -> DUT RGMII TX section. How do you verify the FPGA behavior when connected to the ADIN1300 RX is correct? In 10BASE-T mode, the ADIN1300 receives clock stops during most of the frame preamble and can stretch at the end of the packet. If you use that clock to regenerate the packet on the TX side, it could be problematic. If the setup is as described, it would be preferable to remove the FPGA. Enable preamble regeneration on the ADIN1300 connected to the Xena tester and connect its RGMII RX to the DUT’s RGMII TX directly. Otherwise, you may need to share the logic analyzer captures of the DUT RGMII TX and the DUT MDI.

We have not received any customer reports of issues with the ADIN1300 10BASE-T TX path, and we test the 10BASE-T TX path at full rate (100% throughput) in our internal evaluations.

regards,

Mark

Yes, in an attempt to supply highly detailed error conditions it became confusing.  All packets are generated meeting IEEE standards with the minimum gap of 12 bytes / 96-bits / 9.6us being enforced by the FPGA on the transmit side, regardless of what is received.  FWIW, this is an established tested design using another PHY, we are dropping in and evaluating the ADIN1300 and as mentioned the design works perfectly at 1000M and 100M. So, we believe we are looking at something unique to the 10M mode.

Want to concentrate on your second observation, specifically you state "In 10BASE-T mode, the ADIN1300 receives clock stops during most of the frame preamble and can stretch at the end of the packet"

You are correct when you say: ADIN1300 RX -> FPGA -> DUT (ADIN1300) RGMII TX section and the design does use the RX clock as the TX clock.  The presence of any stops or gaps would be new to us as we don't see this in the data sheets and our past experience with other PHYs is that RX is free-running/contiguous. NOTE: We are not using/implementing LPI.

(1) Is the presence of gaps/stops in the RX clock in 10M always present?  (We are instrumenting out design to monitor RX clock. To date, we use an integrated logic analyzer (ILA) inside the FPGA to confirm RX and TX packets, however an ILA running on RX clock would not reveal gaps in RX clock.

(2) Can this be controlled? We have tried disabling clock stop, @0x9400 RX_MII_CLK_STOP_EN Receive MII Clock Stop Enable Register by setting bit 10 to zero and this did not help.  BTW, we have already enabled Preamble Regeneration on the RX ADIN1300 although our FPGA (i.e. MAC) can handle the shortened preamble.

(3) Are these RX clock gaps unique to 10M mode? 1000M and 100M work fine at all data rates, etc. using RX clock as TX clock (we do not implement LPI)

Any update on my last three questions?

Any update on my last three questions?

Hi wraujr 

I've received your request and will notify the team that supports this product.

Regards,

Shazia