AnsweredAssumed Answered

BF537, NAPI Ethernet driver, out of order data

Question asked by sfe on Feb 26, 2016

Hello!

 

in an earlier Message to Sonic Zhang I described a problem I had with the NAPI ethernet driver on a BF537 CPU:

 

"

Hello Mr. Zhang,

 

currently I'm working on the ethernet driver for a custom Board with an Blackfin BF537 processor.

Kernel Version ist 3.10 off the uCLinux Branch (ADI-2013R1).

But we included the conversion to NAPI Patch from a later version (signed off by you, 2014-07-24).

 

In normal operations the driver works fine. But under very high load (the device is used in audio setups, and sometimes sombody unleashes a multichannel multicast stream on the wrong network) we see unexpected behaviour. The first sign where out of order packages in a ping request startet after the high load.

The situation is worsened by our main application that uses up ~50% of the CPU time.

Looking at the driver more closely it became obvious that the DMA pointers and the internal data pointer where not in sync anymore. So far my attempts to improve the situation failed.

 

Do you have any advice in that matter? What where your considerations regarding synchronisation when you switched to the NAPI model?

I found following paragraph in the ref manual of the BFin:

 

"Interrupt-based synchronization methods must avoid interrupt overrun, or the failure to invoke a DMA channel’s interrupt handler for every interrupt event due to excessive latency in processing of interrupts."

(page 5-55, Rev. 3.4)

 

which seem very relevant for this problem.

 

Any input from your side would be highly appreciated.

 

Regards,

Sebastian Fett

"

 

Updates to that email are:

  • I switched back to a pre-NAPI driver in our application, and the problems disappeared. The impact of the IRQ load is very obvious in the device, but the data is received properly.
  • I wrote a small test (see code below. sorry for the hack). It sends 50 RAW ethernet packages as fast as possible from a PC. One of the data bytes is incremented. The same code can be compiled for the BFin, but then receives the data and checks the order of the data. With a application running in the background (30-50% CPU base load) I can replicate the out of order data easily with the NAPI driver. Without NAPI no out of order data happens.

 

the tool:

It's the same code for PC and BFin. For BFin you need to #define __BFIN_RX__ and use the bfin-linux-uclibc-gcc cross compiler. The PC version can be compiled normally.

The target MAC address is hard coded. The ehternet device can be set as a parameter ( ./testtool eth0). The default is eth0.

 

Possible Tool Output:

Using a RX buffer of 20 pakets I get the following result. The data is the incremented byte in the message and should be +1 in each step.

 

0

0x1

0x2

0x3

0x4

0x5

0x6

0x7

0x8

0x9

0xa

0xb

0xc

0xd

0xe

0xf

0x10

0x11

0x12

0x13

0x14

0x15

0x16

0x17

0x18

0x19

0x1a

0x1b

0x1c

OOO Data expected 0x1d is 0x31

0x31

0x1e

0x1f

0x20

0x21

0x22

0x23

0x24

0x25

0x26

0x27

0x28

0x29

0x2a

0x2b

0x2c

0x2d

0x2e

0x2f

0x30

OOO Data expected 0x31 is 0x10

0x10

 

Her you can see how 0x31 cames way early, practically changing places with 0x10.

 

Another result show more scrambling in the data:

 

0

0x1

0x2

0x3

0x4

0x5

0x6

0x7

0x8

0x9

0xa

0xb

0xc

0xd

0xe

0xf

0x10

0x11

0x12

0x13

0x14

0x15

0x16

OOO Data expected 0x17 is 0x2b

0x2b

OOO Data expected 0x18 is 0x2c

0x2c

OOO Data expected 0x19 is 0x2d

0x2d

OOO Data expected 0x1a is 0x2e

0x2e

OOO Data expected 0x1b is 0x2f

0x2f

OOO Data expected 0x1c is 0x30

0x30

OOO Data expected 0x1d is 0x31

0x31

0x1e

0x1f

0x20

0x21

0x22

0x23

0x24

0x25

0x26

0x27

0x28

0x29

0x2a

 

 

 

Regards

Sebastian

Attachments

Outcomes