2011-07-14 15:14:19 system hangs up from network interface
Peter Gombos (HUNGARY)
Message: 102410
Hi,
We use 2010R1RC5 and used RC1 before. Our system works fine. We use bf537 CPU. We use the eth0 interface in "accept all multicast" mode. Our device transmitting and receiveing raw ethernet frames using multicast destination MAC address range. If I connect the device to an unmanaged switch (like a cheap 8 port SOHO switch). I make a loop between other two ports. As soon as the device transmits an ethernet frame this frame it is repeated with full speed forever.
The device immediately stop responding. After a short time the internal watchdog restarts the board. If I cut the loop sometimes the device restores and operates perfectly but sometimes slows down.
This is an annoying situation because a misconnected cable hangs up all devices on the net. I can accept the restart behavior but the unstable condition is a big problem.
How can I avoid this?
QuoteReplyEditDelete
2011-07-14 20:47:57 Re: system hangs up from network interface
Aaron Wu (CHINA)
Message: 102412
Could you elaborate the detailed steps how we could reproduce this issue, and what ethernet controller are you using?
QuoteReplyEditDelete
2011-07-15 09:52:49 Re: system hangs up from network interface
Peter Gombos (HUNGARY)
Message: 102439
Hi
We spent our day to investigate the situation. We use a customized BF-537 board with 2010R1-RC5. The bfin_mac driver goes into a strange mode. This is the way to reproduce it:
Take a cheap SOHO switch. Connect port 1 to port 2. Connect port 3 to an external network just for a moment. As soon as the switch receives any frame that frame drives the switch crazy because of the loop. As far as I measured approx. 50k frames transmitted in every seconds. This is the perfect frame burst generator.
At first try to ping your bfin board from a computer. Everything seems ok, ping time is very low and there is no lost packet.
Disconnect the computer and connect your frame generator (the crazy switch) to the board. At this moment the board stops responding. Do not keep connected too long because the internal watchdog resets it. Sometimes 1 second is enough. Sometimes it need three times five seconds.
Now the board getting into this state. Try to ping it again. The ping time sometimes OK, sometimes extremely long and sometimes there are duplicated frames. I attached the ping results. The device is not able to recovery from this mode. In this state TCP/IP works but very very slow.
Going deeper. We used an oscilloscope. I put some GPIO toggle in bfin_mac.c driver. I attached a screenshot. There are 3 channels on the screen. The yellow channel coming from phy. It shows the valid frame received. The cyan shows the interrupt entry point (bfin_mac_interrupt() function). The green channel toggle when the real_rx label reached and bfin_mac_rx() function called from the ISR.
It seems that every single frame causes interrupt but not all of then served by bfin_mac_rx(). Sometimes one interrupt causes 20 bfin_mac_rx() call. In this case there are duplicated ping results generated.
pingresult.txt
uclinux rx dma 110715_145638.png
QuoteReplyEditDelete
2011-07-18 02:11:36 Re: system hangs up from network interface
Aaron Wu (CHINA)
Message: 102454
Thanks for clarification, do you mean first use a cross ethernet cable to connect to two ports of a switch, and then connect the switch to a PC at another port, the switch will go crazy automatically and then I just remove one end of the ethernet cable from pc and connect it to blackfin board instead for a while like 1 or 2 seconds, and then the blackfin board will be in the state you mentioned.
QuoteReplyEditDelete
2011-07-18 03:53:10 Re: system hangs up from network interface
Peter Gombos (HUNGARY)
Message: 102458
Exactly.
QuoteReplyEditDelete
2011-07-18 05:01:04 Re: system hangs up from network interface
John Rees (UNITED KINGDOM)
Message: 102463
Hi Peter
Please take a look at my posting, I think we have encountered the same issue. We have modified our IRQ handler in bfin_mac.c and we appear to have fixed our instability which appears to be similar to yours. I posted the changes onto the forum.
Although we have made the change on a 2006-R2-RC2 kernel, the code in this area on the new kernel looks like it has little changes.
May be you would like to try the changes in your kernel.
Looking forward to your thoughts and findings.
Many Thanks
John Rees
QuoteReplyEditDelete
2011-07-19 01:19:17 Re: system hangs up from network interface
Aaron Wu (CHINA)
Message: 102479
How can I determine that the switch becomes a "crazy frame generator" sucsesfully, once i saw the link leds for the cross connected port keep flashing, but this seldom happens. I'm using a TP-LINK TL-SF1008+, a very chip 8 port switch
QuoteReplyEditDelete
2011-07-19 03:40:42 Re: system hangs up from network interface
Peter Gombos (HUNGARY)
Message: 102519
Use wireshark or ethereal. You will see terrible traffic in this state.
QuoteReplyEditDelete
2011-07-19 16:26:15 Re: system hangs up from network interface
James Kosin (UNITED STATES)
Message: 102542
Aaron Wu,
What happens in this state is the switch will forward any broadcast packets to the other interfaces on the same network which everyone is a part of... and you end up with a switch that is completely overwelmed and flooded with traffic. Mostly network broadcast traffic.
The switch normally gets very confused in this state ... so, it make take a while to re-learn or un-learn any mistakes in the routing before things get back to normal. I think my swith defaults to 5-minutes. This is an unusual case of flooding traffic....
James
QuoteReplyEditDelete
2011-07-21 01:31:01 Re: system hangs up from network interface
Aaron Wu (CHINA)
Message: 102558
Thanks James,
Yes I once saw the LED of the switch keeps flashing after connecting the cross cable to two diffrent ports, another connected to a router or computer, but now I can not see this flashing anymore. I guess once it's in the crazy mode the LED flashes, now I've no idea why it can not anymore, I powered down/up the switch for times to try.
QuoteReplyEditDelete
2011-07-21 08:21:34 Re: system hangs up from network interface
John Rees (UNITED KINGDOM)
Message: 102594
Looping back interfaces causes a broad cast storm, broadcasts go around the network for ever! The packets will run around the network at the near maximum bandwidth.
If your switch has spanning tree enabled or a bit of intelligence then it is possible that the switch has detected the situation and blocked the port.
Don't try this test on a company network, you will kill it in an instant.
QuoteReplyEditDelete
2011-07-21 08:23:37 Re: system hangs up from network interface
John Rees (UNITED KINGDOM)
Message: 102595
I forgot to mention, merely looping back interface may not be enough, you have to have a device attached which generates at least some broadcast traffic, directed traffic will only be passed out the interfaces that it needs to be passed out.
