2009-11-06 11:45:54     Kernel Panic on network packet flood.

Document created by Aaronwu Employee on Aug 19, 2013
Version 1Show Document
  • View in full screen mode

2009-11-06 11:45:54     Kernel Panic on network packet flood.

Matt Gilg (UNITED STATES)

Message: 82134   

 

I have PC software that sends multicast VoIP to multiple blackfin controllers running uClinux 2.6.22.19-ADI-2008R1.5-svn602 isp1362-hcd.  When I approach a ~250 kbits of data on a single multicast address, the controllers all begin to report kernel panics and misaligned data access.  The symptom disappears if I assign each multicast stream a different address. (rather than using a common address & different ports)

 

This problem seems to occur when we receive excess multicast packets on a port that the listening socket is not bound to. Might be a bug in the discard logic?  All the errors below seem to relate to the slob allocator freeing memory...

 

Here are a couple of traces: (these exceptions never happen consistently in the same process, acts like something is getting hammered)

 

//=======================================================================================

 

Data access CPLB miss

 

 

- Used by the MMU to signal a CPLB miss on a data access.

Kernel OOPS in progress

Defered Exception context

CURRENT PROCESS:

COMM=core PID=212

TEXT = 0x01400040-0x014aa720        DATA = 0x014aa724-0x014f04e4

BSS = 0x014f04e4-0x0160e3a4  USER-STACK = 0x0170ef74

 

return address: [0x00033b94]; contents of:

0x00033b70:  001a  e10d  8244  9169  09c1  140d  ac4a  09d0

0x00033b80:  180c  09d1  1c05  09c1  1808  09d0  1806  324a

0x00033b90:  09c1  1ff5 [ac4a] 2ff6  9101  3058  c682  8019

0x00033ba0:  5003  305a  0818  1823  0000  bc42  9109  3059

 

SEQUENCER STATUS:               Not tainted

SEQSTAT: 00000026  IPEND: c030  SYSCFG: 0006

  HWERRCAUSE: 0x0

  EXCAUSE   : 0x26

  physical IVG14 asserted : <0xffa00abc> { _evt14_softirq + 0x0 }

  physical IVG15 asserted : <0xffa00e14> { _evt_system_call + 0x0 }

  logical irq   6 mapped  : <0xffa00270> { _timer_interrupt + 0x0 }

  logical irq  12 mapped  : <0x000ef88c> { _rx_handler + 0x0 }

  logical irq  13 mapped  : <0x000ef82c> { _tx_handler + 0x0 }

  logical irq  18 mapped  : <0x000abf70> { _bfin_serial_dma_rx_int + 0x0 }

  logical irq  19 mapped  : <0x000abedc> { _bfin_serial_dma_tx_int + 0x0 }

  logical irq  24 mapped  : <0x000b58dc> { _bf537mac_interrupt + 0x0 }

  logical irq  45 mapped  : <0x000efc44> { _err_handler + 0x0 }

  logical irq  65 mapped  : <0x000ccbc8> { _usb_hcd_irq + 0x0 }

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x006a5bf4> /* unknown address */

RETX: <0x00033b94> { _slob_free + 0x34 }

RETS: <0x00128526> { ___udp4_lib_rcv + 0x36e }

PC  : <0x00033b94> { _slob_free + 0x34 }

DCPLB_FAULT_ADDR: <0x32633a34> /* unknown address */

ICPLB_FAULT_ADDR: <0x00033b94> { _slob_free + 0x34 }

 

PROCESSOR STATE:

R0 : 00000013    R1 : 0000009f    R2 : 0000ffff    R3 : 0192a000

R4 : 6900a8c0    R5 : 0192a03c    R6 : 00000011    R7 : 2205bfa2

P0 : 0049eb40    P1 : 32633a30    P2 : 32633a30    P3 : 00000011

P4 : 0049eb64    P5 : 001a8244    FP : 0049eb40    SP : 006a5b18

LB0: ffa01d54    LT0: ffa01d54    LC0: 00000000

LB1: 01418a11    LT1: 01418a10    LC1: 00000000

B0 : 00000000    L0 : 00000000    M0 : 00000000    I0 : 015dcd62

B1 : 00000000    L1 : 00000000    M1 : 00000000    I1 : 00000000

B2 : 00000000    L2 : 00000000    M2 : 00000000    I2 : 00000000

B3 : 00000000    L3 : 00000000    M3 : 00000000    I3 : 00000000

A0.w: 0000000b   A0.x: 00000000   A1.w: 0000000b   A1.x: 00000000

USP : 0170ee50  ASTAT: 02002000

 

No trace since you do not have CONFIG_DEBUG_BFIN_NO_KERN_HWTRACE enabled

 

Stack from 006a5af8:

        00000000 ffa00758 001a4560 001a4560 001a455c 00000000 001b29ec 001d62bc

        00033b94 0000c030 00000026 00000000 006a5bf4 00033b94 00033b94 00128526

        00000013 02002000 01418a11 ffa01d54 01418a10 ffa01d54 00000000 00000000

        0000000b 00000000 0000000b 00000000 00000000 00000000 00000000 00000000

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

        00000000 00000000 00000000 015dcd62 0170ee50 0049eb40 001a8244 0049eb64

 

Call Trace:

[<0000ffff>] _disallow_signal+0x1f/0x4c

[<0001166e>] _do_softirq+0x4a/0x4c

[<00011836>] _local_bh_enable+0x56/0x94

[<000fd27c>] _dev_queue_xmit+0xf8/0x1cc

[<000fd1d0>] _dev_queue_xmit+0x4c/0x1cc

[<0010ce08>] _ip_local_deliver+0xd0/0x1d0

[<0010d1f0>] _ip_rcv+0x2e8/0x368

[<0010d0e0>] _ip_rcv+0x1d8/0x368

[<0002ac86>] _generic_file_buffered_write+0x1ee/0x568

[<000fb512>] _netif_receive_skb+0x17e/0x1f4

[<000fcc4e>] _process_backlog+0x76/0x11c

[<00045c80>] _touch_atime+0x74/0xec

[<000fcd64>] _net_rx_action+0x70/0xf8

[<0002b1e2>] ___generic_file_aio_write_nolock+0x1e2/0x3a0

[<000115d8>] ___do_softirq+0x60/0xac

[<00001000>] _run_init_process+0x0/0x18

[<0002b3f4>] _generic_file_aio_write+0x54/0xe4

[<00001000>] _run_init_process+0x0/0x18

[<0003b1f2>] _getname+0xba/0xe8

[<0003b1f2>] _getname+0xba/0xe8

[<0003b186>] _getname+0x4e/0xe8

[<00001000>] _run_init_process+0x0/0x18

[<00001000>] _run_init_process+0x0/0x18

[<00034524>] _do_sys_open+0x10/0xac

[<000345e0>] _sys_open+0x1c/0x24

[<000345c4>] _sys_open+0x0/0x24

[<0000fffe>] _disallow_signal+0x1e/0x4c

[<00008000>] _sram_free+0x24/0x44

 

Modules linked in: usbserial

Kernel panic - not syncing: Kernel exception

 

//======================================================================================= 

 

Data access misaligned address violation

- Attempted misaligned data memory or data cache access.

Kernel OOPS in progress

Defered Exception context

CURRENT PROCESS:

COMM=voip-rtp PID=223

TEXT = 0x00500000-0x005104f0        DATA = 0x00e604f0-0x00e61d00

BSS = 0x00e61d00-0x00ea0000  USER-STACK = 0x00ebfeb0

 

return address: [0x00033b94]; contents of:

0x00033b70:  001a  e10d  8244  9169  09c1  140d  ac4a  09d0

0x00033b80:  180c  09d1  1c05  09c1  1808  09d0  1806  324a

0x00033b90:  09c1  1ff5 [ac4a] 2ff6  9101  3058  c682  8019

0x00033ba0:  5003  305a  0818  1823  0000  bc42  9109  3059

 

SEQUENCER STATUS:               Not tainted

SEQSTAT: 00002024  IPEND: 4030  SYSCFG: 0006

  HWERRCAUSE: 0x0

  EXCAUSE   : 0x24

  physical IVG14 asserted : <0xffa00abc> { _evt14_softirq + 0x0 }

  logical irq   6 mapped  : <0xffa00270> { _timer_interrupt + 0x0 }

  logical irq  12 mapped  : <0x000ef88c> { _rx_handler + 0x0 }

  logical irq  13 mapped  : <0x000ef82c> { _tx_handler + 0x0 }

  logical irq  18 mapped  : <0x000abf70> { _bfin_serial_dma_rx_int + 0x0 }

  logical irq  19 mapped  : <0x000abedc> { _bfin_serial_dma_tx_int + 0x0 }

  logical irq  24 mapped  : <0x000b58dc> { _bf537mac_interrupt + 0x0 }

  logical irq  45 mapped  : <0x000efc44> { _err_handler + 0x0 }

  logical irq  65 mapped  : <0x000ccbc8> { _usb_hcd_irq + 0x0 }

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x012f7e1c> /* unknown address */

RETX: <0x00033b94> { _slob_free + 0x34 }

RETS: <0x000f6b08> { _kfree_skbmem + 0xc }

PC  : <0x00033b94> { _slob_free + 0x34 }

DCPLB_FAULT_ADDR: <0x3e223124> /* unknown address */

ICPLB_FAULT_ADDR: <0x00033b94> { _slob_free + 0x34 }

 

PROCESSOR STATE:

R0 : 01517000    R1 : 00000000    R2 : 0000ffff    R3 : 0000000f

R4 : 00000040    R5 : 0000a6d4    R6 : 001d6674    R7 : 00000400

P0 : 01517000    P1 : 3e223122    P2 : 3e223122    P3 : 001d66d4

P4 : 004c30a0    P5 : 001a8244    FP : 003aa020    SP : 012f7d40

LB0: ffa01d55    LT0: ffa01d54    LC0: 00000000

LB1: 00a65c99    LT1: 00a65c98    LC1: 00000000

B0 : 00000000    L0 : 00000000    M0 : 00000000    I0 : 012ddeb0

B1 : 00000000    L1 : 00000000    M1 : 00000000    I1 : 00000000

B2 : 00000000    L2 : 00000000    M2 : 00000000    I2 : 00000000

B3 : 00000000    L3 : 00000000    M3 : 00000000    I3 : 00000000

A0.w: 00007dd5   A0.x: 00000000   A1.w: 00000b27   A1.x: 00000000

USP : 012efce4  ASTAT: 02003005

 

Hardware Trace:

   0 Target : <0x000048c0> { _trap_c + 0x0 }

     Source : <0xffa00754> { _exception_to_level5 + 0xb4 }

   1 Target : <0xffa006a0> { _exception_to_level5 + 0x0 }

     Source : <0xffa005f8> { _ex_trap_c + 0x5c }

   2 Target : <0xffa0059c> { _ex_trap_c + 0x0 }

     Source : <0xffa007f4> { _trap + 0x28 }

   3 Target : <0xffa007cc> { _trap + 0x0 }

     Source : <0x00033b92> { _slob_free + 0x32 }

   4 Target : <0x00033b8e> { _slob_free + 0x2e }

     Source : <0x00033b84> { _slob_free + 0x24 }

   5 Target : <0x00033b82> { _slob_free + 0x22 }

     Source : <0x00033b96> { _slob_free + 0x36 }

   6 Target : <0x00033b8e> { _slob_free + 0x2e }

     Source : <0x00033b84> { _slob_free + 0x24 }

   7 Target : <0x00033b82> { _slob_free + 0x22 }

     Source : <0x00033b96> { _slob_free + 0x36 }

   8 Target : <0x00033b94> { _slob_free + 0x34 }

     Source : <0x00033b7a> { _slob_free + 0x1a }

   9 Target : <0x00033b60> { _slob_free + 0x0 }

     Source : <0x00033d56> { _kfree + 0x8a }

  10 Target : <0x00033d4a> { _kfree + 0x7e }

     Source : <0x00033ce0> { _kfree + 0x14 }

  11 Target : <0x00033ccc> { _kfree + 0x0 }

     Source : <0x000f6d80> { _skb_release_data + 0x60 }

  12 Target : <0x000f6d76> { _skb_release_data + 0x56 }

     Source : <0x000f6d6c> { _skb_release_data + 0x4c }

  13 Target : <0x000f6d5e> { _skb_release_data + 0x3e }

     Source : <0x000f6d2e> { _skb_release_data + 0xe }

  14 Target : <0x000f6d20> { _skb_release_data + 0x0 }

     Source : <0x000f6b04> { _kfree_skbmem + 0x8 }

  15 Target : <0x000f6afc> { _kfree_skbmem + 0x0 }

     Source : <0x000f6c40> { ___kfree_skb + 0x90 }

Stack from 012f7d20:

        012f7d48 ffa00758 001a4560 001a4560 001a455c 012f7d48 00000000 0000001f

        00033b94 00004030 00002024 00000000 012f7e1c 00033b94 00033b94 000f6b08

        01517000 02003005 00a65c99 ffa01d55 00a65c98 ffa01d54 00000000 00000000

        00000b27 00000000 00007dd5 00000000 00000000 00000000 00000000 00000000

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

        00000000 00000000 00000000 012ddeb0 012efce4 003aa020 001a8244 004c30a0

 

Call Trace:

[<0000a6d4>] _account_system_time+0xac/0xe8

[<0000ffff>] _disallow_signal+0x1f/0x4c

[<000fb556>] _netif_receive_skb+0x1c2/0x1f4

[<000fb556>] _netif_receive_skb+0x1c2/0x1f4

[<000fcc4e>] _process_backlog+0x76/0x11c

[<000fcd64>] _net_rx_action+0x70/0xf8

[<0000a6d4>] _account_system_time+0xac/0xe8

[<000115d8>] ___do_softirq+0x60/0xac

[<00071a66>] _fat_fill_super+0x53e/0x9a8

[<000cb913>] _hub_thread+0xc03/0xc68

[<00002000>] _get_cclk+0x30/0x68

[<00007dd5>] __l1_sram_free+0x2d/0xb4

[<00071a66>] _fat_fill_super+0x53e/0x9a8

 

Modules linked in: usbserial

Kernel panic - not syncing: Kernel exception

 

//======================================================================================= 

 

Any thoughts?

 

 

 

QuoteReplyEditDelete

 

 

2009-11-06 13:06:02     Re: Kernel Panic on network packet flood.

Robin Getz (UNITED STATES)

Message: 82137   

 

Matt:

 

1) Is there any possibility of upgrading?

 

2) Does it make a difference if you use slab?

 

3) Is there an easy way of replicating this with standard tools (like ping flood?)

 

-Robin

Attachments

    Outcomes