[#6892] [ltp] execve05 may cause kernel panic on bf561-ezkit SMP kernel due to "IPI message queue overflow"(2011)

[#6892] [ltp] execve05 may cause kernel panic on bf561-ezkit SMP kernel due to "IPI message queue overflow"

Submitted By: Vivi Li

Open Date

2011-11-21 03:37:12

Priority:

Medium Assignee:

steven miao

Status:

Open Fixed In Release:

N/A

Found In Release:

N/A Release:

Category:

N/A Board:

N/A

Processor:

BF561 Silicon Revision:

Is this bug repeatable?:

Yes Resolution:

Not reproducible

Uboot version or rev.:

Toolchain version or rev.:

gcc4.3-2011R1-RC2

App binary format:

N/A

Summary: [ltp] execve05 may cause kernel panic on bf561-ezkit SMP kernel due to "IPI message queue overflow"

Details:

LTP test case execve05 may cause kernel panic on bf561-ezkit SMP kernel due to "IPI message queue overflow" in 2011r1 branch.

It seems this bug always happens in ltp test with wt slab configuration.

root:/> execve05 20 /bin/execve05 /bin/execve05 4

execve05 1 TPASS : Test DONE, pid 888, -- /bin/execve05 0 /bin/execve05 /bin/execve05

execve05 1 TPASS : Test DONE, pid 886, -- /bin/execve05 0 /bin/execve05 /bin/execve05

execve05 1 TPASS : Test DONE, pid 892, -- /bin/execve05 0 /bin/execve05 /bin/execve05

execve05 1 TPASS : Test DONE, pid 891, -- /bin/execve05 0 /bin/execve05 /bin/execve05

execve05 1 TPASS : Test DONE, pid 890, -- /bin/execve05 0 /bin/execve05 /bin/execve05

execve05 1 TPASS : Test DONE, pid 893, -- /bin/execve05 0 /bin/execve05 /bin/execve05

execve05 1 TPASS : Test DONE, pid 889, -- /bin/execve05 0 /bin/execve05 /bin/execve05

execve05 1 TPASS : Test DONE, pid 887, -- /bin/execve05 0 /bin/execve05 /bin/execve05

execve05 0 TINFO : Test PASSED

Kernel panic - not syncing: IPI message queue overflow

Hardware Trace:

0 Target : <0x0012fd68> { _dump_stack + 0x0 }

Source : <0x0012fdf0> { _panic + 0x50 } CALL pcrel

1 Target : <0x0012fdf0> { _panic + 0x50 }

Source : <0x0012ff5c> { _printk + 0x14 } RTS

2 Target : <0x0012ff58> { _printk + 0x10 }

Source : <0x00019b90> { _vprintk + 0x19c } RTS

3 Target : <0x00019b84> { _vprintk + 0x190 }

Source : <0x00019b5a> { _vprintk + 0x166 } IF CC JUMP pcrel

4 Target : <0x00019b52> { _vprintk + 0x15e }

Source : <0x00019cfe> { _vprintk + 0x30a } JUMP.S

5 Target : <0x00019cfe> { _vprintk + 0x30a }

Source : <0x00019718> { _console_unlock + 0x1e4 } RTS

6 Target : <0x000196d4> { _console_unlock + 0x1a0 }

Source : <0x00131a9a> { __raw_spin_unlock_irqrestore + 0x42 } RTS

7 Target : <0x00131a94> { __raw_spin_unlock_irqrestore + 0x3c }

Source : <0x00131a6a> { __raw_spin_unlock_irqrestore + 0x12 } IF CC JUMP pcrel (BP)

8 Target : <0x00131a64> { __raw_spin_unlock_irqrestore + 0xc }

Source : <0x0000caf6> { ___raw_spin_unlock_asm + 0x16 } RTS

9 Target : <0x0000caf4> { ___raw_spin_unlock_asm + 0x14 }

Source : <0x0000c936> { _put_core_lock + 0x1a } RTS

10 Target : <0x0000c91c> { _put_core_lock + 0x0 }

Source : <0x0000ca66> { _end_lock_coherent + 0x20 } JUMP.S

11 Target : <0x0000ca46> { _end_lock_coherent + 0x0 }

Source : <0x0000caf0> { ___raw_spin_unlock_asm + 0x10 } CALL pcrel

12 Target : <0x0000cae8> { ___raw_spin_unlock_asm + 0x8 }

Source : <0x0000c8e8> { _get_core_lock + 0x40 } RTS

13 Target : <0x0000c8cc> { _get_core_lock + 0x24 }

Source : <0x0000c8bc> { _get_core_lock + 0x14 } IF CC JUMP pcrel

14 Target : <0x0000c8a8> { _get_core_lock + 0x0 }

Source : <0x0000cae4> { ___raw_spin_unlock_asm + 0x4 } JUMP.L

15 Target : <0x0000cae0> { ___raw_spin_unlock_asm + 0x0 }

Source : <0x00131a60> { __raw_spin_unlock_irqrestore + 0x8 } JUMP.L

Stack info:

SP: [0x02677cf8] <0x02677cf8> /* kernel dynamic memory (maybe user-space) */

Memory from 0x02677cf0 to 02678000

02677cf0: 02677cf8 02677d4c [0016254c] 0012fdf4 0034f120 0016254c 00192b4f 00192b4f

02677d10: 00192b4f 02677d40 00198f98 0000c4d4 0034f120 00000001 002c0120 cccccccd

02677d30: 02677d48 001a0afc 02677d4c 00000000 00000005 0000001f 00000002 00000002

02677d50: 00000002 00000000 0000c4f4 00315800 0230c64c 020b46ec 02880000 028a2000

02677d70: 02880000 02880000 0000c004 00192260 00000000 00052a96 00131a64 00000006

02677d90: 00000001 00000000 00000000 00000000 00002e74 00022000 00000022 00000000

02677db0: 0205d060 024e8e00 00083de0 0001005b 00000006 0000a560 02676000 00000005

02677dd0: 0000a560 00002e74 024e8e00 00000000 00000000 00000007 00000002 00000000

02677df0: 00000000 00000000 00000000 00000000 0001ffc0 00000000 00000004 0000067a

02677e10: 000019e8 60a50000 d4d30000 401e0100 00022000 00000001 00004a6c 01000000

02677e30: 195fca4e 0001005b 00021e9f 00000000 00000000 00000000 00000000 00000000

02677e50: 00000000 00000000 00000000 00000000 80000040 024e8e00 <0005ae54> 001a6928

02677e70: 000836f8 001a6538 fffffff8 ffe05000 00198f98 00000000 0005adc8 02677f24

02677e90: 00316220 024e8e02 00000000 02aa1ed8 0005bc2e 02a8df94 02aa1fa0 02a8d0e4

02677eb0: 00000000 024e8e00 027ad000 00000001 00001000 02677f24 024e8e00 00000000

02677ed0: 00001cee 00001cb8 0000000b 027ad000 02aa1fa0 02a8df94 02a80220 10000000

02677ef0: 02aa1ec0 00000000 02677f24 <0000b13e> 00000000 ffffe000 02a8a52c 02aa1fd0

02677f10: 02a0a52c 0000fffe 00000003 02a80220 02a8a52c 02a82d22 00008000 00000000

02677f30: 00000000 02678000 02a82d22 02a82d22 02a80412 0000b914 02002000 02a83a3d

02677f50: 02a846ed 02a83a3c 02a846ea 00000000 00000000 00000000 00000000 00000000

02677f70: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

02677f90: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 0260cf98

02677fb0: 02a8d0c8 02aa1ecc 02aa1ed8 00000000 00000005 02a8d0e4 02a8dfbc 02a8a8cf

02677fd0: 0000000b 02aa1f24 00000009 02a8a52c 02a80220 00000003 02a8df94 02aa1fa0

02677ff0: 02aa1fd0 02aa1fd0 0000000b 00000006

Return addresses in stack:

address : <0x0005ae54> { _search_binary_handler + 0x80 }

address : <0x0000b13e> { _system_call + 0x6a }

Follow-ups

--- steven miao 2011-11-25 02:10:52

hold msg queue lock while calling cross function

--- Vivi Li 2011-12-07 04:52:24

OK. Close it.

--- Vivi Li 2012-01-11 05:19:38

This bug still can be seen sometimes in regression log.

root:/bin> ^[[6n

************** STEP 2478: execute [pipe11] ...

pipe11^M

pipe11 0 TINFO : child 1 got 50 chars^M

pipe11 0 TINFO : child 4 got 50 chars^M

pipe11 0 TINFO : child 2 got 50 chars^M

pipe11 0 TINFO : child 3 got 50 chars^M

pipe11 0 TINFO : child 5 got 50 chars^M

pipe11 0 TINFO : child 8 got 50 chars^M

pipe11 0 TINFO : child 7 got 50 chars^M

pipe11 0 TINFO : child 9 got 50 chars^M

pipe11 0 TINFO : child 6 got 50 chars^M

pipe11 0 TINFO : child 11 got 50 chars^M

pipe11 0 TINFO : child 12 got 50 chars^M

pipe11 0 TINFO : child 10 got 50 chars^M

pipe11 0 TINFO : child 14 got 50 chars^M

pipe11 0 TINFO : child 13 got 50 chars^M

pipe11 0 TINFO : child 16 got 50 chars^M

pipe11 0 TINFO : child 17 got 50 chars^M

pipe11 0 TINFO : child 19 got 50 chars^M

pipe11 0 TINFO : child 20 got 50 chars^M

pipe11 K e TrNFOn : echild 2 gop 50achansic - not syncing: IPI message

queue overflow^M

Hardware Trace:^M

0 Target : <0x0012fd68> { _dump_stack + 0x0 }^M

Source : <0x0012fdf0> { _panic + 0x50 } CALL pcrel^M

1 Target : <0x0012fdf0> { _panic + 0x50 }^M

Source : <0x0012ff5c> { _printk + 0x14 } RTS^M

2 Target : <0x0012ff58> { _printk + 0x10 }^M

Source : <0x00019b90> { _vprintk + 0x19c } RTS^M

3 Target : <0x00019b84> { _vprintk + 0x190 }^M

Source : <0x00019b5a> { _vprintk + 0x166 } IF CC JUMP pcrel ^M

4 Target : <0x00019b52> { _vprintk + 0x15e }^M

Source : <0x00019cfe> { _vprintk + 0x30a } JUMP.S^M

5 Target : <0x00019cfe> { _vprintk + 0x30a }^M

Source : <0x00019718> { _console_unlock + 0x1e4 } RTS^M

6 Target : <0x000196d4> { _console_unlock + 0x1a0 }^M

Source : <0x00131a9a> { __raw_spin_unlock_irqrestore + 0x42 } RTS^M

7 Target : <0x00131a94> { __raw_spin_unlock_irqrestore + 0x3c }^M

Source : <0x00131a6a> { __raw_spin_unlock_irqrestore + 0x12 } IF CC

JUMP pcrel (BP)^M

8 Target : <0x00131a64> { __raw_spin_unlock_irqrestore + 0xc }^M

Source : <0x0000caf6> { ___raw_spin_unlock_asm + 0x16 } RTS^M

9 Target : <0x0000caf4> { ___raw_spin_unlock_asm + 0x14 }^M

Source : <0x0000c936> { _put_core_lock + 0x1a } RTS^M

10 Target : <0x0000c91c> { _put_core_lock + 0x0 }^M

Source : <0x0000ca66> { _end_lock_coherent + 0x20 } JUMP.S^M

11 Target : <0x0000ca46> { _end_lock_coherent + 0x0 }^M

Source : <0x0000caf0> { ___raw_spin_unlock_asm + 0x10 } CALL

pcrel^M

12 Target : <0x0000cae8> { ___raw_spin_unlock_asm + 0x8 }^M

Source : <0x0000c8e8> { _get_core_lock + 0x40 } RTS^M

13 Target : <0x0000c8cc> { _get_core_lock + 0x24 }^M

Source : <0x0000c8bc> { _get_core_lock + 0x14 } IF CC JUMP pcrel ^M

14 Target : <0x0000c8a8> { _get_core_lock + 0x0 }^M

Source : <0x0000cae4> { ___raw_spin_unlock_asm + 0x4 } JUMP.L^M

15 Target : <0x0000cae0> { ___raw_spin_unlock_asm + 0x0 }^M

Source : <0x00131a60> { __raw_spin_unlock_irqrestore + 0x8 }

JUMP.L^M

Stack info:^M

SP: [0x0276dcf8] <0x0276dcf8> /* kernel dynamic memory (maybe

user-space) */^M

Memory from 0x0276dcf0 to 0276e000^M

0276dcf0: 0276dcf8 0276dd4c [0016254c] 0012fdf4 0034f120 0016254c 00192b4f

00192b4f ^M

0276dd10: 00192b4f 0276dd40 00198f98 0000c4d4 0034f120 00000001 002c0120

cccccccd ^M

0276dd30: 0276dd48 001a0afc 0276dd4c 00000000 00000005 0000001f 00000002

00000002 ^M

0276dd50: 00000002 00000000 0000c4f4 0031c800 020e1e34 0230f198 02c00000

02c23000 ^M

0276dd70: 02c00000 02c00000 0000c004 00192260 00000000 00052a96 00131a64

00000006 ^M

0276dd90: 00000001 00000000 00000000 00000000 00000000 00023000 00000023

00000000 ^M

0276ddb0: 0276ba80 02473800 00083de0 000100b9 00000004 0000a780 0276c000

00000005 ^M

0276ddd0: 0000a780 00002f24 02473800 00000000 00000000 00000007 00000002

00000000 ^M

0276ddf0: 00000000 00000000 00000000 00000000 0001ff72 00000006 00000004

0000069c ^M

0276de10: 00001a70 80a70000 a4d60000 00210100 00023000 00000001 00004a5c

01000000 ^M

0276de30: 3ed0f04e 000100b9 000221bd 00000000 00000000 00000000 00000000

00000000 ^M

0276de50: 00000000 00000000 00000000 00000000 80000040 02473800

<0005ae54> 001a6928 ^M

0276de70: 000836f8 001a6538 fffffff8 ffe05000 00198f98 00000000 0005adc8

0276df24 ^M

0276de90: 003113c0 02473802 00000000 02662e50 0005bc2e 02662f6c 022ec000

00000ff8 ^M

0276deb0: 00000000 02473800 02692000 00000001 00001000 0276df24 02473800

00000000 ^M

0276ded0: 00001cee 00001cb8 0000000b 02692000 022ec000 02662f6c 02457ff8

<0000b13e>^M

0276def0: 00064a14 0276def8 0276df24 <0000b13e> 00000000 ffffe000

02662fa0 02457ff3 ^M

0276df10: 00001000 020ccee0 0000003a 02457ff8 02662fa0 02648342 00008000

00000000 ^M

0276df30: 00000000 0276e000 02648342 02648342 026482aa 0000b914 02002040

02643ab9 ^M

0276df50: 02644769 02643ab8 02644766 00000000 00000000 0000147a 00000000

0000147a ^M

0276df70: 00000000 00000000 00000000 00000000 00000000 00000000 00000000

00000000 ^M

0276df90: 00000000 00000000 00000000 00000000 00000004 00000000 023d9d98

0264d3b4 ^M

0276dfb0: 02662ec0 02662e44 02662e50 02662fa4 0000002f 00000ff8 0264e228

02662f8c ^M

0276dfd0: 0000000b 00000005 02457ff3 02662fa0 02457ff8 0000003a 02662f6c

022ec000 ^M

0276dff0: 02457ff3 02457ff3 0000000b 00000006 ^M

Return addresses in stack:^M

address : <0x0005ae54> { _search_binary_handler + 0x80 }^M

address : <0x0000b13e> { _system_call + 0x6a }^M

--- steven miao 2012-08-03 06:24:36

IPI message queue has been removed now

Files

Changes

Commits

Dependencies

Duplicates

Associations