2010-02-25 08:37:05 Kernel OOPS
Robert Langley (UNITED STATES)
Message: 86539
I am NOT the software guy on this project, but I want to ask for help from this forum.
About once every three weeks, we have a Kernel OOPS crash on our product. This is a big deal because it happens at a time that can cause a safety issue. I have been able to capture several crashes from two different boards. They are attached. It appears to me (a lowely EE with limited software experience) that there is useful information about a null pointer. Our software guy says there is "nothing" in these dumps that he can use. I would appreciate a second opinion.
There are four separate crashes posted below.
Thanks for your eyes on this problem.
------------------------------------------------------------------------------------------------------------------------------------
DET1[178]: EV_LOG 04007 INFORMATION 3991 348 SystemControlTask.cc 1316 0000000000 1313 CSystemControlTask::GetMessage: Prior to select, state SCT_STATE_READY waiting forever
Data access misaligned address violation
- Attempted misaligned data memory or data cache access.
Kernel OOPS in progress
Defered Exception context
CURRENT PROCESS:
COMM=grep PID=4861
TEXT = 0x033c0040-0x033e9000 DATA = 0x033e9004-0x033efa34
BSS = 0x033efa34-0x033f2ec4 USER-STACK = 0x033f7f48
return address: [0x0003511c]; contents of:
0x000350f0: 2e92 0000 05fc 0167 6f66 3220 3117 0037
0x00035100: e521 0022 0c41 1824 0000 0000 a0c8 0208
0x00035110: 101f 0000 3228 a548 3200 5e85 [9110] b0c8
0x00035120: 60f8 0807 1807 e14a 0015 e10a 4ec0 9110
SEQUENCER STATUS: Not tainted
SEQSTAT: 00000024 IPEND: 8030 SYSCFG: 0006
EXCAUSE : 0x24
physical IVG15 asserted : <0xffa00d14> /* unknown address */
logical irq 6 mapped : <0xffa00178> /* unknown address */
logical irq 13 mapped : <0x0008afc8> /* unknown address */
logical irq 22 mapped : <0x0008ad04> /* unknown address */
logical irq 27 mapped : <0x000ceb40> /* unknown address */
logical irq 29 mapped : <0x00097f84> /* unknown address */
logical irq 30 mapped : <0x000981b8> /* unknown address */
logical irq 31 mapped : <0x00097f84> /* unknown address */
logical irq 32 mapped : <0x000981b8> /* unknown address */
logical irq 35 mapped : <0x0009fd18> /* unknown address */
logical irq 59 mapped : <0x000c7f6c> /* unknown address */
RETE: <0x00000000> /* Maybe null pointer? */
RETN: <0x03101e64> /* unknown address */
RETX: <0x0003511c> /* unknown address */
RETS: <0x00032dd0> /* unknown address */
PC : <0x0003511c> /* unknown address */
DCPLB_FAULT_ADDR: <0x0036a54c> /* unknown address */
ICPLB_FAULT_ADDR: <0x0003511c> /* unknown address */
PROCESSOR STATE:
R0 : 00000000 R1 : 000000d0 R2 : 00032dd0 R3 : 00000004
R4 : 00000000 R5 : 00000000 R6 : 04000021 R7 : 0000ffff
P0 : 00000000 P1 : 0036a540 P2 : 0032ae41 P3 : 00001fff
P4 : ff801a8c P5 : 0032ae41 FP : 00000073 SP : 03101d88
LB0: ffa01abc LT0: ffa01abc LC0: 00000000
LB1: 033dade9 LT1: 033dadca LC1: 00000012
B0 : 00000000 L0 : 00000000 M0 : 00000000 I0 : 033f7fec
B1 : 00000000 L1 : 00000000 M1 : 00000000 I1 : fffffbdc
B2 : 00000000 L2 : 00000000 M2 : 00000000 I2 : 00000000
B3 : 00000000 L3 : 00000000 M3 : 00000000 I3 : 00000000
A0.w: 00000000 A0.x: 00000000 A1.w: 00000000 A1.x: 00000000
USP : 033f7d28 ASTAT: 02003025
Stack from 03101d6c:
03100000 ffa00658 ff8016a8 ff8016a8 00000000 007a1000 00000100 0003511c
00008030 00000024 00000000 03101e64 0003511c 0003511c 00032dd0 00000000
02003025 033dade9 ffa01abc 033dadca ffa01abc 00000012 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 fffffbdc 033f7fec 033f7d28 00000073 0032ae41 ff801a8c 00001fff
Call Trace:
[<0000ffff>][<00032dd0>][<0005f4a0>][<0003b3f2>][<00032dd0>][<00001000>][<00001000>][<000424c6>][<0000ffff>][<000384f4>][<00002a70>][<00001000>][<00002a28>][<00008000>][<00001000>][<00001000>]<5>
Modules linked in: arusb_lnx_25
Please turn on CONFIG_ACCESS_CHECK
Kernel panic - not syncing: Kernel exception
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
DET1[186]: EV_LOG 02010 INFORMATION 698 547 CHCommand.cc 555 0000000000 1827 CConsoleCommand::LogResponse: sent response ExportImage seqNum 194 state iRCLxIG status 0 resp buf PrevCor
Data access misaligned address violation
- Attempted misaligned data memory or data cache access.
Kernel OOPS in progress
Defered Exception context
CURRENT PROCESS:
COMM=grep PID=1104
TEXT = 0x03300040-0x03329000 DATA = 0x03329004-0x0332fa34
BSS = 0x0332fa34-0x03332ec4 USER-STACK = 0x03337f44
return address: [0x0003511c]; contents of:
0x000350f0: 2e92 0000 05fc 0167 6f66 3220 3117 0037
0x00035100: e521 0022 0c41 1824 0000 0000 a0c8 0208
0x00035110: 101f 0000 3228 a548 3200 5e85 [9110] b0c8
0x00035120: 60f8 0807 1807 e14a 0015 e10a 4ec0 9110
SEQUENCER STATUS: Not tainted
SEQSTAT: 00000024 IPEND: 8030 SYSCFG: 0006
EXCAUSE : 0x24
physical IVG15 asserted : <0xffa00d14> /* unknown address */
logical irq 6 mapped : <0xffa00178> /* unknown address */
logical irq 13 mapped : <0x0008afc8> /* unknown address */
logical irq 22 mapped : <0x0008ad04> /* unknown address */
logical irq 27 mapped : <0x000ceb40> /* unknown address */
logical irq 29 mapped : <0x00097f84> /* unknown address */
logical irq 30 mapped : <0x000981b8> /* unknown address */
logical irq 31 mapped : <0x00097f84> /* unknown address */
logical irq 32 mapped : <0x000981b8> /* unknown address */
logical irq 35 mapped : <0x0009fd18> /* unknown address */
logical irq 59 mapped : <0x000c7f6c> /* unknown address */
RETE: <0x00000000> /* Maybe null pointer? */
RETN: <0x03217e64> /* unknown address */
RETX: <0x0003511c> /* unknown address */
RETS: <0x00032dd0> /* unknown address */
PC : <0x0003511c> /* unknown address */
DCPLB_FAULT_ADDR: <0x0036a54c> /* unknown address */
ICPLB_FAULT_ADDR: <0x0003511c> /* unknown address */
PROCESSOR STATE:
R0 : 00000000 R1 : 000000d0 R2 : 00032dd0 R3 : 00000004
R4 : 00000000 R5 : 00000000 R6 : 04000021 R7 : 0000ffff
P0 : 00000000 P1 : 0036a540 P2 : 0032a3c1 P3 : 00001fff
P4 : ff801a8c P5 : 0032a3c1 FP : 00000073 SP : 03217d88
LB0: ffa01abc LT0: ffa01abc LC0: 00000000
LB1: 0331ade9 LT1: 0331adca LC1: 00000012
B0 : 00000000 L0 : 00000000 M0 : 00000000 I0 : 03337fe9
B1 : 00000000 L1 : 00000000 M1 : 00000000 I1 : fffffbdc
B2 : 00000000 L2 : 00000000 M2 : 00000000 I2 : 00000000
B3 : 00000000 L3 : 00000000 M3 : 00000000 I3 : 00000000
A0.w: 00000000 A0.x: 00000000 A1.w: 00000000 A1.x: 00000000
USP : 03337d24 ASTAT: 02003025
Stack from 03217d6c:
03216000 ffa00658 ff8016a8 ff8016a8 00000000 00000000 00000000 0003511c
00008030 00000024 00000000 03217e64 0003511c 0003511c 00032dd0 00000000
02003025 0331ade9 ffa01abc 0331adca ffa01abc 00000012 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 fffffbdc 03337fe9 03337d24 00000073 0032a3c1 ff801a8c 00001fff
Call Trace:
[<0000ffff>][<00032dd0>][<0001ab6c>][<00032dd0>][<00001000>][<00001000>][<00010b9c>][<00002a70>][<00001000>][<00010e6e>][<00002a28>][<00008000>][<00001000>][<00001000>]<5>
Modules linked in: arusb_lnx_25
Please turn on CONFIG_ACCESS_CHECK
Kernel panic - not syncing: Kernel exception
----------------------------------------------------------------------------------------------------------------------------------------------------
DET1[178]: EV_LOG 04007 INFORMATION 3991 348 SystemControlTask.cc 1316 0000000000 1313 CSystemControlTask::GetMessage: Prior to select, state SCT_STATE_READY waiting forever
Data access misaligned address violation
- Attempted misaligned data memory or data cache access.
Kernel OOPS in progress
Defered Exception context
CURRENT PROCESS:
COMM=grep PID=4861
TEXT = 0x033c0040-0x033e9000 DATA = 0x033e9004-0x033efa34
BSS = 0x033efa34-0x033f2ec4 USER-STACK = 0x033f7f48
return address: [0x0003511c]; contents of:
0x000350f0: 2e92 0000 05fc 0167 6f66 3220 3117 0037
0x00035100: e521 0022 0c41 1824 0000 0000 a0c8 0208
0x00035110: 101f 0000 3228 a548 3200 5e85 [9110] b0c8
0x00035120: 60f8 0807 1807 e14a 0015 e10a 4ec0 9110
SEQUENCER STATUS: Not tainted
SEQSTAT: 00000024 IPEND: 8030 SYSCFG: 0006
EXCAUSE : 0x24
physical IVG15 asserted : <0xffa00d14> /* unknown address */
logical irq 6 mapped : <0xffa00178> /* unknown address */
logical irq 13 mapped : <0x0008afc8> /* unknown address */
logical irq 22 mapped : <0x0008ad04> /* unknown address */
logical irq 27 mapped : <0x000ceb40> /* unknown address */
logical irq 29 mapped : <0x00097f84> /* unknown address */
logical irq 30 mapped : <0x000981b8> /* unknown address */
logical irq 31 mapped : <0x00097f84> /* unknown address */
logical irq 32 mapped : <0x000981b8> /* unknown address */
logical irq 35 mapped : <0x0009fd18> /* unknown address */
logical irq 59 mapped : <0x000c7f6c> /* unknown address */
RETE: <0x00000000> /* Maybe null pointer? */
RETN: <0x03101e64> /* unknown address */
RETX: <0x0003511c> /* unknown address */
RETS: <0x00032dd0> /* unknown address */
PC : <0x0003511c> /* unknown address */
DCPLB_FAULT_ADDR: <0x0036a54c> /* unknown address */
ICPLB_FAULT_ADDR: <0x0003511c> /* unknown address */
PROCESSOR STATE:
R0 : 00000000 R1 : 000000d0 R2 : 00032dd0 R3 : 00000004
R4 : 00000000 R5 : 00000000 R6 : 04000021 R7 : 0000ffff
P0 : 00000000 P1 : 0036a540 P2 : 0032ae41 P3 : 00001fff
P4 : ff801a8c P5 : 0032ae41 FP : 00000073 SP : 03101d88
LB0: ffa01abc LT0: ffa01abc LC0: 00000000
LB1: 033dade9 LT1: 033dadca LC1: 00000012
B0 : 00000000 L0 : 00000000 M0 : 00000000 I0 : 033f7fec
B1 : 00000000 L1 : 00000000 M1 : 00000000 I1 : fffffbdc
B2 : 00000000 L2 : 00000000 M2 : 00000000 I2 : 00000000
B3 : 00000000 L3 : 00000000 M3 : 00000000 I3 : 00000000
A0.w: 00000000 A0.x: 00000000 A1.w: 00000000 A1.x: 00000000
USP : 033f7d28 ASTAT: 02003025
Stack from 03101d6c:
03100000 ffa00658 ff8016a8 ff8016a8 00000000 007a1000 00000100 0003511c
00008030 00000024 00000000 03101e64 0003511c 0003511c 00032dd0 00000000
02003025 033dade9 ffa01abc 033dadca ffa01abc 00000012 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 fffffbdc 033f7fec 033f7d28 00000073 0032ae41 ff801a8c 00001fff
Call Trace:
[<0000ffff>][<00032dd0>][<0005f4a0>][<0003b3f2>][<00032dd0>][<00001000>][<00001000>][<000424c6>][<0000ffff>][<000384f4>][<00002a70>][<00001000>][<00002a28>][<00008000>][<00001000>][<00001000>]<5>
Modules linked in: arusb_lnx_25
Please turn on CONFIG_ACCESS_CHECK
Kernel panic - not syncing: Kernel exception
--------------------- Below is from a second board. All of the above were from a single board... -----------------------------------------
DET1[178]: EV_LOG 04007 INFORMATION 752059939 722 SystemControlTask.cc 1326 0000000000 5039 CSystemControlTask::GetMessage: Prior to select, state SCT_STATE_DISARMING duration 5.000000
DET1[178]: EData access misaligned address violation
- Attempted misaligned data memory or data cache access.
Kernel OOPS in progress
Defered Exception context
CURRENT PROCESS:
COMM=grep PID=5980
TEXT = 0x02100040-0x02129000 DATA = 0x02129004-0x0212fa34
BSS = 0x0212fa34-0x02132ec4 USER-STACK = 0x02137f44
return address: [0x0003511c]; contents of:
0x000350f0: 2e92 0000 05fc 0167 6f66 3220 3117 0037
0x00035100: e521 0022 0c41 1824 0000 0000 a0c8 0208
0x00035110: 101f 0000 3228 a548 3200 5e85 [9110] b0c8
0x00035120: 60f8 0807 1807 e14a 0015 e10a 4ec0 9110
SEQUENCER STATUS: Not tainted
SEQSTAT: 00000024 IPEND: 8030 SYSCFG: 0006
EXCAUSE : 0x24
physical IVG15 asserted : <0xffa00d14> /* unknown address */
logical irq 6 mapped : <0xffa00178> /* unknown address */
logical irq 13 mapped : <0x0008afc8> /* unknown address */
logical irq 22 mapped : <0x0008ad04> /* unknown address */
logical irq 27 mapped : <0x000ceb40> /* unknown address */
logical irq 29 mapped : <0x00097f84> /* unknown address */
logical irq 30 mapped : <0x000981b8> /* unknown address */
logical irq 31 mapped : <0x00097f84> /* unknown address */
logical irq 32 mapped : <0x000981b8> /* unknown address */
logical irq 35 mapped : <0x0009fd18> /* unknown address */
logical irq 59 mapped : <0x000c7f6c> /* unknown address */
RETE: <0x00000000> /* Maybe null pointer? */
RETN: <0x03107e64> /* unknown address */
RETX: <0x0003511c> /* unknown address */
RETS: <0x00032dd0> /* unknown address */
PC : <0x0003511c> /* unknown address */
DCPLB_FAULT_ADDR: <0x0036a54c> /* unknown address */
ICPLB_FAULT_ADDR: <0x0003511c> /* unknown address */
PROCESSOR STATE:
R0 : 00000000 R1 : 000000d0 R2 : 00032dd0 R3 : 00000004
R4 : 00000000 R5 : 00000000 R6 : 04000021 R7 : 0000ffff
P0 : 00000000 P1 : 0036a540 P2 : 0032ae41 P3 : 00001fff
P4 : ff801a8c P5 : 0032ae41 FP : 00000073 SP : 03107d88
LB0: ffa01abc LT0: ffa01abc LC0: 00000000
LB1: 0211ade9 LT1: 0211adca LC1: 00000012
B0 : 00000000 L0 : 00000000 M0 : 00000000 I0 : 02137fec
B1 : 00000000 L1 : 00000000 M1 : 00000000 I1 : fffffbdc
B2 : 00000000 L2 : 00000000 M2 : 00000000 I2 : 00000000
B3 : 00000000 L3 : 00000000 M3 : 00000000 I3 : 00000000
A0.w: 00000000 A0.x: 00000000 A1.w: 00000000 A1.x: 00000000
USP : 02137d24 ASTAT: 02003025
Stack from 03107d6c:
03106000 ffa00658 ff8016a8 ff8016a8 00000000 03104000 0000002f 0003511c
00008030 00000024 00000000 03107e64 0003511c 0003511c 00032dd0 00000000
02003025 0211ade9 ffa01abc 0211adca ffa01abc 00000012 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 fffffbdc 02137fec 02137d24 00000073 0032ae41 ff801a8c 00001fff
Call Trace:
[<0000ffff>][<00032dd0>][<0005f4a0>][<0003b3f2>][<00032dd0>][<00001000>][<00001000>][<000424c6>][<0000ffff>][<000384f4>][<00002a70>][<00001000>][<00002a28>][<00008000>][<00001000>][<00001000>]<5>
Modules linked in: arusb_lnx_25
Please turn on CONFIG_ACCESS_CHECK
Kernel panic - not syncing: Kernel exception
QuoteReplyEditDelete
2010-02-25 09:12:28 Kernel OOPS
Michael Hennerich (GERMANY)
Message: 86540 Your kernel is configured without useful debug information:
It might be useful to turn on more debug information.
From the dumps you can see that the crash always happens in the grep application.
COMM=grep
Data access misaligned address violation
DCPLB_FAULT_ADDR: <0x0036a54c> /* unknown address */
Very likely a 32-bit word access on a 16-bit address stored in P1+C
-Michael
QuoteReplyEditDelete
2010-02-25 09:19:44 Re: Kernel OOPS
Michael Hennerich (GERMANY)
Message: 86541
COMM=grep seems to be always running - but looking at the crash address it looks more like a kernel address.
ICPLB_FAULT_ADDR: <0x0003511c> /* unknown address */
Take a look at your System.map file and look up the symbol for 0x0003511c.
-Michael
QuoteReplyEditDelete
2010-02-25 12:41:39 Re: Kernel OOPS
Robert Langley (UNITED STATES)
Message: 86545
How do I turn on the advanced debugging?
QuoteReplyEditDelete
2010-02-25 13:09:48 Re: Kernel OOPS
Robin Getz (UNITED STATES)
Message: 86548
Robert:
re-build the kernel with CONFIG_KALLSYMS set.
Also - it would be nice to know what kernel version (svn or release?) your product is based on.
Thanks
-Robin
QuoteReplyEditDelete
2010-02-26 06:54:28 Re: Kernel OOPS
Robin Getz (UNITED STATES)
Message: 86604
Also:
>Please turn on CONFIG_ACCESS_CHECK
Did you do that? And when that is on -do you still get the error?
-Robin