2010-02-25 08:37:05     Kernel OOPS

Document created by Aaronwu Employee on Aug 20, 2013
Version 1Show Document
  • View in full screen mode

2010-02-25 08:37:05     Kernel OOPS

Robert Langley (UNITED STATES)

Message: 86539   

 

I am NOT the software guy on this project, but I want to ask for help from this forum.

 

About once every three weeks, we have a Kernel OOPS crash on our product.  This is a big deal because it happens at a time that can cause a safety issue.  I have been able to capture several crashes from two different boards.  They are attached.  It appears to me (a lowely EE with limited software experience) that there is useful information about a null pointer.  Our software guy says there is "nothing"  in these dumps that he can use.  I would appreciate a second opinion.

 

There are four separate crashes posted below.

 

Thanks for your eyes on this problem.

 

------------------------------------------------------------------------------------------------------------------------------------

 

 

DET1[178]: EV_LOG   04007  INFORMATION       3991 348             SystemControlTask.cc 1316  0000000000 1313 CSystemControlTask::GetMessage: Prior to select, state SCT_STATE_READY waiting forever

Data access misaligned address violation

- Attempted misaligned data memory or data cache access.

Kernel OOPS in progress

Defered Exception context

CURRENT PROCESS:

COMM=grep PID=4861

TEXT = 0x033c0040-0x033e9000        DATA = 0x033e9004-0x033efa34

BSS = 0x033efa34-0x033f2ec4  USER-STACK = 0x033f7f48

 

return address: [0x0003511c]; contents of:

0x000350f0:  2e92  0000  05fc  0167  6f66  3220  3117  0037

0x00035100:  e521  0022  0c41  1824  0000  0000  a0c8  0208

0x00035110:  101f  0000  3228  a548  3200  5e85 [9110] b0c8

0x00035120:  60f8  0807  1807  e14a  0015  e10a  4ec0  9110

 

SEQUENCER STATUS:  Not tainted

SEQSTAT: 00000024  IPEND: 8030  SYSCFG: 0006

  EXCAUSE   : 0x24

  physical IVG15 asserted : <0xffa00d14> /* unknown address */

  logical irq   6 mapped  : <0xffa00178> /* unknown address */

  logical irq  13 mapped  : <0x0008afc8> /* unknown address */

  logical irq  22 mapped  : <0x0008ad04> /* unknown address */

  logical irq  27 mapped  : <0x000ceb40> /* unknown address */

  logical irq  29 mapped  : <0x00097f84> /* unknown address */

  logical irq  30 mapped  : <0x000981b8> /* unknown address */

  logical irq  31 mapped  : <0x00097f84> /* unknown address */

  logical irq  32 mapped  : <0x000981b8> /* unknown address */

  logical irq  35 mapped  : <0x0009fd18> /* unknown address */

  logical irq  59 mapped  : <0x000c7f6c> /* unknown address */

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x03101e64> /* unknown address */

RETX: <0x0003511c> /* unknown address */

RETS: <0x00032dd0> /* unknown address */

PC  : <0x0003511c> /* unknown address */

DCPLB_FAULT_ADDR: <0x0036a54c> /* unknown address */

ICPLB_FAULT_ADDR: <0x0003511c> /* unknown address */

 

PROCESSOR STATE:

R0 : 00000000    R1 : 000000d0    R2 : 00032dd0    R3 : 00000004

R4 : 00000000    R5 : 00000000    R6 : 04000021    R7 : 0000ffff

P0 : 00000000    P1 : 0036a540    P2 : 0032ae41    P3 : 00001fff

P4 : ff801a8c    P5 : 0032ae41    FP : 00000073    SP : 03101d88

LB0: ffa01abc    LT0: ffa01abc    LC0: 00000000

LB1: 033dade9    LT1: 033dadca    LC1: 00000012

B0 : 00000000    L0 : 00000000    M0 : 00000000    I0 : 033f7fec

B1 : 00000000    L1 : 00000000    M1 : 00000000    I1 : fffffbdc

B2 : 00000000    L2 : 00000000    M2 : 00000000    I2 : 00000000

B3 : 00000000    L3 : 00000000    M3 : 00000000    I3 : 00000000

A0.w: 00000000   A0.x: 00000000   A1.w: 00000000   A1.x: 00000000

USP : 033f7d28  ASTAT: 02003025

 

Stack from 03101d6c:

        03100000 ffa00658 ff8016a8 ff8016a8 00000000 007a1000 00000100 0003511c

        00008030 00000024 00000000 03101e64 0003511c 0003511c 00032dd0 00000000

        02003025 033dade9 ffa01abc 033dadca ffa01abc 00000012 00000000 00000000

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

        00000000 fffffbdc 033f7fec 033f7d28 00000073 0032ae41 ff801a8c 00001fff

 

Call Trace:

[<0000ffff>][<00032dd0>][<0005f4a0>][<0003b3f2>][<00032dd0>][<00001000>][<00001000>][<000424c6>][<0000ffff>][<000384f4>][<00002a70>][<00001000>][<00002a28>][<00008000>][<00001000>][<00001000>]<5>

Modules linked in: arusb_lnx_25

Please turn on CONFIG_ACCESS_CHECK

Kernel panic - not syncing: Kernel exception

 

 

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

 

DET1[186]: EV_LOG   02010  INFORMATION        698 547                     CHCommand.cc  555  0000000000 1827 CConsoleCommand::LogResponse: sent response ExportImage seqNum 194 state iRCLxIG status 0 resp buf PrevCor

Data access misaligned address violation

- Attempted misaligned data memory or data cache access.

Kernel OOPS in progress

Defered Exception context

CURRENT PROCESS:

COMM=grep PID=1104

TEXT = 0x03300040-0x03329000        DATA = 0x03329004-0x0332fa34

BSS = 0x0332fa34-0x03332ec4  USER-STACK = 0x03337f44

 

return address: [0x0003511c]; contents of:

0x000350f0:  2e92  0000  05fc  0167  6f66  3220  3117  0037

0x00035100:  e521  0022  0c41  1824  0000  0000  a0c8  0208

0x00035110:  101f  0000  3228  a548  3200  5e85 [9110] b0c8

0x00035120:  60f8  0807  1807  e14a  0015  e10a  4ec0  9110

 

SEQUENCER STATUS:  Not tainted

SEQSTAT: 00000024  IPEND: 8030  SYSCFG: 0006

  EXCAUSE   : 0x24

  physical IVG15 asserted : <0xffa00d14> /* unknown address */

  logical irq   6 mapped  : <0xffa00178> /* unknown address */

  logical irq  13 mapped  : <0x0008afc8> /* unknown address */

  logical irq  22 mapped  : <0x0008ad04> /* unknown address */

  logical irq  27 mapped  : <0x000ceb40> /* unknown address */

  logical irq  29 mapped  : <0x00097f84> /* unknown address */

  logical irq  30 mapped  : <0x000981b8> /* unknown address */

  logical irq  31 mapped  : <0x00097f84> /* unknown address */

  logical irq  32 mapped  : <0x000981b8> /* unknown address */

  logical irq  35 mapped  : <0x0009fd18> /* unknown address */

  logical irq  59 mapped  : <0x000c7f6c> /* unknown address */

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x03217e64> /* unknown address */

RETX: <0x0003511c> /* unknown address */

RETS: <0x00032dd0> /* unknown address */

PC  : <0x0003511c> /* unknown address */

DCPLB_FAULT_ADDR: <0x0036a54c> /* unknown address */

ICPLB_FAULT_ADDR: <0x0003511c> /* unknown address */

 

PROCESSOR STATE:

R0 : 00000000    R1 : 000000d0    R2 : 00032dd0    R3 : 00000004

R4 : 00000000    R5 : 00000000    R6 : 04000021    R7 : 0000ffff

P0 : 00000000    P1 : 0036a540    P2 : 0032a3c1    P3 : 00001fff

P4 : ff801a8c    P5 : 0032a3c1    FP : 00000073    SP : 03217d88

LB0: ffa01abc    LT0: ffa01abc    LC0: 00000000

LB1: 0331ade9    LT1: 0331adca    LC1: 00000012

B0 : 00000000    L0 : 00000000    M0 : 00000000    I0 : 03337fe9

B1 : 00000000    L1 : 00000000    M1 : 00000000    I1 : fffffbdc

B2 : 00000000    L2 : 00000000    M2 : 00000000    I2 : 00000000

B3 : 00000000    L3 : 00000000    M3 : 00000000    I3 : 00000000

A0.w: 00000000   A0.x: 00000000   A1.w: 00000000   A1.x: 00000000

USP : 03337d24  ASTAT: 02003025

 

Stack from 03217d6c:

        03216000 ffa00658 ff8016a8 ff8016a8 00000000 00000000 00000000 0003511c

        00008030 00000024 00000000 03217e64 0003511c 0003511c 00032dd0 00000000

        02003025 0331ade9 ffa01abc 0331adca ffa01abc 00000012 00000000 00000000

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

        00000000 fffffbdc 03337fe9 03337d24 00000073 0032a3c1 ff801a8c 00001fff

 

Call Trace:

[<0000ffff>][<00032dd0>][<0001ab6c>][<00032dd0>][<00001000>][<00001000>][<00010b9c>][<00002a70>][<00001000>][<00010e6e>][<00002a28>][<00008000>][<00001000>][<00001000>]<5>

Modules linked in: arusb_lnx_25

Please turn on CONFIG_ACCESS_CHECK

Kernel panic - not syncing: Kernel exception

 

----------------------------------------------------------------------------------------------------------------------------------------------------

 

 

DET1[178]: EV_LOG   04007  INFORMATION       3991 348             SystemControlTask.cc 1316  0000000000 1313 CSystemControlTask::GetMessage: Prior to select, state SCT_STATE_READY waiting forever

Data access misaligned address violation

- Attempted misaligned data memory or data cache access.

Kernel OOPS in progress

Defered Exception context

CURRENT PROCESS:

COMM=grep PID=4861

TEXT = 0x033c0040-0x033e9000        DATA = 0x033e9004-0x033efa34

BSS = 0x033efa34-0x033f2ec4  USER-STACK = 0x033f7f48

 

return address: [0x0003511c]; contents of:

0x000350f0:  2e92  0000  05fc  0167  6f66  3220  3117  0037

0x00035100:  e521  0022  0c41  1824  0000  0000  a0c8  0208

0x00035110:  101f  0000  3228  a548  3200  5e85 [9110] b0c8

0x00035120:  60f8  0807  1807  e14a  0015  e10a  4ec0  9110

 

SEQUENCER STATUS:  Not tainted

SEQSTAT: 00000024  IPEND: 8030  SYSCFG: 0006

  EXCAUSE   : 0x24

  physical IVG15 asserted : <0xffa00d14> /* unknown address */

  logical irq   6 mapped  : <0xffa00178> /* unknown address */

  logical irq  13 mapped  : <0x0008afc8> /* unknown address */

  logical irq  22 mapped  : <0x0008ad04> /* unknown address */

  logical irq  27 mapped  : <0x000ceb40> /* unknown address */

  logical irq  29 mapped  : <0x00097f84> /* unknown address */

  logical irq  30 mapped  : <0x000981b8> /* unknown address */

  logical irq  31 mapped  : <0x00097f84> /* unknown address */

  logical irq  32 mapped  : <0x000981b8> /* unknown address */

  logical irq  35 mapped  : <0x0009fd18> /* unknown address */

  logical irq  59 mapped  : <0x000c7f6c> /* unknown address */

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x03101e64> /* unknown address */

RETX: <0x0003511c> /* unknown address */

RETS: <0x00032dd0> /* unknown address */

PC  : <0x0003511c> /* unknown address */

DCPLB_FAULT_ADDR: <0x0036a54c> /* unknown address */

ICPLB_FAULT_ADDR: <0x0003511c> /* unknown address */

 

PROCESSOR STATE:

R0 : 00000000    R1 : 000000d0    R2 : 00032dd0    R3 : 00000004

R4 : 00000000    R5 : 00000000    R6 : 04000021    R7 : 0000ffff

P0 : 00000000    P1 : 0036a540    P2 : 0032ae41    P3 : 00001fff

P4 : ff801a8c    P5 : 0032ae41    FP : 00000073    SP : 03101d88

LB0: ffa01abc    LT0: ffa01abc    LC0: 00000000

LB1: 033dade9    LT1: 033dadca    LC1: 00000012

B0 : 00000000    L0 : 00000000    M0 : 00000000    I0 : 033f7fec

B1 : 00000000    L1 : 00000000    M1 : 00000000    I1 : fffffbdc

B2 : 00000000    L2 : 00000000    M2 : 00000000    I2 : 00000000

B3 : 00000000    L3 : 00000000    M3 : 00000000    I3 : 00000000

A0.w: 00000000   A0.x: 00000000   A1.w: 00000000   A1.x: 00000000

USP : 033f7d28  ASTAT: 02003025

 

Stack from 03101d6c:

        03100000 ffa00658 ff8016a8 ff8016a8 00000000 007a1000 00000100 0003511c

        00008030 00000024 00000000 03101e64 0003511c 0003511c 00032dd0 00000000

        02003025 033dade9 ffa01abc 033dadca ffa01abc 00000012 00000000 00000000

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

        00000000 fffffbdc 033f7fec 033f7d28 00000073 0032ae41 ff801a8c 00001fff

 

Call Trace:

[<0000ffff>][<00032dd0>][<0005f4a0>][<0003b3f2>][<00032dd0>][<00001000>][<00001000>][<000424c6>][<0000ffff>][<000384f4>][<00002a70>][<00001000>][<00002a28>][<00008000>][<00001000>][<00001000>]<5>

Modules linked in: arusb_lnx_25

Please turn on CONFIG_ACCESS_CHECK

Kernel panic - not syncing: Kernel exception

 

 

---------------------  Below is from a second board.   All of the above were from a single board... -----------------------------------------

 

 

DET1[178]: EV_LOG   04007  INFORMATION  752059939 722             SystemControlTask.cc 1326  0000000000 5039 CSystemControlTask::GetMessage: Prior to select, state SCT_STATE_DISARMING duration 5.000000

DET1[178]: EData access misaligned address violation

- Attempted misaligned data memory or data cache access.

Kernel OOPS in progress

Defered Exception context

CURRENT PROCESS:

COMM=grep PID=5980

TEXT = 0x02100040-0x02129000        DATA = 0x02129004-0x0212fa34

BSS = 0x0212fa34-0x02132ec4  USER-STACK = 0x02137f44

 

return address: [0x0003511c]; contents of:

0x000350f0:  2e92  0000  05fc  0167  6f66  3220  3117  0037

0x00035100:  e521  0022  0c41  1824  0000  0000  a0c8  0208

0x00035110:  101f  0000  3228  a548  3200  5e85 [9110] b0c8

0x00035120:  60f8  0807  1807  e14a  0015  e10a  4ec0  9110

 

SEQUENCER STATUS:  Not tainted

SEQSTAT: 00000024  IPEND: 8030  SYSCFG: 0006

  EXCAUSE   : 0x24

  physical IVG15 asserted : <0xffa00d14> /* unknown address */

  logical irq   6 mapped  : <0xffa00178> /* unknown address */

  logical irq  13 mapped  : <0x0008afc8> /* unknown address */

  logical irq  22 mapped  : <0x0008ad04> /* unknown address */

  logical irq  27 mapped  : <0x000ceb40> /* unknown address */

  logical irq  29 mapped  : <0x00097f84> /* unknown address */

  logical irq  30 mapped  : <0x000981b8> /* unknown address */

  logical irq  31 mapped  : <0x00097f84> /* unknown address */

  logical irq  32 mapped  : <0x000981b8> /* unknown address */

  logical irq  35 mapped  : <0x0009fd18> /* unknown address */

  logical irq  59 mapped  : <0x000c7f6c> /* unknown address */

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x03107e64> /* unknown address */

RETX: <0x0003511c> /* unknown address */

RETS: <0x00032dd0> /* unknown address */

PC  : <0x0003511c> /* unknown address */

DCPLB_FAULT_ADDR: <0x0036a54c> /* unknown address */

ICPLB_FAULT_ADDR: <0x0003511c> /* unknown address */

 

PROCESSOR STATE:

R0 : 00000000    R1 : 000000d0    R2 : 00032dd0    R3 : 00000004

R4 : 00000000    R5 : 00000000    R6 : 04000021    R7 : 0000ffff

P0 : 00000000    P1 : 0036a540    P2 : 0032ae41    P3 : 00001fff

P4 : ff801a8c    P5 : 0032ae41    FP : 00000073    SP : 03107d88

LB0: ffa01abc    LT0: ffa01abc    LC0: 00000000

LB1: 0211ade9    LT1: 0211adca    LC1: 00000012

B0 : 00000000    L0 : 00000000    M0 : 00000000    I0 : 02137fec

B1 : 00000000    L1 : 00000000    M1 : 00000000    I1 : fffffbdc

B2 : 00000000    L2 : 00000000    M2 : 00000000    I2 : 00000000

B3 : 00000000    L3 : 00000000    M3 : 00000000    I3 : 00000000

A0.w: 00000000   A0.x: 00000000   A1.w: 00000000   A1.x: 00000000

USP : 02137d24  ASTAT: 02003025

 

Stack from 03107d6c:

        03106000 ffa00658 ff8016a8 ff8016a8 00000000 03104000 0000002f 0003511c

        00008030 00000024 00000000 03107e64 0003511c 0003511c 00032dd0 00000000

        02003025 0211ade9 ffa01abc 0211adca ffa01abc 00000012 00000000 00000000

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

        00000000 fffffbdc 02137fec 02137d24 00000073 0032ae41 ff801a8c 00001fff

 

Call Trace:

[<0000ffff>][<00032dd0>][<0005f4a0>][<0003b3f2>][<00032dd0>][<00001000>][<00001000>][<000424c6>][<0000ffff>][<000384f4>][<00002a70>][<00001000>][<00002a28>][<00008000>][<00001000>][<00001000>]<5>

Modules linked in: arusb_lnx_25

Please turn on CONFIG_ACCESS_CHECK

Kernel panic - not syncing: Kernel exception

QuoteReplyEditDelete

 

 

2010-02-25 09:12:28     Kernel OOPS

Michael Hennerich (GERMANY)

Message: 86540    Your kernel is configured without useful debug information:

It might be useful to turn on more debug information.

 

From the dumps you can see that the crash always happens in the grep application.

 

COMM=grep

 

Data access misaligned address violation

DCPLB_FAULT_ADDR: <0x0036a54c> /* unknown address */

 

Very likely a 32-bit word access on a 16-bit address stored in P1+C

 

-Michael

QuoteReplyEditDelete

 

 

2010-02-25 09:19:44     Re: Kernel OOPS

Michael Hennerich (GERMANY)

Message: 86541   

 

COMM=grep seems to be always running - but looking at the crash address it looks more like a kernel address.

 

ICPLB_FAULT_ADDR: <0x0003511c> /* unknown address */

 

 

 

Take a look at your System.map file and look up the symbol for 0x0003511c.

 

 

 

-Michael

QuoteReplyEditDelete

 

 

2010-02-25 12:41:39     Re: Kernel OOPS

Robert Langley (UNITED STATES)

Message: 86545   

 

How do I turn on the advanced debugging?

QuoteReplyEditDelete

 

 

2010-02-25 13:09:48     Re: Kernel OOPS

Robin Getz (UNITED STATES)

Message: 86548   

 

Robert:

 

re-build the kernel with CONFIG_KALLSYMS set.

 

Also - it would be nice to know what kernel version (svn or release?) your product is based on.

 

Thanks

 

-Robin

QuoteReplyEditDelete

 

 

2010-02-26 06:54:28     Re: Kernel OOPS

Robin Getz (UNITED STATES)

Message: 86604   

 

Also:

 

>Please turn on CONFIG_ACCESS_CHECK

 

Did you do that? And when that is on -do you still get the error?

 

-Robin

Attachments

    Outcomes