2009-02-06 08:25:46 2008R1-RC8 crash in BF537
supriya d (INDIA)
Message: 68989
Hi,
We have ported application (working on 2006R1) to 2008R1. When we execute this application we always see a crash after some time. The crash is always a DCPLB miss on a L1 Program memory, which is strange.The crash varies and looks like below pasted Log1 and Log 2.
Both the logs shows that the crash the following lines are common..
==================
[<001d5024>] _process_backlog+0x78/0x148
[<001d5162>] _net_rx_action+0x6e/0x108
[<00111114>] ___do_softirq+0x60/0xac
==================
The logs follow...
Log 1:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Data access CPLB protection violation
- Attempted read or write to Supervisor resource,
or illegal data memory access.
Kernel OOPS in progress
Defered Exception context
CURRENT PROCESS:
COMM=adiVP PID=47
TEXT = 0x00800000-0x00b4c28c DATA = 0x00c0028c-0x00c92040
BSS = 0x00c92040-0x01380000 USER-STACK = 0x0139fed0
return address: [0x001d3756]; contents of:
0x001d3730: e102 0dd4 5010 3210 3030 9155 e12a 0094
0x001d3740: 5ad4 6f45 2003 adad 6f45 3045 64c0 0830
0x001d3750: 1827 0000 0000 [9528] 0807 17f6 a069 0c01
0x001d3760: 1806 0000 0000 a160 0801 17ee 0c41 1816
SEQUENCER STATUS: Not tainted
SEQSTAT: 00000023 IPEND: c030 SYSCFG: 0006
HWERRCAUSE: 0x0
EXCAUSE : 0x23
physical IVG14 asserted : <0xffa00b18> { _evt14_softirq + 0x0 }
physical IVG15 asserted : <0xffa01238> { _evt_system_call + 0x0 }
logical irq 6 mapped : <0xffa00280> { _timer_interrupt + 0x0 }
logical irq 12 mapped : <0xffa02784> { _AudioRxPPI_ERR + 0x0 }
logical irq 18 mapped : <0x001a5acc> { _bfin_serial_dma_rx_int + 0x0 }
logical irq 19 mapped : <0x001a567c> { _bfin_serial_dma_tx_int + 0x0 }
logical irq 24 mapped : <0x001b4990> { _bf537mac_interrupt + 0x0 }
logical irq 40 mapped : <0x001beab8> { _AudioDataISR + 0x0 }
RETE: <0x00000000> /* Maybe null pointer? */
RETN: <0x007cfc64> /* unknown address */
RETX: <0x001d3756> { _netif_receive_skb + 0x136 }
RETS: <0x001d5024> { _process_backlog + 0x78 }
PC : <0x001d3756> { _netif_receive_skb + 0x136 }
DCPLB_FAULT_ADDR: <0xffa02778> { _AudioTXVideoRx + 0x0 }
ICPLB_FAULT_ADDR: <0x001d3756> { _netif_receive_skb + 0x136 }
PROCESSOR STATE:
R0 : ffa02790 R1 : 00000800 R2 : 002b0dd4 R3 : 0000000f
R4 : 00000040 R5 : 0144d800 R6 : 002b0dd4 R7 : 00000008
P0 : 002b0a30 P1 : 00000000 P2 : 00000094 P3 : 0145dab4
P4 : 0145da20 P5 : ffa02778 FP : 007cfc74 SP : 007cfb88
LB0: ffa02308 LT0: ffa02306 LC0: 00000000
LB1: 00f5e077 LT1: 00f5e076 LC1: 00000000
B0 : 00000082 L0 : 00000000 M0 : 00000008 I0 : 00f79ba0
B1 : 00000006 L1 : 00000000 M1 : 00c056b7 I1 : 00000000
B2 : ffffff80 L2 : 00000000 M2 : 00000000 I2 : 00000000
B3 : 00000001 L3 : 00000000 M3 : 00000000 I3 : 00000008
A0.w: 00000000 A0.x: 00000000 A1.w: 00000000 A1.x: 00000000
USP : 007d2c94 ASTAT: 02003006
Hardware Trace:
0 Target : <0x00103828> { _trap_c + 0x0 }
Source : <0xffa0079c> { _exception_to_level5 + 0xb4 }
1 Target : <0xffa006e8> { _exception_to_level5 + 0x0 }
Source : <0xffa00644> { _ex_trap_c + 0x5c }
2 Target : <0xffa0058c> { _handle_bad_cplb + 0x0 }
Source : <0xffa0161c> { __cplb_hdr + 0x5c }
3 Target : <0xffa0161a> { __cplb_hdr + 0x5a }
Source : <0x00103752> { _panic_cplb_error + 0x16 }
4 Target : <0x0010373c> { _panic_cplb_error + 0x0 }
Source : <0xffa01616> { __cplb_hdr + 0x56 }
5 Target : <0xffa01612> { __cplb_hdr + 0x52 }
Source : <0xffa0160a> { __cplb_hdr + 0x4a }
6 Target : <0xffa01604> { __cplb_hdr + 0x44 }
Source : <0xffa015fe> { __cplb_hdr + 0x3e }
7 Target : <0xffa015fc> { __cplb_hdr + 0x3c }
Source : <0xffa015f6> { __cplb_hdr + 0x36 }
8 Target : <0xffa015f4> { __cplb_hdr + 0x34 }
Source : <0xffa015ec> { __cplb_hdr + 0x2c }
9 Target : <0xffa015e8> { __cplb_hdr + 0x28 }
Source : <0xffa01780> { _cplb_mgr + 0x160 }
10 Target : <0xffa0176c> { _cplb_mgr + 0x14c }
Source : <0xffa01796> { _cplb_mgr + 0x176 }
11 Target : <0xffa01782> { _cplb_mgr + 0x162 }
Source : <0xffa01624> { _cplb_mgr + 0x4 }
12 Target : <0xffa01620> { _cplb_mgr + 0x0 }
Source : <0xffa015e4> { __cplb_hdr + 0x24 }
13 Target : <0xffa015e0> { __cplb_hdr + 0x20 }
Source : <0xffa015ce> { __cplb_hdr + 0xe }
14 Target : <0xffa015c0> { __cplb_hdr + 0x0 }
Source : <0xffa004d8> { _ex_dcplb_miss + 0x64 }
15 Target : <0xffa00474> { _ex_dcplb_miss + 0x0 }
Source : <0xffa00470> { _ex_workaround_261 + 0x20 }
Stack from 007cfb68:
ff1a790a ffa007a0 00274630 00274630 0027462c ff1a790a 00000011 00201b36
001d3756 0000c030 00000023 00000000 007cfc64 001d3756 001d3756 001d5024
ffa02790 02003006 00f5e077 ffa02308 00f5e076 ffa02306 00000000 00000000
00000000 00000000 00000000 00000000 00000001 ffffff80 00000006 00000082
00000000 00000000 00000000 00000000 00000000 00000000 00c056b7 00000008
00000008 00000000 00000000 00f79ba0 007d2c94 007cfc74 ffa02778 0145da20
Call Trace:
[<001d5024>] _process_backlog+0x78/0x148
[<001d5162>] _net_rx_action+0x6e/0x108
[<00111114>] ___do_softirq+0x60/0xac
[<001beed4>] WaitForStatus1End+0x0/0x8
[<001beed4>] WaitForStatus1End+0x0/0x8
[<001bee9e>] _ReadCodecRegister+0x12/0x16
[<001beed4>] WaitForStatus1End+0x0/0x8
[<001beec4>] WaitForStatus1Start+0x0/0x10
[<001bdf1c>] _AudioIoctl+0x0/0x304
[<001bdf1c>] _AudioIoctl+0x0/0x304
[<001be46a>] _ControlAD1985CodecAPI+0x16/0x20
[<001be1e2>] _AudioIoctl+0x2c6/0x304
[<00141e28>] _do_ioctl+0x48/0x4c
[<00141e78>] _vfs_ioctl+0x4c/0x1e4
[<00181d50>] _sys_msgrcv+0x24/0x48
[<00142038>] _sys_ioctl+0x28/0x54
[<00142010>] _sys_ioctl+0x0/0x54
Modules linked in:
Kernel panic - not syncing: Kernel exception
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Log 2:
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Data access CPLB protection violation
- Attempted read or write to Supervisor resource,
or illegal data memory access.
Kernel OOPS in progress
Defered Exception context
No Valid process in current context
return address: [0x001d55c6]; contents of:
0x001d55a0: e102 2df4 5010 3210 3030 9155 e12a 0094
0x001d55b0: 5ad4 6f45 2003 adad 6f45 3045 64c0 0830
0x001d55c0: 1827 0000 0000 [9528] 0807 17f6 a069 0c01
0x001d55d0: 1806 0000 0000 a160 0801 17ee 0c41 1816
SEQUENCER STATUS: Not tainted
SEQSTAT: 00000023 IPEND: c030 SYSCFG: 0006
HWERRCAUSE: 0x0
EXCAUSE : 0x23
physical IVG14 asserted : <0xffa00b18> { _evt14_softirq + 0x0 }
physical IVG15 asserted : <0xffa01238> { _evt_system_call + 0x0 }
logical irq 6 mapped : <0xffa00280> { _timer_interrupt + 0x0 }
logical irq 12 mapped : <0xffa02784> { _AudioRxPPI_ERR + 0x0 }
logical irq 18 mapped : <0x001a5acc> { _bfin_serial_dma_rx_int + 0x0 }
logical irq 19 mapped : <0x001a567c> { _bfin_serial_dma_tx_int + 0x0 }
logical irq 24 mapped : <0x001b4990> { _bf537mac_interrupt + 0x0 }
logical irq 40 mapped : <0x001beab8> { _AudioDataISR + 0x0 }
RETE: <0x00000000> /* Maybe null pointer? */
RETN: <0x00273dec> /* unknown address */
RETX: <0x001d55c6> { _netif_receive_skb + 0x136 }
RETS: <0x001d6e94> { _process_backlog + 0x78 }
PC : <0x001d55c6> { _netif_receive_skb + 0x136 }
DCPLB_FAULT_ADDR: <0xffa02778> { _AudioTXVideoRx + 0x0 }
ICPLB_FAULT_ADDR: <0x001d55c6> { _netif_receive_skb + 0x136 }
PROCESSOR STATE:
R0 : ffa02790 R1 : 00000800 R2 : 002b2df4 R3 : 0000000f
R4 : 00000040 R5 : 0034f800 R6 : 002b2df4 R7 : 00000008
P0 : 002b2a50 P1 : 00000000 P2 : 00000094 P3 : 016f7354
P4 : 016f72c0 P5 : ffa02778 FP : 00273dfc SP : 00273d10
LB0: ffa02308 LT0: ffa02306 LC0: 00000000
LB1: 00d5e077 LT1: 00d5e076 LC1: 00000000
B0 : 00000076 L0 : 00000000 M0 : 00000008 I0 : 00d79ba0
B1 : 00000001 L1 : 00000000 M1 : 0060554f I1 : 00000000
B2 : 00000040 L2 : 00000000 M2 : 00000000 I2 : 0008ca20
B3 : 00000002 L3 : 00000000 M3 : 00000000 I3 : 00000000
A0.w: 00000000 A0.x: 00000000 A1.w: 00000000 A1.x: 00000000
USP : 00274000 ASTAT: 02003006
Hardware Trace:
0 Target : <0x00103828> { _trap_c + 0x0 }
Source : <0xffa0079c> { _exception_to_level5 + 0xb4 }
1 Target : <0xffa006e8> { _exception_to_level5 + 0x0 }
Source : <0xffa00644> { _ex_trap_c + 0x5c }
2 Target : <0xffa0058c> { _handle_bad_cplb + 0x0 }
Source : <0xffa0161c> { __cplb_hdr + 0x5c }
3 Target : <0xffa0161a> { __cplb_hdr + 0x5a }
Source : <0x00103752> { _panic_cplb_error + 0x16 }
4 Target : <0x0010373c> { _panic_cplb_error + 0x0 }
Source : <0xffa01616> { __cplb_hdr + 0x56 }
5 Target : <0xffa01612> { __cplb_hdr + 0x52 }
Source : <0xffa0160a> { __cplb_hdr + 0x4a }
6 Target : <0xffa01604> { __cplb_hdr + 0x44 }
Source : <0xffa015fe> { __cplb_hdr + 0x3e }
7 Target : <0xffa015fc> { __cplb_hdr + 0x3c }
Source : <0xffa015f6> { __cplb_hdr + 0x36 }
8 Target : <0xffa015f4> { __cplb_hdr + 0x34 }
Source : <0xffa015ec> { __cplb_hdr + 0x2c }
9 Target : <0xffa015e8> { __cplb_hdr + 0x28 }
Source : <0xffa01780> { _cplb_mgr + 0x160 }
10 Target : <0xffa0176c> { _cplb_mgr + 0x14c }
Source : <0xffa01796> { _cplb_mgr + 0x176 }
11 Target : <0xffa01782> { _cplb_mgr + 0x162 }
Source : <0xffa01624> { _cplb_mgr + 0x4 }
12 Target : <0xffa01620> { _cplb_mgr + 0x0 }
Source : <0xffa015e4> { __cplb_hdr + 0x24 }
13 Target : <0xffa015e0> { __cplb_hdr + 0x20 }
Source : <0xffa015ce> { __cplb_hdr + 0xe }
14 Target : <0xffa015c0> { __cplb_hdr + 0x0 }
Source : <0xffa004d8> { _ex_dcplb_miss + 0x64 }
15 Target : <0xffa00474> { _ex_dcplb_miss + 0x0 }
Source : <0xffa00470> { _ex_workaround_261 + 0x20 }
Stack from 00273cf0:
016f72c0 ffa007a0 00276630 00276630 0027662c 00dca080 00000011 00000001
001d55c6 0000c030 00000023 00000000 00273dec 001d55c6 001d55c6 001d6e94
ffa02790 02003006 00d5e077 ffa02308 00d5e076 ffa02306 00000000 00000000
00000000 00000000 00000000 00000000 00000002 00000040 00000001 00000076
00000000 00000000 00000000 00000000 00000000 00000000 0060554f 00000008
00000000 0008ca20 00000000 00d79ba0 00274000 00273dfc ffa02778 016f72c0
Call Trace:
[<00108bd0>] ___wake_up+0x20/0x40
[<001d6e94>] _process_backlog+0x78/0x148
[<001d6fd2>] _net_rx_action+0x6e/0x108
[<00111114>] ___do_softirq+0x60/0xac
[<00282a3c>] _start_kernel+0x214/0x284
[<00282378>] _unknown_bootoption+0x0/0x21c
[<002821cc>] _real_start+0x50/0x94
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
thanks in advance,
Supriya
QuoteReplyEditDelete
2009-02-06 08:41:03 Re: 2008R1-RC8 crash in BF537
Michael Hennerich (GERMANY)
Message: 68990
DCPLB_FAULT_ADDR: <0xffa02778> { _AudioTXVideoRx + 0x0 }
Something is reading from L1 Instruction Memory - This is causing the crash.
The core can't read from L1 Instruction memory.
ICPLB_FAULT_ADDR: <0x001d3756> { _netif_receive_skb + 0x136 }
It looks like netif_receive_skb() is issueing this access.
Does your application use L1 Data Memory???
Do you have the BFIN_MAC_USE_L1 option enabled?
-Michael
QuoteReplyEditDelete
2009-02-06 08:44:29 Re: 2008R1-RC8 crash in BF537
Michael Hennerich (GERMANY)
Message: 68991
Did you compile the kernel with the right silicon revision?
You can't use a kernel compiled for Rev 0.3 on Rev 0.2 Silicon ...
-Michael
QuoteReplyEditDelete
2009-02-09 05:20:50 Re: 2008R1-RC8 crash in BF537
supriya d (INDIA)
Message: 69078
Hi Michael,
Firstly, Thanks for your reply.
We are using the rev 0.2 board and the distribution is compiled for 0.2 rev.
We were able to find some way out for the problem over the weekend!. With more debugging and analysis we saw that when the UART messages(printk and printfs) are reduced the application crashes after a long time. On disabling UART completely the application went on for 2 + days without crash. Currently we have put the UART in PIO mode(earlier we were using DMA mode), with this we donot see any crash.
earlier, the DMAs that were used in the application,
1. Audio over SPORT
2. UART
3. MAC
Currently we are having only Audio and MAC dma's.
Though PIO mode works we just wanted to check why a UART DMA mode is giving issues in application.
Also please suggest if we have to configure DMAs without using the DMA core driver(part of distribution) are there any points we need to specially take care of?
Thanks,
Supriya
QuoteReplyEditDelete
2009-02-09 07:59:21 Re: 2008R1-RC8 crash in BF537
Robin Getz (UNITED STATES)
Message: 69122
Supriya:
The only way to tell what is going on, is to recompile the kernel with DEBUG_BFIN_NO_KERN_HWTRACE set on (in kernel hacking - "Trace user apps (turn off hwtrace in kernel)".
Do that - recreate the failure - post the result - we will fix it.
-Robin