[#6057] Kernel crashed with slob configuration(2010)

[#6057] Kernel crashed with slob configuration

Submitted By: Vivi Li

Open Date

2010-06-03 03:30:05

Priority:

Medium Assignee:

Bob Liu

Status:

Open Fixed In Release:

N/A

Found In Release:

2010R1-RC5 Release:

Category:

N/A Board:

EZBRD

Processor:

BF526 Silicon Revision:

Is this bug repeatable?:

Yes Resolution:

Under Debugging

Uboot version or rev.:

Toolchain version or rev.:

gcc4.3-2010_Apr_07

App binary format:

N/A

Summary: Kernel crashed with slob configuration

Details:

Kernel crashed with slob configuration on BFIN526_EZBRD. Kernel config is attached.

Last passed version:

kernel: Linux release 2.6.34-ADI-2010R1-pre-svn8831, build #80 Mon May 24 13:10:41 CST 2010

toolchain: bfin-uclinux-gcc release gcc version 4.3.4 (ADI-trunk/svn-3951)

user-dist: release svn-9641, build #1492 Mon May 24 13:09:55 CST 2010

First failed version:

kernel: Linux release 2.6.34-ADI-2010R1-pre-svn8840, build #84 Tue May 25 21:20:21 CST 2010

toolchain: bfin-uclinux-gcc release gcc version 4.3.4 (ADI-trunk/svn-3951)

user-dist: release svn-9644, build #1570 Tue May 25 21:19:41 CST 2010

Bellow is the detailed log:

Linux version 2.6.34-ADI-2010R1-pre-svn8863 (test@uclinux81-bf526) (gcc version 4.3.4 (ADI-trunk/svn-3951) ) #104 Wed Jun 2 04:48:00 CST 2010

bootconsole [early_shadow0] enabled

bootconsole [early_BFuart1] enabled

early printk enabled on early_BFuart1

Board Memory: 64MB

Kernel Managed Memory: 64MB

Memory map:

fixedcode = 0x00000400-0x00000490

text = 0x00001000-0x00132458

rodata = 0x00132458-0x00198120

bss = 0x00199000-0x001a9d54

data = 0x001a9d54-0x001be000

stack = 0x001bc000-0x001be000

init = 0x001be000-0x0030c000

available = 0x0030c000-0x03f00000

DMA Zone = 0x03f00000-0x04000000

Hardware Trace Active and Enabled

Boot Mode: 1

Compiled for ADSP-BF526 Rev 0.0

Blackfin Linux support by http://blackfin.uclinux.org/

Processor Speed: 400 MHz core clock and 80 MHz System Clock

NOMPU: setting up cplb tables

Instruction Cache Enabled for CPU0

External memory: cacheable in instruction cache

Data Cache Enabled for CPU0

External memory: cacheable (write-through) in data cache

Built 1 zonelists in Zone order, mobility grouping off. Total pages: 16002

Kernel command line: root=/dev/mtdblock0 rw clkin_hz=25000000 earlyprintk=serial,uart1,57600 console=ttyBF1,57600 ip=10.100.4.50:10.100.4.174:10.100.4.174:255.255.255.0:bf526-ezbrd:eth0:off

PID hash table entries: 256 (order: -2, 1024 bytes)

Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)

Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)

Memory available: 60824k/65536k RAM, (1336k init code, 1221k kernel code, 559k data, 1024k dma, 572k reserved)

Hierarchical RCU implementation.

NR_IRQS:159

Configuring Blackfin Priority Driven Interrupts

console [ttyBF1] enabled, bootconsole disabled

Calibrating delay loop... 792.57 BogoMIPS (lpj=1585152)

Security Framework initialized

Mount-cache hash table entries: 512

Blackfin Scratchpad data SRAM: 4 KB

Blackfin L1 Data A SRAM: 16 KB (16 KB free)

Blackfin L1 Data B SRAM: 16 KB (16 KB free)

Blackfin L1 Instruction SRAM: 48 KB (42 KB free)

NET: Registered protocol family 16

Blackfin DMA Controller

ezbrd_init(): registering device resources

bio: create slab <bio-0> at 0

NULL pointer access

Kernel OOPS in progress

Deferred Exception context

CURRENT PROCESS:

COMM=swapper PID=1 CPU=0

invalid mm

return address: [0x000404de]; contents of:

0x000404b0: b371 323a 0a05 1451 e14c 001b e10c 1914

0x000404c0: 0034 61f8 0040 9165 640d c682 8ffd 67fd

0x000404d0: 6f45 6006 2003 3229 6f45 3255 6cc2 [a1a9]

0x000404e0: 0854 1826 0000 0000 0000 a928 09b8 1ff4

ADSP-BF526-0.0 400(MHz CCLK) 80(MHz SCLK) (mpu off)

Linux version 2.6.34-ADI-2010R1-pre-svn8863 (test@uclinux81-bf526) (gcc version 4.3.4 (ADI-trunk/svn-3951) ) #104 Wed Jun 2 04:48:00 CST 2010

SEQUENCER STATUS: Not tainted

SEQSTAT: 00000027 IPEND: 8008 IMASK: 003f SYSCFG: 0006

Peripheral interrupts masked off

Kernel interrupts masked off

EXCAUSE : 0x27

physical IVG3 asserted : <0xffa00720> { _trap + 0x0 }

physical IVG15 asserted : <0xffa00f5c> { _evt_system_call + 0x0 }

logical irq 6 mapped : <0xffa00358> { _bfin_coretmr_interrupt + 0x0 }

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x02009ed8> /* kernel dynamic memory (maybe user-space) */

RETX: <0x00000480> /* Maybe fixed code section */

RETS: <0x000404fc> { _slob_alloc + 0x58 }

PC : <0x000404de> { _slob_alloc + 0x3a }

DCPLB_FAULT_ADDR: <0x00000000> /* Maybe null pointer? */

ICPLB_FAULT_ADDR: <0x000404de> { _slob_alloc + 0x3a }

PROCESSOR STATE:

R0 : 00000000 R1 : 00000000 R2 : 0000000c R3 : 0000000c

R4 : 0000ffff R5 : 00000048 R6 : 00000000 R7 : 00000024

P0 : 02036000 P1 : 02035000 P2 : 00000000 P3 : 0034e758

P4 : 001b1914 P5 : ffffffe8 FP : 00000020 SP : 02009dfc

LB0: ffa0159c LT0: ffa01596 LC0: 00000000

LB1: 0008dd38 LT1: 0008dd2c LC1: 00000000

B0 : 00000000 L0 : 00000000 M0 : 0000002c I0 : 00000fff

B1 : 00000000 L1 : 00000000 M1 : 00000000 I1 : 00000001

B2 : 00000000 L2 : 00000000 M2 : 00000000 I2 : ffffffe0

B3 : 00000000 L3 : 00000000 M3 : 00000000 I3 : 00000000

A0.w: 00000005 A0.x: 00000000 A1.w: 00000005 A1.x: 00000000

USP : 00000000 ASTAT: 00002022

Hardware Trace:

...

Follow-ups

--- Mike Frysinger 2010-06-03 13:04:07

i'm pretty sure we'll see same behavior with SLOB on all boards

--- Vivi Li 2010-06-03 22:46:40

This bug first happens on May, 25th. And I haven't seen this bug on other plat

form yet.

--- Vivi Li 2010-06-23 06:01:57

I also see it hang at "bio: create slab <bio-0> at 0" on

bf527-ezkit, though it doesn't happen all the time.

--- Bob Liu 2010-07-08 02:55:05

I can also see it in the Last passed version and even 2.6.33 kernel version.

It seems that the page's memory was modified exceptionally.

--- Bob Liu 2010-07-08 04:01:24

537-stamp have this problem if SCSI support is added in kernel config.

And I also think we can see it on all boards.

--- Vivi Li 2011-04-29 03:15:57

Now I see this crash on slob on both bf527 and bf526.

--- Bob Liu 2011-06-01 04:28:19

This bug should be fixed by commit:

http://blackfin.uclinux.org/git/?p=linux-kernel;a=commitdiff;h=e1066aa334b3155befa2779bad1028334dbe13fa

--- Vivi Li 2011-06-08 01:39:14

It seems OK on bf526-ezbrd now. But I can still see slob kernel crash from time

to time, for example on regression machine 10.99.29.57 with bf537-stamp.

Bellow is the log:

Linux version 2.6.39-ADI-2011R1-pre-svn9919

(test@uclinux57-usb1761-vs6624-ad1371) (gcc version 4.3.5 (ADI-trunk/svn-5013) )

#12 Thu Jun 2 04:53:32 CST 2011

bootconsole [early_shadow0] enabled

bootconsole [early_BFuart0] enabled

early printk enabled on early_BFuart0

Limiting kernel memory to 56MB due to anomaly 05000263

Board Memory: 64MB

Kernel Managed Memory: 64MB

Memory map:

fixedcode = 0x00000400-0x00000490

text = 0x00001000-0x00117830

rodata = 0x00117830-0x0016e8b8

bss = 0x0016f000-0x0017f8d8

data = 0x0017f8d8-0x00190000

stack = 0x0018e000-0x00190000

init = 0x00190000-0x00312000

available = 0x00312000-0x03800000

DMA Zone = 0x03f00000-0x04000000

Hardware Trace active and enabled

Boot Mode: 0

Compiled for ADSP-BF537 Rev 0.2

Blackfin Linux support by http://blackfin.uclinux.org/

Processor Speed: 500 MHz core clock and 125 MHz System Clock

NOMPU: setting up cplb tables

Instruction Cache Enabled for CPU0

External memory: cacheable in instruction cache

Data Cache Enabled for CPU0

External memory: cacheable (write-back) in data cache

Built 1 zonelists in Zone order, mobility grouping off. Total pages: 14224

Kernel command line: root=/dev/mtdblock0 rw clkin_hz=25000000

earlyprintk=serial,uart0,57600 console=ttyBF0,57600

ip=10.100.4.50:10.100.4.174:10.100.4.174:255.255.255.0:bf537-stamp:eth0:off

PID hash table entries: 256 (order: -2, 1024 bytes)

Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)

Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)

Memory available: 53592k/65536k RAM, (1544k init code, 1114k kernel code, 482k

data, 1024k dma, 7780k reserved)

NR_IRQS:146

Configuring Blackfin Priority Driven Interrupts

bfin-rtc: invalid date; resetting

console [ttyBF0] enabled, bootconsole disabled

Calibrating delay loop... 995.32 BogoMIPS (lpj=1990656)

pid_max: default: 32768 minimum: 301

Mount-cache hash table entries: 512

Blackfin Scratchpad data SRAM: 4 KB

Blackfin L1 Data A SRAM: 16 KB (16 KB free)

Blackfin L1 Data B SRAM: 16 KB (16 KB free)

Blackfin L1 Instruction SRAM: 48 KB (35 KB free)

NET: Registered protocol family 16

Blackfin DMA Controller

stamp_init(): registering device resources

bio: create slab <bio-0> at 0

bfin-spi bfin-spi.0: Blackfin on-chip SPI Controller Driver, Version 1.0,

regs_base@ffc00500, dma channel@7

Switching to clocksource bfin_cs_cycles

NET: Registered protocol family 2

IP route cache hash table entries: 1024 (order: 0, 4096 bytes)

TCP established hash table entries: 2048 (order: 2, 16384 bytes)

TCP bind hash table entries: 2048 (order: 1, 8192 bytes)

TCP: Hash tables configured (established 2048 bind 2048)

TCP reno registered

UDP hash table entries: 256 (order: 0, 4096 bytes)

UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)

NET: Registered protocol family 1

debug-mmrs: setting up Blackfin MMR debugfs

msgmni has been set to 104

io scheduler noop registered (default)

bfin-uart: Blackfin serial driver

bfin-uart.0: ttyBF0 at MMIO 0xffc00400 (irq = 18) is a BFIN-UART

brd: module loaded

bfin_mii_bus: probed

bfin_mac: attached PHY driver [SMSC LAN83C185] (mii_bus:phy_addr=0:01, irq=-1,

mdc_clk=2500000Hz(mdc_div=24)@sclk=125MHz)

bfin_mac bfin_mac.0: eth0: Features changed: 0x00004800 -> 0x00004000

bfin_mac bfin_mac.0: eth0: Blackfin on-chip Ethernet MAC driver, Version 1.1

Jump to NULL address

Kernel OOPS in progress

Deferred Exception context

CURRENT PROCESS:

COMM=rcu_kthread PID=6 CPU=0

invalid mm

return address: [0x (null)]; contents of:

ADSP-BF537-0.2 500(MHz CCLK) 125(MHz SCLK) (mpu off)

Linux version 2.6.39-ADI-2011R1-pre-svn9919

(test@uclinux57-usb1761-vs6624-ad1371) (gcc version 4.3.5 (ADI-trunk/svn-5013) )

#12 Thu Jun 2 04:53:32 CST 2011

SEQUENCER STATUS: Not tainted

SEQSTAT: 0000002d IPEND: 8008 IMASK: ffff SYSCFG: 0006

EXCAUSE : 0x2d

physical IVG3 asserted : <0xffa007b8> { _trap + 0x0 }

physical IVG15 asserted : <0xffa0109c> { _evt_system_call + 0x0 }

logical irq 6 mapped : <0xffa003cc> { _bfin_coretmr_interrupt + 0x0

}

logical irq 106 mapped : <0x000bdda8> { _bfin_mac_interrupt + 0x0 }

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x02017f68> /* kernel dynamic memory (maybe user-space) */

RETX: <0x00000480> /* Maybe fixed code section */

RETS: <0x00035c60> { _rcu_process_callbacks + 0x58 }

PC : <0x00000000> /* Maybe null pointer? */

DCPLB_FAULT_ADDR: <0x023090e0> /* kernel dynamic memory (maybe

user-space) */

ICPLB_FAULT_ADDR: <0x00000000> /* Maybe null pointer? */

PROCESSOR STATE:

R0 : 023090dc R1 : 00000200 R2 : 00000200 R3 : 0000016c

R4 : 00180e1c R5 : 000241a4 R6 : 02016000 R7 : 00000000

P0 : 022ef380 P1 : 022ef000 P2 : 00000000 P3 : 02016000

P4 : 02017f8c P5 : 00000000 FP : 02017f98 SP : 02017e8c

LB0: ffa01770 LT0: ffa0176e LC0: 00000000

LB1: 0009524c LT1: 00095240 LC1: 00000000

B0 : 0000001b L0 : 00000000 M0 : ffffffff I0 : 00000146

B1 : 00000000 L1 : 00000000 M1 : 00000000 I1 : 00000001

B2 : 00000000 L2 : 00000000 M2 : 00000000 I2 : fffffffc

B3 : 00000000 L3 : 00000000 M3 : 00000000 I3 : 0000000f

A0.w: 00000000 A0.x: 00000000 A1.w: 00000000 A1.x: 00000000

USP : 00000000 ASTAT: 00002000

Hardware Trace:

...

--- Vivi Li 2011-08-22 05:56:52

I can still see this crash on bf537-stamp in buildroot.

To reproduce this bug we need to reboot kernel several times.

--- Vivi Li 2011-08-30 02:10:19

Attach config in buildroot.

--- Vivi Li 2011-10-19 04:49:53

Please try again use attached kernel config. Userspace config is not related and

you can use default user config.

I can always see crash with this config on bf537-stamp.

--- Tomasz Motylewski 2011-11-28 11:39:21

I also get this bug triggered on 2.6.39.1 - 2.6.39.3. It is very sensitive to

minute changes in kernel config and also in u-boot environment. It is also

sensitive to RAM contents before kernel boots (or whatever state preserved over

board reset).

when I symlink kernel (vmImage) as vmlinux-SVN-N3.img (nfsboot) it may crash,

while when I load the same kernel named vmlinux-SVN.img over nfsboot it works.

Then I change unrelated option and it reproducibly crashes (after CPU reset

pin). But when I then do hard power-down-up it suddenly runs OK. all with

u-boot-1.1.6.

BF537-stamp-like board.

BTW. With which bootloader do you use it? 1.1.6 has a bug - it leaves UART_IER

enabled, and when send serial char while kernel boots, it will crash after

ttyBF0 init. This I have fixed (UART_IER = 0) in my

native_machine_early_platform_add_devices() or stamp_init(). Whether if fixes

SLOB bug - not sure yet.

--- Sonic Zhang 2011-11-28 22:12:28

Please ask your uboot question on Analog EngineerZone uboot forum.

Please always use the same release version of uClinux-dist and u-boot.

For example, 2010R1 uClinux-dist and 2010R1 u-boot or trunk uClinux-dist and

trunk u-boot.

Files

Changes

Commits

Dependencies

Duplicates

Associations