[#5197] running traps test case in fdpic format would randomly make kernel reboot

Submitted By: Mingquan Pan

Open Date

2009-06-03 05:29:04 Close Date

2009-08-18 01:15:18

Priority:

Medium High Assignee:

Mingquan Pan

Status:

Closed Fixed In Release:

N/A

Found In Release:

N/A Release:

Category:

N/A Board:

N/A

Processor:

BF537 Silicon Revision:

Is this bug repeatable?:

Yes Resolution:

Assigned (Not Start)

Uboot version or rev.:

Toolchain version or rev.:

09r1-rc7

App binary format:

N/A

Summary: running traps test case in fdpic format would randomly make kernel reboot

Details:

running traps test case in fdpic format would randomly make kernel reboot on branche head on bf537 stamp board.

Sometimes it would stop at case 14, sometimes stops at 11, 9.

It looks that it is not seen on bf533 stamp board.

log is like:

...

TCP cubic registered

NET: Registered protocol family 17

rtc-bfin rtc-bfin: setting system clock to 1970-01-17 02:20:56 UTC (1390856)

Freeing unused kernel memory: 6572k freed

dma_alloc_init: dma_page @ 0x007e2000 - 256 pages at 0x03f00000

_____________________________________

a8888b. / Welcome to the uClinux distribution \

d888888b. / _ _ \

8P"YP"Y88 / | | |_| __ __ (TM) |

8|o||o|88 _____/ | | _ ____ _ _ \ \/ / |

8' .88 \ | | | | _ \| | | | \ / |

8`._.' Y8. \ | |__ | | | | | |_| | / \ |

d/ `8b. \ \____||_|_| |_|\____|/_/\_\ |

dP . Y8b. \ For embedded processors including |

d8:' " `::88b \ the Analog Devices Blackfin /

d8" 'Y88b \___________________________________/

:8P ' :888

8a. : _a88P For further information, check out:

._/"Yaa_: .| 88P| - http://blackfin.uclinux.org/

\ YP" `| 8P `. - http://docs.blackfin.uclinux.org/

/ \.___.d| .' - http://www.uclinux.org/

`--..__)8888P`._.' jgs/a:f - http://www.analog.com/blackfin

Have a lot of fun...

BusyBox v1.13.4 (2009-06-03 17:02:20 CST) built-in shell (msh)

Enter 'help' for a list of built-in commands.

root:/> version

kernel: Linux release 2.6.28.10-ADI-2009R1-svn6543, build #4 Wed Jun 3 17:14:47 CST 2009

toolchain: bfin-linux-uclibc-gcc release gcc version 4.1.2 (ADI svn)

user-dist: release svn-8191, build #2 Wed Jun 3 17:14:02 CST 2009

root:/>

root:/> cd bin/

root:/bin>

root:/bin> ./traps_test -1

...

... address : <0x0062537c> [ /bin/traps_test + 0x137c ]

address : <0x00633b24> [ traps_test + 0xb24 ]

address : <0x00633b2c> [ traps_test + 0xb2c ]

address : <0x00633b34> [ traps_test + 0xb34 ]

address : <0x0063c178> [ /lib/ld-uClibc.so.0 + 0x7178 ]

address : <0x0067ffa2> [ traps_test + 0x1ffa2 ]

address : <0x0067ffaa> [ traps_test + 0x1ffaa ]

PASS (test failed, as expected by signal 4: Illegal instruction)

Running test 9 for exception 0x0a: EXCPT 0x0A

... Unimplmented exception occured

- Maybe you forgot to install a custom exception handler?

Deferred Exception context

CURRENT PROCESS:

COMM=traps_test PID=173

CPU = 0

TEXT = 0x00624000-0x00626ca0 DATA = 0x0063eca0-0x0063f264

BSS = 0x0063f264-0x00660000 USER-STACK = 0x0067fea0

return address: [0x006256ee]; contents of:

0x006256c0: 0000 0010 0000 0000 e800 0000 00a8 e801

0x006256d0: 0000 0010 0000 0000 e800 0000 00a9 e801

0x006256e0: 0000 0010 0000 0000 e800 0000 00aa [e801]

0x006256f0: 0000 0010 0000 0000 e800 0000 00ab e801

ADSP-BF537-0.2 500(MHz CCLK) 100(MHz SCLK) (mpu off)

Linux version 2.6.28.10-ADI-2009R1-svn6543

Built with gcc version 4.1.2 (ADI svn)

SEQUENCER STATUS: Not tainted

SEQSTAT: 0006000a IPEND: 0030 SYSCFG: 0006

EXCAUSE : 0xa

interrupts disabled

physical IVG5 asserted : <0xffa00c14> { _evt_ivhw + 0x0 }

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x0063c000> /*

U-Boot 1.1.6-svn1152 (ADI-2008R2-pre) (Feb 27 2008 - 18:20:24)

CPU: ADSP bf537-0.2 (Detected Rev: 0.2)

Board: ADI BF537 stamp board

Support: http://blackfin.uclinux.org/

Clock: VCO: 500 MHz, Core: 500 MHz, System: 100 MHz

RAM: 64 MB

Flash: 4 MB

In: serial

Out: serial

Err: serial

Net: Blackfin EMAC

MAC: 00:E0:22:FE:B5:B5

I2C: ready

Follow-ups

--- Mike Frysinger 2009-06-03 05:34:04

make sure it isnt the watchdog causing the reboot. the next kernel boot will

tell you why the reset happened. or try using `dmesg -n1` before running the

traps_test.

--- Mingquan Pan 2009-06-03 06:11:10

Yeah, you are right. After adding "dmesg -n1" before running the case,

it fails at case 18 as using flat kernel every time.

Is there a way to make watchdog not so sensitive? Since in flat format, running

the case wouldn't trigger the watchdog and reboot like this. Otherwise I have to

add this command in script, and would miss the dump info in log.

--- Mike Frysinger 2009-06-03 06:18:05

you could try launching watchdogd with -w 60 -k 5

--- Robin Getz 2009-06-03 09:50:37

Grace:

There should be no need to set dmesg, or change the watchdog.

root:/> time ./traps_test -d0 -1

[snip]

55/55 tests passed

real 0m 1.02s

user 0m 0.01s

sys 0m 0.04s

All the traps test take less than a second to run. Even if we trace them,

root:/> time ./traps_test -d0 -t -1

[snip]

55/55 tests passed

real 0m 5.05s

user 0m 1.09s

sys 0m 3.53s

it takes less than 5 seconds.

If you get a test failure - then you can run it with "-v" (verbose)

to see the kernel messages.

-Robin

--- Mingquan Pan 2009-06-03 22:06:26

Are "-t" "-v" the parameters for ./traps_test?

Why it doesn't show by help?

root:/bin> ./traps_test -h

Usage: ./traps_test [-c count] [-d milliseconds] [-q] [-l] [starting test

number] [ending test number]

-c count Repeat the test(s) count times before stopping

-d seconds The number of milliseconds to delay between flushing stdout,

and

running the test (default is 1)

-l List tests, then quit

-q Quiet (don't print out test info)

If no test number is specified, the number of tests available will be shown.

If a single test number is specified (0 <= n < # of tests), that test

will be run.

If two tests numbers are specified (0 <= start < end < # tests), those

tests will be run

If you specify -1, then all tests will be run in order.

root:/bin>

root:/bin> time ./traps_test -d0 -t -1

./traps_test: invalid option -- t

Usage: ./traps_test [-c count] [-d milliseconds] [-q] [-l] [starting test

number] [ending test number]

-c count Repeat the test(s) count times before stopping

-d seconds The number of milliseconds to delay between flushing stdout,

and

running the test (default is 1)

-l List tests, then quit

-q Quiet (don't print out test info)

If no test number is specified, the number of tests available will be shown.

If a single test number is specified (0 <= n < # of tests), that test

will be run.

If two tests numbers are specified (0 <= start < end < # tests), those

tests will be run

If you specify -1, then all tests will be run in order.

ERROR: unknown option

Command exited with non-zero status 1

real 0m 0.00s

user 0m 0.00s

sys 0m 0.00s

--- Robin Getz 2009-06-03 23:37:15

Grace:

What toolchain are you using? I'm getting some problems with 4.3 on 0.3

silicon, compiled as 0.2 -- I'm still digging into things to figure it out.

--- Robin Getz 2009-06-04 00:34:32

Grace:

For now - use the traps test from trunk. I have not checked into the branch

yet.

-Robin

root:/> traps_test -h

Usage: traps_test [-c count] [-d milliseconds] [-e] [-q] [-l] [-p] [-t] [-v]

[starting test number] [ending test number]

-c count Repeat the test(s) count times before stopping

-d seconds The number of milliseconds to delay between flushing stdout,

and

running the test (default is 1)

-e exclude the checking of excause is as expected

-l List tests, then quit

-q Quiet (don't print out test info)

-p Run the test in the parent process, otherwise fork a child

process.

Only valid for a single test.

-t Trace (single step) through the failing tests, to simulate gdb

-v Verbose. equivilent to 'dmesg -n 7', so that all kernel

messages print out

otherwise do 'dmesg -n 3', so things are a little quieter

If no test number is specified, the number of tests available will be shown.

If a single test number is specified (0 <= n < # of tests), that test

will be run.

If two tests numbers are specified (0 <= start < end < # tests), those

tests will be run

If you specify -1, then all tests will be run in order.

--- Robin Getz 2009-06-14 23:22:47

Grace:

Can you make sure what is on trunk fixes things for you - then I will commit to

the branch.

-Robin

--- Mingquan Pan 2009-06-16 06:36:28

Yeah, it is really useful by saving the lots of dump info.

the bf537 board won't reboot randomly now when running: ./traps_test -d0 -1 in

fdpic format.

But it looks running ./traps_test with -t or not would get results

different(built in fdpic).

./traps_test -d0 -1 running ok.

./traps_test -d0 -t -1 would stuck at case 3.

root:/bin> ./traps_test -d0 -t -1

Running test 0 for exception 0x01: EXCPT 0x01

... PASS (test failed, as expected by signal 5: Trace/breakpoint trap)

Running test 1 for exception 0x02: EXCPT 0x02

... PASS (test failed, as expected by signal 4: Illegal instruction)

Running test 2 for exception 0x03: EXCPT 0x03

... PASS (test failed, as expected by signal 11: Segmentation fault)

Running test 3 for exception 0x04: EXCPT 0x04

...

stucks here.

Run case 3 alone with -t wouldn't have this problem.

root:/> cat /proc/cp

/proc/cplbinfo/ /proc/cpuinfo

root:/> cat /proc/cpuinfo

processor : 0

vendor_id : Analog Devices

cpu family : 0x27c8

model name : ADSP-BF537 500(MHz CCLK) 125(MHz SCLK) (mpu off)

stepping : 2

cpu MHz : 500.000/125.000

bogomips : 997.37

Calibration : 498688000 loops

cache size : 16 KB(L1 icache) 32 KB(L1 dcache-wb) 0 KB(L2 cache)

dbank-A/B : cache/cache

icache setup : 4 Sub-banks/4 Ways, 32 Lines/Way

dcache setup : 2 Super-banks/4 Sub-banks/2 Ways, 64 Lines/Way

board name : ADI BF537-STAMP

board memory : 65536 kB (0x(null) -> 0x04000000)

kernel memory : 57336 kB (0x00001000 -> 0x037ff000)

root:/> version

kernel: Linux release 2.6.30-ADI-2010R1-pre-svn6699, build #3 Tue Jun 16

18:23:20 CST 2009

toolchain: bfin-linux-uclibc-gcc release gcc version 4.1.2 (ADI svn)

user-dist: release svn-8293, build #1 Tue Jun 16 18:22:18 CST 2009

--- Robin Getz 2009-06-16 14:38:45

On trunk it works...

root:/> cat /proc/cpuinfo

processor : 0

vendor_id : Analog Devices

cpu family : 0x27c8

model name : ADSP-BF537 500(MHz CCLK) 125(MHz SCLK) (mpu off)

stepping : 2

root:/> version

kernel: Linux release 2.6.28.10-ADI-2010R1-pre-svn6607, build #41 Tue Jun 16

10:57:35 EDT 2009

toolchain: bfin-linux-uclibc-gcc release gcc version 4.3.3

(ADI-trunk/svn-3407)

user-dist: release svn-8255, build #274 Tue Jun 16 10:57:01 EDT 2009

./traps_test -d0 -t -1

[snip]

55/55 tests passed

/traps_test -d0 -t -c 100 -1

[snip]

55/55 tests passed

I'm just loading up the release kernel, and seeing how that works.

-Robin

--- Robin Getz 2009-06-16 15:57:38

root:/> cat /proc/cpuinfo

processor : 0

vendor_id : Analog Devices

cpu family : 0x27c8

model name : ADSP-BF537 500(MHz CCLK) 125(MHz SCLK) (mpu off)

stepping : 2

root:/> version

kernel: Linux release 2.6.28.10-ADI-2009R1-svn6753, build #2 Tue Jun 16

14:49:05 EDT 2009

toolchain: bfin-linux-uclibc-gcc release gcc version 4.3.3

(ADI-trunk/svn-3407)

user-dist: release svn-8255, build #275 Tue Jun 16 14:48:27 EDT 2009

works for me too:

./traps_test -d0 -t -1

[snip]

55/55 tests passed

For some reason - single stepping takes __way__ longer on the release branch

than it does on trunk.... (I still needed to turn the watchdog off when tracing

things)

--- Mike Frysinger 2009-06-16 16:01:24

a ptrace scheduling fix was added in the 2.6.29 timeline which should have

greatly improved performance

--- Mingquan Pan 2009-06-17 06:42:46

I change to 09r1-7 4.3.3 toolchain as you, the traps test ./traps_test -d0 -t

-1 can pass as well. (I am also using trunk kernel.). So it should be due to the

difference of toolchain?

--- Sonic Zhang 2009-06-19 05:24:49

Confirmed on bf537-stamp built into FDPIC by 2009R1-rc7 gcc 4.1.

test suite blocks if run case 1,2 and 3 "traps_test -d0 -t 1 3" in

consequence. test suite passes if run case individually or combine 2/3 and 1/3.

--- Robin Getz 2009-06-21 23:07:38

Hmm...

Just to narrow things down - if compile the kernel with gcc 4.1 - and userspace

with 4.3 (add the 4.1 compiler to your path, do a "make linux", when

that is done, switch the path to 4.3, and "make") does that make

things work or fail? if fail - can you swap and do kernel with 4.3 and userspace

with 4.1?

Thanks

-Robin

--- Mingquan Pan 2009-06-22 03:11:18

Try like this:

kernel uclinux-dist trap_test -d0 -t -1

toolchain 4.1 4.3 ok

4.3 4.1 fail at test 3

And simply booting up uImage with 4.1 kernel and 4.3 uclinux-dist, after

getting the right results, rcp trap_test from the romfs/bin ,which is built from

4.3 kernel and 4.1 uclinux-dist, to the target board, the trap_test -d0 -t -1

also fails at test 3.

## Booting kernel from Legacy Image at 02000000 ...

Image Name: Linux-2.6.30-ADI-2010R1-pre-svn6

Created: 2009-06-22 5:17:53 UTC

Image Type: Blackfin Linux Kernel Image (gzip compressed)

Data Size: 3817025 Bytes = 3.6 MB

Load Address: 00001000

Entry Point: 00187254

Verifying Checksum ... OK

Uncompressing Kernel Image ... OK

Starting Kernel at = 00187254

Linux version 2.6.30-ADI-2010R1-pre-svn6760 (test@linux) (gcc version 4.3.3

(ADI-09r1-7) ) #6 Mon Jun 22 13:17:42 GMT-8 2009

console [early_BFuart0] enabled

early printk enabled on early_BFuart0

Warning: limiting memory to 56MB due to hardware anomaly 05000263

Board Memory: 64MB

Kernel Managed Memory: 64MB

Memory map:

fixedcode = 0x00000400-0x00000490

text = 0x00001000-0x00106100

rodata = 0x00106100-0x00159884

bss = 0x0015a000-0x0016aadc

data = 0x0016aadc-0x0017c000

stack = 0x0017a000-0x0017c000

init = 0x0017c000-0x0078d000

available = 0x0078d000-0x037ff000

DMA Zone = 0x03f00000-0x04000000

Hardware Trace Active and Enabled

Boot Mode: 0

Compiled for ADSP-BF537 Rev 0.2

Blackfin Linux support by http://blackfin.uclinux.org/

Processor Speed: 500 MHz core clock and 125 MHz System Clock

NOMPU: setting up cplb tables

Instruction Cache Enabled for CPU0

External memory: cacheable in instruction cache

Data Cache Enabled for CPU0

External memory: cacheable (write-back) in data cache

Built 1 zonelists in Zone order, mobility grouping off. Total pages: 14223

Kernel command line: root=/dev/mtdblock0 rw earlyprintk=serial,uart0,57600

NR_IRQS:98

Configuring Blackfin Priority Driven Interrupts

PID hash table entries: 256 (order: 8, 1024 bytes)

console handover: boot [early_BFuart0] -> real [ttyBF0]

Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)

Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)

Memory available: 49040k/65536k RAM, (6212k init code, 1044k kernel code, 472k

data, 1024k dma, 7744k reserved)

Calibrating delay loop... 997.37 BogoMIPS (lpj=1994752)

Security Framework initialized

Mount-cache hash table entries: 512

Blackfin Scratchpad data SRAM: 4 KB

Blackfin L1 Data A SRAM: 16 KB (16 KB free)

Blackfin L1 Data B SRAM: 16 KB (16 KB free)

Blackfin L1 Instruction SRAM: 48 KB (36 KB free)

net_namespace: 296 bytes

NET: Registered protocol family 16

Blackfin DMA Controller

stamp_init(): registering device resources

bio: create slab <bio-0> at 0

NET: Registered protocol family 2

IP route cache hash table entries: 1024 (order: 0, 4096 bytes)

TCP established hash table entries: 2048 (order: 2, 16384 bytes)

TCP bind hash table entries: 2048 (order: 1, 8192 bytes)

TCP: Hash tables configured (established 2048 bind 2048)

TCP reno registered

NET: Registered protocol family 1

msgmni has been set to 95

io scheduler noop registered

io scheduler anticipatory registered (default)

io scheduler cfq registered

Serial: Blackfin serial driver

bfin-uart.1: ttyBF0 at MMIO 0xffc00400 (irq = 18) is a BFIN-UART

brd: module loaded

bfin_mii_bus: probed

bfin_mac: attached PHY driver [SMSC LAN83C185] (mii_bus:phy_addr=0:01, irq=-1,

mdc_clk=2500000Hz(mdc_div=24)@sclk=125MHz)

bfin_mac bfin_mac.0: Blackfin on-chip Ethernet MAC driver, Version 1.1

bfin-spi bfin-spi.0: Blackfin on-chip SPI Controller Driver, Version 1.0,

regs_base@ffc00500, dma channel@7

rtc-bfin rtc-bfin: rtc core: registered rtc-bfin as rtc0

bfin-wdt: initialized: timeout=20 sec (nowayout=0)

TCP cubic registered

NET: Registered protocol family 17

rtc-bfin rtc-bfin: setting system clock to 1970-01-01 03:25:04 UTC (12304)

Freeing unused kernel memory: 6212k freed

dma_alloc_init: dma_page @ 0x03039000 - 256 pages at 0x03f00000

_____________________________________

a8888b. / Welcome to the uClinux distribution \

d888888b. / _ _ \

8P"YP"Y88 / | | |_| __ __ (TM) |

8|o||o|88 _____/ | | _ ____ _ _ \ \/ / |

8' .88 \ | | | | _ \| | | | \ / |

8`._.' Y8. \ | |__ | | | | | |_| | / \ |

. d/ `8b. \ \____||_|_| |_|\____|/_/\_\ |

dP . Y8b. \ For embedded processors including |

d8:' " `::88b \ the Analog Devices Blackfin /

d8" 'Y88b \___________________________________/

:8P ' :888

8a. : _a88P For further information, check out:

._/"Yaa_: .| 88P| - http://blackfin.uclinux.org/

\ YP" `| 8P `. - http://docs.blackfin.uclinux.org/

/ \.___.d| .' - http://www.uclinux.org/

`--..__)8888P`._.' jgs/a:f - http://www.analog.com/blackfin

Have a lot of fun...

BusyBox v1.14.1 (2009-06-22 13:13:10 GMT-8) hush - the humble shell

root:/> time /bin/traps_test -d0 -t -1

Running test 0 for exception 0x01: EXCPT 0x01

... PASS (test failed, as expected by signal 5: Trace/breakpoint trap)

Running test 1 for exception 0x02: EXCPT 0x02

... PASS (test failed, as expected by signal 4: Illegal instruction)

Running test 2 for exception 0x03: EXCPT 0x03

... PASS (test failed, as expected by signal 11: Segmentation fault)

Running test 3 for exception 0x04: EXCPT 0x04

... PASS (test failed, as expected by signal 4: Illegal instruction)

Running test 4 for exception 0x05: EXCPT 0x05

... PASS (test failed, as expected by signal 4: Illegal instruction)

Running test 5 for exception 0x06: EXCPT 0x06

... PASS (test failed, as expected by signal 4: Illegal instruction)

Running test 6 for exception 0x07: EXCPT 0x07

... PASS (test failed, as expected by signal 4: Illegal instruction)

...

Running test 54 for exception 0x3f: Stack push to non-existant L1

... PASS (test failed, as expected by signal 7: Bus error)

55/55 tests passed

real 0m 16.53s

user 0m 3.29s

sys 0m 11.14s

root:/>

root:/> ifconfig eth0 10.100.4.50

On host, copy trap_test from romfs/bin which uclinux-dist built with 4.1

toolchain,kernel with 4.3.

rcp /tftpboot/traps_test root@10.100.4.50:/

...

On target again:

root:/> time /traps_test -d0 -t -1

Running test 0 for exception 0x01: EXCPT 0x01

... PASS (test failed, as expected by signal 5: Trace/breakpoint trap)

Running test 1 for exception 0x02: EXCPT 0x02

... PASS (test failed, as expected by signal 4: Illegal instruction)

Running test 2 for exception 0x03: EXCPT 0x03

... PASS (test failed, as expected by signal 11: Segmentation fault)

Running test 3 for exception 0x04: EXCPT 0x04

...

stuck here.

--- Robin Getz 2009-08-14 11:37:16

I can't reproduce - you will need to have Jie have a look.

-Robin

--- Robin Getz 2009-08-14 11:38:12

Assign back to Grace - since I can't reproduce.

--- Mingquan Pan 2009-08-18 06:15:47

Yeah, I try with 09r1-rc9 toolchain, this issue can't be reproduced ,either.

There is something changed between 09r1-rc7 and 09r1-rc9 fixed this, because I

use the same kernel and uclinux source, the former has this bug and the latter

hasn't.

So close.

Files

Changes

Commits

Dependencies

Duplicates

Associations