[#5197] running traps test case in fdpic format would randomly make kernel reboot

Document created by Aaronwu Employee on Sep 3, 2013
Version 1Show Document
  • View in full screen mode

[#5197] running traps test case in fdpic format would randomly make kernel reboot

Submitted By: Mingquan Pan

Open Date

2009-06-03 05:29:04     Close Date

2009-08-18 01:15:18

Priority:

Medium High     Assignee:

Mingquan Pan

Status:

Closed     Fixed In Release:

N/A

Found In Release:

N/A     Release:

Category:

N/A     Board:

N/A

Processor:

BF537     Silicon Revision:

Is this bug repeatable?:

Yes     Resolution:

Assigned (Not Start)

Uboot version or rev.:

    Toolchain version or rev.:

09r1-rc7

App binary format:

N/A     

Summary: running traps test case in fdpic format would randomly make kernel reboot

Details:

 

running traps test case in fdpic format would randomly make kernel reboot on branche head on bf537 stamp board.

 

Sometimes it would stop at case 14, sometimes stops at 11, 9.

 

It looks that it is not seen on bf533 stamp board.

 

log is like:

...

TCP cubic registered

NET: Registered protocol family 17

rtc-bfin rtc-bfin: setting system clock to 1970-01-17 02:20:56 UTC (1390856)

Freeing unused kernel memory: 6572k freed

dma_alloc_init: dma_page @ 0x007e2000 - 256 pages at 0x03f00000

                           _____________________________________

        a8888b.           / Welcome to the uClinux distribution \

       d888888b.         /       _     _                         \

       8P"YP"Y88        /       | |   |_|            __  __ (TM)  |

       8|o||o|88  _____/        | |    _ ____  _   _ \ \/ /       |

       8'    .88       \        | |   | |  _ \| | | | \  /        |

       8`._.' Y8.       \       | |__ | | | | | |_| | /  \        |

      d/      `8b.       \      \____||_|_| |_|\____|/_/\_\       |

     dP   .    Y8b.       \   For embedded processors including   |

    d8:'  "  `::88b        \    the Analog Devices Blackfin      /

   d8"         'Y88b        \___________________________________/

  :8P    '      :888

   8a.   :     _a88P         For further information, check out:

._/"Yaa_:   .| 88P|            - http://blackfin.uclinux.org/

\    YP"    `| 8P  `.          - http://docs.blackfin.uclinux.org/

/     \.___.d|    .'           - http://www.uclinux.org/

`--..__)8888P`._.'  jgs/a:f    - http://www.analog.com/blackfin

 

Have a lot of fun...

 

 

BusyBox v1.13.4 (2009-06-03 17:02:20 CST) built-in shell (msh)

Enter 'help' for a list of built-in commands.

 

root:/> version

kernel:    Linux release 2.6.28.10-ADI-2009R1-svn6543, build #4 Wed Jun 3 17:14:47 CST 2009

toolchain: bfin-linux-uclibc-gcc release gcc version 4.1.2 (ADI svn)

user-dist: release svn-8191, build #2 Wed Jun 3 17:14:02 CST 2009

root:/>

root:/> cd bin/

root:/bin>

root:/bin> ./traps_test -1

...

...    address : <0x0062537c> [ /bin/traps_test + 0x137c ]

    address : <0x00633b24> [ traps_test + 0xb24 ]

    address : <0x00633b2c> [ traps_test + 0xb2c ]

    address : <0x00633b34> [ traps_test + 0xb34 ]

    address : <0x0063c178> [ /lib/ld-uClibc.so.0 + 0x7178 ]

    address : <0x0067ffa2> [ traps_test + 0x1ffa2 ]

    address : <0x0067ffaa> [ traps_test + 0x1ffaa ]

PASS (test failed, as expected by signal 4: Illegal instruction)

 

Running test 9 for exception 0x0a: EXCPT 0x0A

... Unimplmented exception occured

- Maybe you forgot to install a custom exception handler?

Deferred Exception context

CURRENT PROCESS:

COMM=traps_test PID=173

CPU = 0

TEXT = 0x00624000-0x00626ca0        DATA = 0x0063eca0-0x0063f264

BSS = 0x0063f264-0x00660000  USER-STACK = 0x0067fea0

 

return address: [0x006256ee]; contents of:

0x006256c0:  0000  0010  0000  0000  e800  0000  00a8  e801

0x006256d0:  0000  0010  0000  0000  e800  0000  00a9  e801

0x006256e0:  0000  0010  0000  0000  e800  0000  00aa [e801]

0x006256f0:  0000  0010  0000  0000  e800  0000  00ab  e801

 

ADSP-BF537-0.2 500(MHz CCLK) 100(MHz SCLK) (mpu off)

Linux version 2.6.28.10-ADI-2009R1-svn6543

Built with gcc version 4.1.2 (ADI svn)

 

SEQUENCER STATUS:               Not tainted

SEQSTAT: 0006000a  IPEND: 0030  SYSCFG: 0006

  EXCAUSE   : 0xa

  interrupts disabled

  physical IVG5 asserted : <0xffa00c14> { _evt_ivhw + 0x0 }

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x0063c000> /*

 

U-Boot 1.1.6-svn1152 (ADI-2008R2-pre) (Feb 27 2008 - 18:20:24)

 

CPU:   ADSP bf537-0.2 (Detected Rev: 0.2)

Board: ADI BF537 stamp board

       Support: http://blackfin.uclinux.org/

Clock: VCO: 500 MHz, Core: 500 MHz, System: 100 MHz

RAM:   64 MB

Flash:  4 MB

In:    serial

Out:   serial

Err:   serial

Net:   Blackfin EMAC

MAC:   00:E0:22:FE:B5:B5

I2C:   ready                                                                                                                                              

 

Follow-ups

 

--- Mike Frysinger                                           2009-06-03 05:34:04

make sure it isnt the watchdog causing the reboot.  the next kernel boot will

tell you why the reset happened.  or try using `dmesg -n1` before running the

traps_test.

 

--- Mingquan Pan                                             2009-06-03 06:11:10

Yeah, you are right. After adding "dmesg -n1" before running the case,

it fails at case 18 as using flat kernel every time.

 

Is there a way to make watchdog not so sensitive? Since in flat format, running

the case wouldn't trigger the watchdog and reboot like this. Otherwise I have to

add this command in script, and would miss the dump info in log.

 

--- Mike Frysinger                                           2009-06-03 06:18:05

you could try launching watchdogd with -w 60 -k 5

 

--- Robin Getz                                               2009-06-03 09:50:37

Grace:

 

There should be no need to set dmesg, or change the watchdog.

 

root:/> time ./traps_test -d0  -1

[snip]

55/55 tests passed

real    0m 1.02s

user    0m 0.01s

sys     0m 0.04s

 

All the traps test take less than a second to run. Even if we trace them,

 

root:/> time ./traps_test -d0 -t -1

[snip]

55/55 tests passed

real    0m 5.05s

user    0m 1.09s

sys     0m 3.53s

 

it takes less than 5 seconds.

 

If you get a test failure - then you can run it with "-v" (verbose)

to see the kernel messages.

 

-Robin

 

--- Mingquan Pan                                             2009-06-03 22:06:26

Are "-t" "-v" the parameters for ./traps_test?

Why it doesn't show by help?

root:/bin> ./traps_test -h

Usage: ./traps_test [-c count] [-d milliseconds] [-q] [-l] [starting test

number] [ending test number]

 

-c count        Repeat the test(s) count times before stopping

-d seconds      The number of milliseconds to delay between flushing stdout,

and

                running the test (default is 1)

-l              List tests, then quit

-q              Quiet (don't print out test info)

If no test number is specified, the number of tests available will be shown.

 

If a single test number is specified (0 <= n < # of tests), that test

will be run.

 

If two tests numbers are specified (0 <= start < end < # tests), those

tests will be run

 

If you specify -1, then all tests will be run in order.

 

root:/bin>

 

root:/bin> time ./traps_test -d0 -t -1

./traps_test: invalid option -- t

Usage: ./traps_test [-c count] [-d milliseconds] [-q] [-l] [starting test

number] [ending test number]

 

-c count        Repeat the test(s) count times before stopping

-d seconds      The number of milliseconds to delay between flushing stdout,

and

                running the test (default is 1)

-l              List tests, then quit

-q              Quiet (don't print out test info)

If no test number is specified, the number of tests available will be shown.

 

If a single test number is specified (0 <= n < # of tests), that test

will be run.

 

If two tests numbers are specified (0 <= start < end < # tests), those

tests will be run

 

If you specify -1, then all tests will be run in order.

 

 

ERROR: unknown option

Command exited with non-zero status 1

real    0m 0.00s

user    0m 0.00s

sys     0m 0.00s

 

--- Robin Getz                                               2009-06-03 23:37:15

Grace:

 

What toolchain are you using? I'm getting some problems with 4.3 on 0.3

silicon, compiled as 0.2 -- I'm still digging into things to figure it out.

 

--- Robin Getz                                               2009-06-04 00:34:32

Grace:

 

For now - use the traps test from trunk. I have not checked into the branch

yet.

 

-Robin

 

root:/> traps_test -h

Usage: traps_test [-c count] [-d milliseconds] [-e] [-q] [-l] [-p] [-t] [-v]

[starting test number] [ending test number]

 

-c count        Repeat the test(s) count times before stopping

-d seconds      The number of milliseconds to delay between flushing stdout,

and

                running the test (default is 1)

-e              exclude the checking of excause is as expected

-l              List tests, then quit

-q              Quiet (don't print out test info)

-p              Run the test in the parent process, otherwise fork a child

process.

                Only valid for a single test.

-t              Trace (single step) through the failing tests, to simulate gdb

-v              Verbose. equivilent to 'dmesg -n 7', so that all kernel

messages print out

                otherwise do 'dmesg -n 3', so things are a little quieter

 

If no test number is specified, the number of tests available will be shown.

 

If a single test number is specified (0 <= n < # of tests), that test

will be run.

 

If two tests numbers are specified (0 <= start < end < # tests), those

tests will be run

 

If you specify -1, then all tests will be run in order.

 

 

--- Robin Getz                                               2009-06-14 23:22:47

Grace:

 

Can you make sure what is on trunk fixes things for you - then I will commit to

the branch.

 

-Robin

 

--- Mingquan Pan                                             2009-06-16 06:36:28

Yeah, it is really useful by saving the lots of dump info.

 

the bf537 board won't reboot randomly now when running: ./traps_test -d0 -1 in

fdpic format.

 

But it looks running ./traps_test  with -t or not would get results

different(built in fdpic).

./traps_test -d0  -1 running ok.

./traps_test -d0  -t -1 would stuck at case 3.

 

root:/bin> ./traps_test -d0  -t -1

 

Running test 0 for exception 0x01: EXCPT 0x01

... PASS (test failed, as expected by signal 5: Trace/breakpoint trap)

 

Running test 1 for exception 0x02: EXCPT 0x02

... PASS (test failed, as expected by signal 4: Illegal instruction)

 

Running test 2 for exception 0x03: EXCPT 0x03

... PASS (test failed, as expected by signal 11: Segmentation fault)

 

Running test 3 for exception 0x04: EXCPT 0x04

...

 

stucks here.

 

Run case 3 alone with -t wouldn't have this problem.

 

root:/> cat /proc/cp

/proc/cplbinfo/  /proc/cpuinfo

root:/> cat /proc/cpuinfo

processor       : 0

vendor_id       : Analog Devices

cpu family      : 0x27c8

model name      : ADSP-BF537 500(MHz CCLK) 125(MHz SCLK) (mpu off)

stepping        : 2

cpu MHz         : 500.000/125.000

bogomips        : 997.37

Calibration     : 498688000 loops

cache size      : 16 KB(L1 icache) 32 KB(L1 dcache-wb) 0 KB(L2 cache)

dbank-A/B       : cache/cache

icache setup    : 4 Sub-banks/4 Ways, 32 Lines/Way

dcache setup    : 2 Super-banks/4 Sub-banks/2 Ways, 64 Lines/Way

board name      : ADI BF537-STAMP

board memory    : 65536 kB (0x(null) -> 0x04000000)

kernel memory   : 57336 kB (0x00001000 -> 0x037ff000)

 

root:/> version

kernel:    Linux release 2.6.30-ADI-2010R1-pre-svn6699, build #3 Tue Jun 16

18:23:20 CST 2009

toolchain: bfin-linux-uclibc-gcc release gcc version 4.1.2 (ADI svn)

user-dist: release svn-8293, build #1 Tue Jun 16 18:22:18 CST 2009

 

--- Robin Getz                                               2009-06-16 14:38:45

On trunk it works...

 

root:/> cat /proc/cpuinfo

processor       : 0

vendor_id       : Analog Devices

cpu family      : 0x27c8

model name      : ADSP-BF537 500(MHz CCLK) 125(MHz SCLK) (mpu off)

stepping        : 2

 

root:/> version

kernel:    Linux release 2.6.28.10-ADI-2010R1-pre-svn6607, build #41 Tue Jun 16

10:57:35 EDT 2009

toolchain: bfin-linux-uclibc-gcc release gcc version 4.3.3

(ADI-trunk/svn-3407)

user-dist: release svn-8255, build #274 Tue Jun 16 10:57:01 EDT 2009

 

./traps_test -d0  -t -1

[snip]

55/55 tests passed

 

/traps_test -d0 -t -c 100 -1  

[snip]

55/55 tests passed

 

I'm just loading up the release kernel, and seeing how that works.

 

-Robin

 

--- Robin Getz                                               2009-06-16 15:57:38

root:/> cat /proc/cpuinfo

processor       : 0

vendor_id       : Analog Devices

cpu family      : 0x27c8

model name      : ADSP-BF537 500(MHz CCLK) 125(MHz SCLK) (mpu off)

stepping        : 2

 

 

root:/> version

kernel:    Linux release 2.6.28.10-ADI-2009R1-svn6753, build #2 Tue Jun 16

14:49:05 EDT 2009

toolchain: bfin-linux-uclibc-gcc release gcc version 4.3.3

(ADI-trunk/svn-3407)

user-dist: release svn-8255, build #275 Tue Jun 16 14:48:27 EDT 2009

 

works for me too:

 

./traps_test -d0  -t -1

[snip]

55/55 tests passed

 

For some reason - single stepping takes __way__ longer on the release branch

than it does on trunk.... (I still needed to turn the watchdog off when tracing

things)

 

 

--- Mike Frysinger                                           2009-06-16 16:01:24

a ptrace scheduling fix was added in the 2.6.29 timeline which should have

greatly improved performance

 

--- Mingquan Pan                                             2009-06-17 06:42:46

I change to 09r1-7 4.3.3 toolchain as you, the traps test ./traps_test -d0  -t

-1 can pass as well. (I am also using trunk kernel.). So it should be due to the

difference of toolchain?

 

--- Sonic Zhang                                              2009-06-19 05:24:49

Confirmed on bf537-stamp built into FDPIC by 2009R1-rc7 gcc 4.1.

test suite blocks if run case 1,2 and 3 "traps_test -d0  -t 1 3" in

consequence. test suite passes if run case individually or combine 2/3 and 1/3.

 

--- Robin Getz                                               2009-06-21 23:07:38

Hmm...

 

Just to narrow things down - if compile the kernel with gcc 4.1 - and userspace

with 4.3 (add the 4.1 compiler to your path, do a "make linux", when

that is done, switch the path to 4.3, and "make") does that make

things work or fail? if fail - can you swap and do kernel with 4.3 and userspace

with 4.1?

 

Thanks

-Robin

 

--- Mingquan Pan                                             2009-06-22 03:11:18

Try like this:

 

            kernel  uclinux-dist       trap_test -d0 -t -1

toolchain     4.1     4.3                  ok

              4.3     4.1                  fail at test 3

 

And  simply booting up uImage with 4.1 kernel and 4.3 uclinux-dist, after

getting the right results, rcp trap_test from the romfs/bin ,which is built from

4.3 kernel and 4.1 uclinux-dist,  to the target board, the  trap_test -d0 -t -1

also fails at test 3.

 

## Booting kernel from Legacy Image at 02000000 ...

   Image Name:   Linux-2.6.30-ADI-2010R1-pre-svn6

   Created:      2009-06-22   5:17:53 UTC

   Image Type:   Blackfin Linux Kernel Image (gzip compressed)

   Data Size:    3817025 Bytes =  3.6 MB

   Load Address: 00001000

   Entry Point:  00187254

   Verifying Checksum ... OK

   Uncompressing Kernel Image ... OK

Starting Kernel at = 00187254

Linux version 2.6.30-ADI-2010R1-pre-svn6760 (test@linux) (gcc version 4.3.3

(ADI-09r1-7) ) #6 Mon Jun 22 13:17:42 GMT-8 2009

console [early_BFuart0] enabled

early printk enabled on early_BFuart0

Warning: limiting memory to 56MB due to hardware anomaly 05000263

Board Memory: 64MB

Kernel Managed Memory: 64MB

Memory map:

  fixedcode = 0x00000400-0x00000490

  text      = 0x00001000-0x00106100

  rodata    = 0x00106100-0x00159884

  bss       = 0x0015a000-0x0016aadc

  data      = 0x0016aadc-0x0017c000

    stack   = 0x0017a000-0x0017c000

  init      = 0x0017c000-0x0078d000

  available = 0x0078d000-0x037ff000

  DMA Zone  = 0x03f00000-0x04000000

Hardware Trace Active and Enabled

Boot Mode: 0

Blackfin support (C) 2004-2009 Analog Devices, Inc.

Compiled for ADSP-BF537 Rev 0.2

Blackfin Linux support by http://blackfin.uclinux.org/

Processor Speed: 500 MHz core clock and 125 MHz System Clock

NOMPU: setting up cplb tables

Instruction Cache Enabled for CPU0

  External memory: cacheable in instruction cache

Data Cache Enabled for CPU0

  External memory: cacheable (write-back) in data cache

Built 1 zonelists in Zone order, mobility grouping off.  Total pages: 14223

Kernel command line: root=/dev/mtdblock0 rw earlyprintk=serial,uart0,57600

NR_IRQS:98

Configuring Blackfin Priority Driven Interrupts

PID hash table entries: 256 (order: 8, 1024 bytes)

console handover: boot [early_BFuart0] -> real [ttyBF0]

Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)

Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)

Memory available: 49040k/65536k RAM, (6212k init code, 1044k kernel code, 472k

data, 1024k dma, 7744k reserved)

Calibrating delay loop... 997.37 BogoMIPS (lpj=1994752)

Security Framework initialized

Mount-cache hash table entries: 512

Blackfin Scratchpad data SRAM: 4 KB

Blackfin L1 Data A SRAM: 16 KB (16 KB free)

Blackfin L1 Data B SRAM: 16 KB (16 KB free)

Blackfin L1 Instruction SRAM: 48 KB (36 KB free)

net_namespace: 296 bytes

NET: Registered protocol family 16

Blackfin DMA Controller

stamp_init(): registering device resources

bio: create slab <bio-0> at 0

NET: Registered protocol family 2

IP route cache hash table entries: 1024 (order: 0, 4096 bytes)

TCP established hash table entries: 2048 (order: 2, 16384 bytes)

TCP bind hash table entries: 2048 (order: 1, 8192 bytes)

TCP: Hash tables configured (established 2048 bind 2048)

TCP reno registered

NET: Registered protocol family 1

msgmni has been set to 95

io scheduler noop registered

io scheduler anticipatory registered (default)

io scheduler cfq registered

Serial: Blackfin serial driver

bfin-uart.1: ttyBF0 at MMIO 0xffc00400 (irq = 18) is a BFIN-UART

brd: module loaded

bfin_mii_bus: probed

bfin_mac: attached PHY driver [SMSC LAN83C185] (mii_bus:phy_addr=0:01, irq=-1,

mdc_clk=2500000Hz(mdc_div=24)@sclk=125MHz)

bfin_mac bfin_mac.0: Blackfin on-chip Ethernet MAC driver, Version 1.1

bfin-spi bfin-spi.0: Blackfin on-chip SPI Controller Driver, Version 1.0,

regs_base@ffc00500, dma channel@7

rtc-bfin rtc-bfin: rtc core: registered rtc-bfin as rtc0

bfin-wdt: initialized: timeout=20 sec (nowayout=0)

TCP cubic registered

NET: Registered protocol family 17

rtc-bfin rtc-bfin: setting system clock to 1970-01-01 03:25:04 UTC (12304)

Freeing unused kernel memory: 6212k freed

dma_alloc_init: dma_page @ 0x03039000 - 256 pages at 0x03f00000

                           _____________________________________

        a8888b.           / Welcome to the uClinux distribution \

       d888888b.         /       _     _                         \

       8P"YP"Y88        /       | |   |_|            __  __ (TM)  |

       8|o||o|88  _____/        | |    _ ____  _   _ \ \/ /       |

       8'    .88       \        | |   | |  _ \| | | | \  /        |

       8`._.' Y8.       \       | |__ | | | | | |_| | /  \        |

  .   d/      `8b.       \      \____||_|_| |_|\____|/_/\_\       |

     dP   .    Y8b.       \   For embedded processors including   |

    d8:'  "  `::88b        \    the Analog Devices Blackfin      /

   d8"         'Y88b        \___________________________________/

  :8P    '      :888

   8a.   :     _a88P         For further information, check out:

._/"Yaa_:   .| 88P|            - http://blackfin.uclinux.org/

\    YP"    `| 8P  `.          - http://docs.blackfin.uclinux.org/

/     \.___.d|    .'           - http://www.uclinux.org/

`--..__)8888P`._.'  jgs/a:f    - http://www.analog.com/blackfin

 

Have a lot of fun...

 

 

BusyBox v1.14.1 (2009-06-22 13:13:10 GMT-8) hush - the humble shell

 

root:/> time /bin/traps_test -d0 -t -1

 

Running test 0 for exception 0x01: EXCPT 0x01

... PASS (test failed, as expected by signal 5: Trace/breakpoint trap)

 

Running test 1 for exception 0x02: EXCPT 0x02

... PASS (test failed, as expected by signal 4: Illegal instruction)

 

Running test 2 for exception 0x03: EXCPT 0x03

... PASS (test failed, as expected by signal 11: Segmentation fault)

 

Running test 3 for exception 0x04: EXCPT 0x04

... PASS (test failed, as expected by signal 4: Illegal instruction)

 

Running test 4 for exception 0x05: EXCPT 0x05

... PASS (test failed, as expected by signal 4: Illegal instruction)

 

Running test 5 for exception 0x06: EXCPT 0x06

... PASS (test failed, as expected by signal 4: Illegal instruction)

 

Running test 6 for exception 0x07: EXCPT 0x07

... PASS (test failed, as expected by signal 4: Illegal instruction)

                                                                              

                                                                           ...

Running test 54 for exception 0x3f: Stack push to non-existant L1

... PASS (test failed, as expected by signal 7: Bus error)

 

55/55 tests passed

real    0m 16.53s

user    0m 3.29s

sys     0m 11.14s

root:/>

root:/> ifconfig eth0  10.100.4.50

 

 

On host, copy trap_test from romfs/bin which uclinux-dist built with 4.1

toolchain,kernel with 4.3.

..

rcp /tftpboot/traps_test root@10.100.4.50:/

...

 

On target again:

root:/> time /traps_test -d0 -t -1

 

Running test 0 for exception 0x01: EXCPT 0x01

... PASS (test failed, as expected by signal 5: Trace/breakpoint trap)

 

Running test 1 for exception 0x02: EXCPT 0x02

... PASS (test failed, as expected by signal 4: Illegal instruction)

 

Running test 2 for exception 0x03: EXCPT 0x03

... PASS (test failed, as expected by signal 11: Segmentation fault)

 

Running test 3 for exception 0x04: EXCPT 0x04

...

stuck here.

 

--- Robin Getz                                               2009-08-14 11:37:16

I can't reproduce - you will need to have Jie have a look.

 

-Robin

 

--- Robin Getz                                               2009-08-14 11:38:12

Assign back to Grace - since I can't reproduce.

 

--- Mingquan Pan                                             2009-08-18 06:15:47

Yeah, I try with 09r1-rc9 toolchain, this issue can't be reproduced ,either.

 

There is something changed between 09r1-rc7 and 09r1-rc9 fixed this, because I

use the same kernel and uclinux source, the former has this bug and the latter

hasn't.

 

So close.

 

 

 

    Files

    Changes

    Commits

    Dependencies

    Duplicates

    Associations

    Tags

 

File Name     File Type     File Size     Posted By

No Files Were Found

Attachments

    Outcomes