[#4899] ethernet card would fail to function sometimes

Document created by Aaronwu Employee on Aug 30, 2013
Version 1Show Document
  • View in full screen mode

[#4899] ethernet card would fail to function sometimes

Submitted By: Mingquan Pan

Open Date

2009-02-12 21:01:04     Close Date

2010-01-13 04:02:36

Priority:

Medium     Assignee:

Mingquan Pan

Status:

Closed     Fixed In Release:

2009R1-RC6

Found In Release:

2009R1-RC6     Release:

Category:

N/A     Board:

N/A

Processor:

BF548     Silicon Revision:

Is this bug repeatable?:

Yes     Resolution:

Not reproducible

Uboot version or rev.:

    Toolchain version or rev.:

4.1 toolchain of Jan 16

App binary format:

N/A     

Summary: ethernet card would fail to function sometimes

Details:

 

ethernet card on bf548 ezkit would fail to function sometimes. It looks if the board is left alone after booting up for an hour, it could be happening. Because in toolchain testing, after doing simulator test which is about an hour, and then it comes to flat testing, the board would be stuck at:

 

On host:

test     24376  0.0  0.0   2104   756 pts/3    S    Feb12   0:00 rcp root 10.100.4.50 /bin/gdbserver /home/test/work/cruise/temp/regtest_build4.1/tests/har

postfix  25651  0.0  0.1   5836  1804 ?        S    09:24   0:00 pickup -l -t fifo -u

 

the log is:

Targets: uclinux

Components: binutils gas ld gdb gcc g++ gfortranlibstdc++ libmudflap newlib

Host: i686-pc-linux-gnu

Build: i686-pc-linux-gnu

Found multiple versions of runtest, using the one at /usr/bin/runtest

Found multiple versions of bc, using the one at /usr/bin/bc

Found multiple versions of killall, using the one at /usr/bin/killall

Found multiple versions of pstree, using the one at /usr/bin/pstree

Found multiple versions of rsh, using the one at /usr/bin/rsh

Found multiple versions of rcp, using the one at /usr/bin/rcp

Toolchain source is at      /home/test/work/cruise/checkouts/toolchain

Build will be done in       /home/test/work/cruise/temp/regtest_build4.1

Path for test results

  uclinux (10.100.4.50):  /home/test/work/cruise/temp/regtest_build4.1/tests/hardware-flat

Creating log file           /home/test/work/cruise/temp/regtest_build4.1/logs/regtest.log

 

 

On the target:

 

root:/> version

kernel:    Linux release 2.6.28.3-ADI-2009R1-pre-svn6089, build #29 Thu Feb 12 14:18:53 CST 2009

toolchain: bfin-uclinux-gcc release gcc version 4.1.2 (ADI svn)

user-dist: release svn-7730, build #10 Thu Feb 12 14:17:41 CST 2009

root:/> ifconfig

eth0      Link encap:Ethernet  HWaddr 00:E0:22:FE:BD:0A

          inet addr:10.100.4.50  Bcast:10.100.4.255  Mask:255.255.255.0

          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

          RX packets:72 errors:0 dropped:0 overruns:0 frame:0

          TX packets:81 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:1000

          RX bytes:5093 (4.9 KiB)  TX bytes:48920 (47.7 KiB)

          Interrupt:175

 

lo        Link encap:Local Loopback

          inet addr:127.0.0.1  Mask:255.0.0.0

          UP LOOPBACK RUNNING  MTU:16436  Metric:1

          RX packets:0 errors:0 dropped:0 overruns:0 frame:0

          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

          collisions:0 txqueuelen:0

          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

 

root:/> ping 10.100.4.174

PING 10.100.4.174 (10.100.4.174): 56 data bytes

^C

--- 10.100.4.174 ping statistics ---

17 packets transmitted, 0 packets received, 100% packet loss

root:/> uptime

14:40:47 up 17:50, load average: 0.00, 0.00, 0.00

root:/> dmesg

Linux version 2.6.28.3-ADI-2009R1-pre-svn6089 (test@44-bf548-toolchain) (gcc version 4.1.2 (ADI svn)) #29 Thu Feb 12 14:18:53 CST 2009

console [early_BFuart0] enabled

early printk enabled on early_BFuart0

Board Memory: 64MB

Kernel Managed Memory: 64MB

Memory map:

  fixedcode = 0x00000400-0x00000490

  text      = 0x00001000-0x001a9740

  rodata    = 0x001a9740-0x002378b4

  bss       = 0x002378c0-0x00249954

  data      = 0x00249954-0x00262000

    stack   = 0x00260000-0x00262000

  init      = 0x00262000-0x005b6000

  available = 0x005b6000-0x03dff000

  DMA Zone  = 0x03e00000-0x04000000

Hardware Trace Active and Enabled

Boot Mode: 3

Blackfin support (C) 2004-2009 Analog Devices, Inc.

Compiled for ADSP-BF548 Rev 0.0

Warning: Compiled for Rev 0, but running on Rev 2

Blackfin Linux support by   blackfin.uclinux.org/

Processor Speed: 525 MHz core clock and 131 MHz System Clock

boot memmap: 00000000005b6000 - 0000000003dff000 (usable)

On node 0 totalpages: 15871

free_area_init_node: node 0, pgdat 0025d6dc, node_mem_map 005bb000

  DMA zone: 124 pages used for memmap

  DMA zone: 0 pages reserved

  DMA zone: 15747 pages, LIFO batch:3

  Normal zone: 0 pages used for memmap

  Movable zone: 0 pages used for memmap

NOMPU: setting up cplb tables

Instruction Cache Enabled for CPU0

Data Cache Enabled for CPU0 (write-through)

Built 1 zonelists in Zone order, mobility grouping off.  Total pages: 15747

Kernel command line: root=/dev/mtdblock0 rw earlyprintk=serial,uart0,57600 panic=3 ip=10.100.4.50 ip=10.100.4.50:10.100.4.174:10.100.4.174:255.255.255.0:bf

Configuring Blackfin Priority Driven Interrupts

PID hash table entries: 256 (order: 8, 1024 bytes)

console handover: boot [early_BFuart0] -> real [ttyBF0]

Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)

Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)

Kernel managed physical pages: 15871

Memory available: 57056k/65536k RAM, (3408k init code, 1697k kernel code, 739k data, 2048k dma, 584k reserved)

Calibrating delay loop... 1046.52 BogoMIPS (lpj=2093056)

Security Framework initialized

Mount-cache hash table entries: 512

Blackfin Scratchpad data SRAM: 4 KB

Blackfin L1 Data A SRAM: 16 KB (15 KB free)

Blackfin L1 Data B SRAM: 16 KB (16 KB free)

Blackfin L1 Instruction SRAM: 48 KB (42 KB free)

Blackfin L2 SRAM: 128 KB (128 KB free)

PDA for CPU0 reserved at 00239178

net_namespace: 288 bytes

NET: Registered protocol family 16

Blackfin GPIO Controller

Blackfin DMA Controller

ezkit_init(): registering device resources

SCSI subsystem initialized

libata version 3.00 loaded.

usbcore: registered new interface driver usbfs

usbcore: registered new interface driver hub

usbcore: registered new device driver usb

musb_hdrc: version 6.0, musb-dma, host, debug=0

musb_hdrc: ConfigData=0x00 (UTMI-8)

musb_hdrc: HDRC RTL version 0.0

musb_hdrc: hw_ep 0shared, max 64

musb_hdrc: hw_ep 1tx, max 128

musb_hdrc: hw_ep 1rx, max 128

musb_hdrc: hw_ep 2tx, max 128

musb_hdrc: hw_ep 2rx, max 128

musb_hdrc: hw_ep 3tx, max 128

musb_hdrc: hw_ep 3rx, max 128

musb_hdrc: hw_ep 4tx, max 128

musb_hdrc: hw_ep 4rx, max 128

musb_hdrc: hw_ep 5tx, max 1024

musb_hdrc: hw_ep 5rx, max 1024

musb_hdrc: hw_ep 6tx, max 1024

musb_hdrc: hw_ep 6rx, max 1024

musb_hdrc: hw_ep 7tx, max 1024

musb_hdrc: hw_ep 7rx, max 1024

musb_hdrc: USB Host mode controller at ffc03c00 using DMA, IRQ 82

musb_hdrc musb_hdrc.0: MUSB HDRC host driver

musb_hdrc musb_hdrc.0: new USB bus registered, assigned bus number 1

usb usb1: configuration #1 chosen from 1 choice

hub 1-0:1.0: USB hub found

dma_alloc_init: dma_page @ 0x0066c000 - 512 pages at 0x03e00000

hub 1-0:1.0: 1 port detected

NET: Registered protocol family 2

IP route cache hash table entries: 1024 (order: 0, 4096 bytes)

TCP established hash table entries: 2048 (order: 2, 16384 bytes)

TCP bind hash table entries: 2048 (order: 1, 8192 bytes)

TCP: Hash tables configured (established 2048 bind 2048)

TCP reno registered

NET: Registered protocol family 1

Setting up Blackfin MMR debugfs

msgmni has been set to 111

io scheduler noop registered

io scheduler anticipatory registered (default)

io scheduler cfq registered

bf54x-lq043: FrameBuffer initializing...

bfin-otp: initialized

Serial: Blackfin serial driver

bfin-uart.1: ttyBF0 at MMIO 0xffc02000 (irq = 48) is a BFIN-UART

brd: module loaded

smsc911x: Driver version 2007-07-13.

eth0: SMSC911x MAC Address: 00:e0:22:fe:bd:0a

Driver 'sd' needs updating - please use bus_type methods

Driver 'sr' needs updating - please use bus_type methods

register bfin atapi driver

scsi0 : pata-bf54x

ata1: PATA max UDMA/66 irq 68

ata1.00: ATA-7: FUJITSU MHW2040AT, 0000000B, max UDMA/100

ata1.00: 78140160 sectors, multi 16: LBA

ata1.00: configured for UDMA/66

blk_queue_max_hw_segments: set to minimum 1

scsi 0:0:0:0: Direct-Access     ATA      FUJITSU MHW2040A 0000 PQ: 0 ANSI: 5

sd 0:0:0:0: [sda] 78140160 512-byte hardware sectors: (40.0 GB/37.2 GiB)

sd 0:0:0:0: [sda] Write Protect is off

sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00

sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

sd 0:0:0:0: [sda] 78140160 512-byte hardware sectors: (40.0 GB/37.2 GiB)

sd 0:0:0:0: [sda] Write Protect is off

sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00

sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA

sda: sda1

sd 0:0:0:0: [sda] Attached SCSI disk

physmap platform flash device: 02000000 at 20000000

physmap-flash.0: Found 1 x16 devices at 0x0 in 16-bit bank

physmap-flash.0: Found an alias at 0x1000000 for the chip at 0x0

Intel/Sharp Extended Query Table at 0x010A

Intel/Sharp Extended Query Table at 0x010A

Intel/Sharp Extended Query Table at 0x010A

Intel/Sharp Extended Query Table at 0x010A

Intel/Sharp Extended Query Table at 0x010A

Using buffer write method

Using auto-unlock on power-up/resume

cfi_cmdset_0001: Erase suspend on write enabled

erase region 0: offset=0x0,size=0x20000,blocks=127

erase region 1: offset=0xfe0000,size=0x8000,blocks=4

RedBoot partition parsing not available

Using physmap partition information

Creating 3 MTD partitions on "physmap-flash.0":

0x00000000-0x00040000 : "bootloader(nor)"

0x00040000-0x00440000 : "linux kernel(nor)"

0x00440000-0x01000000 : "file system(nor)"

BF5xx on-chip NAND FLash Controller Driver, Version 1.2 (c) 2007 Analog Devices, Inc.

bf5xx-nand bf5xx-nand.0: page_size=256, data_width=8, wr_dly=3, rd_dly=3

NAND device: Manufacturer ID: 0x20, Chip ID: 0xda (ST Micro NAND 256MiB 3,3V 8-bit)

Creating 2 MTD partitions on "NAND 256MiB 3,3V 8-bit":

0x00000000-0x00400000 : "linux kernel(nand)"

0x00400000-0x10000000 : "file system(nand)"

m25p80 spi0.1: m25p16 (2048 Kbytes)

Creating 2 MTD partitions on "m25p80":

0x00000000-0x00040000 : "bootloader(spi)"

0x00040000-0x00200000 : "linux kernel(spi)"

bfin-spi bfin-spi.0: Blackfin on-chip SPI Controller Driver, Version 1.0, regs_base@ffc00500, dma channel@4

bfin-spi bfin-spi.1: Blackfin on-chip SPI Controller Driver, Version 1.0, regs_base@ffc02300, dma channel@5

input: bf54x-keys as /devices/platform/bf54x-keys/input/input0

bf54x-keys: Blackfin BF54x Keypad registered IRQ 76

rtc-bfin rtc-bfin: rtc core: registered rtc-bfin as rtc0

i2c /dev entries driver

i2c-bfin-twi i2c-bfin-twi.0: Blackfin BF5xx on-chip I2C TWI Contoller, regs_base@ffc00700

i2c-bfin-twi i2c-bfin-twi.1: Blackfin BF5xx on-chip I2C TWI Contoller, regs_base@ffc02200

hid_zpff: Unknown symbol _input_ff_create_memless

usbcore: registered new interface driver usbhid

usbhid: v2.6:USB HID core driver

Advanced Linux Sound Architecture Driver Version 1.0.18rc3.

ASoC version 0.13.2

dma rx:0 tx:1, err irq:10, regs:ffc00800

AD1980 SoC Audio Codec

asoc: AC97 <-> bf5xx-ac97 mapping ok

ALSA device list:

  #0: bf5xx-board (AD1980)

TCP cubic registered

NET: Registered protocol family 17

rtc-bfin rtc-bfin: setting system clock to 1970-01-28 20:50:29 UTC (2407829)

eth0: SMSC911x/921x identified at 0x24000000, IRQ: 175

eth0: link down

IP-Config: Complete:

     device=eth0, addr=10.100.4.50, mask=255.255.255.0, gw=10.100.4.174,

     host=bf548-ezkit, domain=, nis-domain=(none),

     bootserver=10.100.4.174, rootserver=10.100.4.174, rootpath=

Freeing unused kernel memory: 3408k freed

eth0: link up, 100Mbps, full-duplex, lpa 0x41E1

 

Follow-ups

 

--- Mingquan Pan                                             2009-02-12 21:30:31

It looks this problem is not specific to Bf548, it is also found on bf527 ezkit

board. But when it fails to ping from the host, the bf548 is still active on

console, while bf527 is complete stuck and no response to key stroke at all.

 

est      3641  0.0  0.0   9928   580 ?        S    Jan15   0:04 sshd:

test@pts/2

test      3642  0.0  0.1   5052  1280 pts/2    Ss+  Jan15   0:03 -bash

test      9861  0.0  2.1 260288 21932 pts/2    Sl   Feb12   0:49

/usr/lib/jvm/java-1.5.0-sun-1.5.0_update16/bin/java -cp

/usr/lib/jvm/java/lib/tools.jar:/h

root      9927  0.0  0.0      0     0 ?        S    Feb12   0:06 [pdflush]

root      9932  0.0  0.0      0     0 ?        S    Feb12   0:01 [pdflush]

test      9955  0.0  1.9 247476 20236 pts/2    Sl   Feb12   0:30

/usr/lib/jvm/java-1.5.0-sun-1.5.0_update16/bin/java -classpath

/home/test/work/cruise/apac

test      9972  0.0  0.1   4256  1412 pts/2    S    Feb12   0:00 /bin/sh

../../test_scripts/toolchain/toolchain_test

test     23571  1.7  4.0 113580 41036 ?        SNl  Feb10  69:28 beagled-helper

/usr/lib/beagle/IndexHelper.exe

postfix  27914  0.0  0.1   5836  1808 ?        S    09:43   0:00 pickup -l -t

fifo -u

test     28011 97.9  0.0   7792   644 ?        Rs   Jan20 34348:21

bfin-linux-uclibc-gdb -nw -nx -i=mi

test     28371  0.0  0.0   2580   992 pts/1    R+   10:24   0:00 ps aux

test     28899  0.0  0.1   4520  1748 pts/2    S    Feb12   0:02 /bin/sh

./toolchain-regtest -s /home/test/work/cruise/checkouts/toolchain -o

/home/test/wo

test     30334  0.0  0.0   4520   804 pts/2    S    Feb12   0:00 /bin/sh

./toolchain-regtest -s /home/test/work/cruise/checkouts/toolchain -o

/home/test/wo

test     30335  0.0  0.0   2092   740 pts/2    S    Feb12   0:00 rsh -l root

10.100.4.50 ls /  1>/dev/null

test     30336  0.0  0.0   2896   576 pts/2    S    Feb12   0:00 wc -l

test     30338  0.0  0.0      0     0 pts/2    Z    Feb12   0:00 [rsh]

<defunct>

test@45-bf527-toolchain:~/work/cruise/test_scripts/toolchain/toolchain-build>

ping 10.100.4.50

PING 10.100.4.50 (10.100.4.50) 56(84) bytes of data.

^C

--- 10.100.4.50 ping statistics ---

2 packets transmitted, 0 received, 100% packet loss, time 999ms

 

And this stuck on bf527 happens between the libstdc++ flat testing.

 

from the log, we can see:

PASS: 23_containers/deque/cons/clear_allocator.cc (test for excess errors)

Executing on bfin-uclinux: /tmp/clear_allocator.exe.21041    (timeout = 300)

Executing on bfin-uclinux: rm -f  /tmp/clear_allocator.exe.21041    (timeout =

300)

Executed ./clear_allocator.exe, status 0

PASS: 23_containers/deque/cons/clear_allocator.cc execution test

extra_tool_flags are:

 

Executing on host: bfin-uclinux-g++ -g -O2 -D_GLIBCXX_ASSERT -fmessage-length=0

-DLOCALEDIR="/home/test/work/cruise/temp/bfin-uclinux/bin/../share/locale"

-I/home/test/work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite

/home/test/work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite/23_co

ntainers/deque/explicit_instantiation/1.cc     -S  -mstack-check-l1   -o 1.s  

(timeout = 300)

spawn bfin-uclinux-g++ -g -O2 -D_GLIBCXX_ASSERT -fmessage-length=0

-DLOCALEDIR="/home/test/work/cruise/temp/bfin-uclinux/bin/../share/locale"

-I/home/test/

work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite

/home/test/work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite/23_containers/dequ

e/explicit_instantiation/1.cc -S -mstack-check-l1 -o 1.s^M

PASS: 23_containers/deque/explicit_instantiation/1.cc (test for excess errors)

extra_tool_flags are:

 

Executing on host: bfin-uclinux-g++ -g -O2 -D_GLIBCXX_ASSERT -fmessage-length=0

-DLOCALEDIR="/home/test/work/cruise/temp/bfin-uclinux/bin/../share/locale"

-I/home/test/work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite

/home/test/work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite/23_co

ntainers/deque/explicit_instantiation/2.cc     -S  -mstack-check-l1   -o 2.s  

(timeout = 300)

spawn bfin-uclinux-g++ -g -O2 -D_GLIBCXX_ASSERT -fmessage-length=0

-DLOCALEDIR="/home/test/work/cruise/temp/bfin-uclinux/bin/../share/locale"

-I/home/test/

work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite

/home/test/work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite/23_containers/dequ

e/explicit_instantiation/2.cc -S -mstack-check-l1 -o 2.s^M

PASS: 23_containers/deque/explicit_instantiation/2.cc (test for excess errors)

extra_tool_flags are:

 

Executing on host: bfin-uclinux-g++ -g -O2 -D_GLIBCXX_ASSERT -fmessage-length=0

-DLOCALEDIR="/home/test/work/cruise/temp/bfin-uclinux/bin/../share/locale"

-I/home/test/work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite

/home/test/work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite/23_co

ntainers/deque/explicit_instantiation/3.cc     -S  -mstack-check-l1   -o 3.s  

(timeout = 300)

spawn bfin-uclinux-g++ -g -O2 -D_GLIBCXX_ASSERT -fmessage-length=0

-DLOCALEDIR="/home/test/work/cruise/temp/bfin-uclinux/bin/../share/locale"

-I/home/test/

work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite

/home/test/work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite/23_containers/dequ

e/explicit_instantiation/3.cc -S -mstack-check-l1 -o 3.s^M

PASS: 23_containers/deque/explicit_instantiation/3.cc (test for excess errors)

extra_tool_flags are:

 

Executing on host: bfin-uclinux-g++ -g -O2 -D_GLIBCXX_ASSERT -fmessage-length=0

-DLOCALEDIR="/home/test/work/cruise/temp/bfin-uclinux/bin/../share/locale"

-I/home/test/work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite

testsuite_abi.o testsuite_allocator.o testsuite_character.o testsuite_hooks.o /

home/test/work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite/23_containers/deque/invalidation/1.cc

     -mstack-check-l1  -Wl,-elf2flt=-s80000

-lm   -o ./1.exe    (timeout = 300)

spawn bfin-uclinux-g++ -g -O2 -D_GLIBCXX_ASSERT -fmessage-length=0

-DLOCALEDIR="/home/test/work/cruise/temp/bfin-uclinux/bin/../share/locale"

-I/home/test/

work/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite testsuite_abi.o

testsuite_allocator.o testsuite_character.o testsuite_hooks.o /home/test/wor

k/cruise/checkouts/toolchain/gcc-4.1/libstdc++-v3/testsuite/23_containers/deque/invalidation/1.cc

-mstack-check-l1 -Wl,-elf2flt=-s80000 -lm -o ./1.exe^M

PASS: 23_containers/deque/invalidation/1.cc (test for excess errors)

Download of ./1.exe to bfin-uclinux failed.

UNRESOLVED: 23_containers/deque/invalidation/1.cc execution test

extra_tool_flags are:

 

 

--- Michael Hennerich                                        2009-02-18 06:44:43

> It looks this problem is not specific to Bf548, it is also found on bf527

ezkit board.

 

I’ve been watching this bug a few days now.

Friday last week I setup a BF548-EZKIT as well as a BF527-EZKIT running the

entire weekend.

- Still responding to telnet on Monday morning.

 

On my end I’ve not seen such a bug since months.

 

I wonder if it is related to using crossover cables in conjunction with your

test Host machine.

I could imagine the Ethernet card enters some kind of power saving mode if

there is silence for more than 1h and fails to wakeup?

 

Or that the Ethernet outages are related to some broken testcases you’re

running overwriting kernel text or data?

 

Nevertheless I setup two boards running this simple script trying to simulate

your toolchain tests.

 

#!/bin/bash

c=1800;

echo Crash Test $c

while :

do

while [ $c -ne 0 ]

do

  echo Crash Test $c

  rcp /home/michael/devel/svn/uclinux-dist-trunk_3/romfs/bin/version

root@10.44.2.45:/bin/

  rsh -l root 10.44.2.45 'date;md5sum /root/version;/root/version'

  sleep 1;

  c=`expr $c - 1`

done

sleep 3700

done

 

If it is still running tomorrow – I’ll do the same test with an X-over

cable to see if there are any differences.

 

-Michael

 

--- Michael Hennerich                                        2009-02-19 03:56:34

Both platforms running over night – no Network outages seen …

 

root:/> cat /proc/cpuinfo;uptime

processor       : 0

vendor_id       : Analog Devices

cpu family      : 0x27de

model name      : ADSP-BF548 525(MHz CCLK) 131(MHz SCLK) (mpu off)

stepping        : 1

cpu MHz         : 525.000/131.250000

bogomips        : 1046.52

Calibration     : 523264000 loops

cache size      : 16 KB(L1 icache) 32 KB(L1 dcache-wb) 0 KB(L2 cache)

dbank-A/B       : cache/cache

icache setup    : 4 Sub-banks/4 Ways, 32 Lines/Way

dcache setup    : 2 Super-banks/4 Sub-banks/2 Ways, 64 Lines/Way

L2 SRAM         : 128KB

board name      : ADI BF548-EZKIT

board memory    : 65536 kB (0x00000000 -> 0x04000000)

kernel memory   : 63480 kB (0x00001000 -> 0x03dff000)

 

21:28:21 up 21:28, load average: 0.00, 0.00, 0.00

root:/>

 

 

root:/> cat /proc/cpuinfo;uptime

processor       : 0

vendor_id       : Analog Devices

cpu family      : 0x27e0

model name      : ADSP-BF527 525(MHz CCLK) 131(MHz SCLK) (mpu off)

stepping        : 0

cpu MHz         : 525.000/131.250000

bogomips        : 1046.52

Calibration     : 523264000 loops

cache size      : 16 KB(L1 icache) 32 KB(L1 dcache-wb) 0 KB(L2 cache)

dbank-A/B       : cache/cache

icache setup    : 4 Sub-banks/4 Ways, 32 Lines/Way

dcache setup    : 2 Super-banks/4 Sub-banks/2 Ways, 64 Lines/Way

board name      : ADI BF527-EZKIT

board memory    : 65536 kB (0x00000000 -> 0x04000000)

kernel memory   : 64504 kB (0x00001000 -> 0x03eff000)

 

22:12:11 up 20:04, load average: 0.45, 0.38, 0.26

root:/>

 

--- Michael Hennerich                                        2009-02-20 06:54:27

 

I run tests with a crossover cable, as well.

My little test script simulating your toolchain tests can run forever on my

side.

However I did notice that the first time booting I get u-boot network timeouts

(see 1.) with bad dropped packages. Once the board booted it will always boot

without any timeouts shown. (see 2.)

Once I reboot my Host machine I see the issue again. So this might be related

to the Auto-DIMX feature in my network card.    

 

1.)

smc911x: initializing                                                         

                                                                               

                                                                

smc911x: detected LAN9218 controller                                          

                                                                               

                                                                

smc911x: phy initialized                                                      

                                                                               

                                                                

smc911x: MAC 00:e0:22:fe:b8:44                                                

                                                                               

                                                                

TFTP from server 192.168.1.1; our IP address is 192.168.1.45                  

                                                                               

                                                                

Filename 'uImage'.                                                            

                                                                               

                                                                

Load address: 0x1000000                                                       

                                                                               

                                                                

Loading: smc911x: dropped bad packet. Status: 0x04e88026                      

                                                                               

                                                                

smc911x: dropped bad packet. Status: 0x05488022                               

                                                                               

                                                                

T smc911x: dropped bad packet. Status: 0x05a88022

T T smc911x: dropped bad packet. Status: 0x01248026                           

                                                                               

                                                                

T T T T smc911x: dropped bad packet. Status: 0x01848026   

T #################################################################           

                                                                               

                                                                

         #################################################################    

                                                                               

                                                                

         #################################################################    

                                                                               

                                                                

         #################################################################    

                                                                               

                                                                

         #################################################################    

                                                                               

                                                                

         #######################                                              

                                                                               

                                                                

done 

 

 

2.)

smc911x: initializing                                                         

                                                                               

                                                                

smc911x: detected LAN9218 controller                                          

                                                                               

                                                                

smc911x: phy initialized                                                      

                                                                               

                                                                

smc911x: MAC 00:e0:22:fe:b8:44                                                

                                                                               

                                                                

TFTP from server 192.168.1.1; our IP address is 192.168.1.45                  

                                                                               

                                                                

Filename 'uImage'.                                                            

                                                                               

                                                                

Load address: 0x1000000                                                       

                                                                               

                                                                

Loading: #################################################################    

                                                                               

                                                                

         #################################################################    

                                                                               

                                                                

         #################################################################    

                                                                               

                                                                

         #################################################################    

                                                                               

                                                                

         #################################################################    

                                                                               

                                                                

         #######################                                              

                                                                               

                                                                

done

 

 

On my end I can't recreate whatever you are seeing.

Can you please try two things?

 

1) On you test machine connect the failing boards via an Ethernet switch.

2) If 1) doesn't help try to run a known good toolchain test. Maybe from our

last release? It might me that some broken test cases screwing up the kernel.

 

 

-Michael

 

--- Mingquan Pan                                             2009-02-23 22:40:18

There is something I missing, would you extend the sleep time to 4 hours and

have a try?

Because it is found that this issue mostly happens on the machines that I use

toolchain-regtest doing the test instead of the ones I use BuildToolChain. The

difference between the two scripts in testing that I can tell is that the

testing sequence is changed. In BuildToolChain, it is testing as long as

building the three toolchains while the toolchain-regtest do the whole testing

after the building is finished completely. So the waiting time in

toolchain-regtest is much longer.

 

 

--- Mingquan Pan                                             2009-02-25 04:51:40

Now it is frequently stuck at :

test     17479  0.0  3.1 260800 31844 pts/2    Sl   Feb16  11:50

/usr/lib/jvm/java-1.5.0-sun-1.5.0_update16/bin/java -cp

/usr/lib/jvm/java/lib/tools.jar:/h

test     17842  0.0  0.1   4524  1740 pts/2    S    Feb24   0:00 /bin/sh

./toolchain-regtest -s /home/test/work/cruise/checkouts/toolchain -o

/home/test/wo

root     23008  0.0  0.0      0     0 ?        S    Feb24   0:03 [pdflush]

test     25687  0.0  0.0   4524   792 pts/2    S    Feb24   0:00 /bin/sh

./toolchain-regtest -s /home/test/work/cruise/checkouts/toolchain -o

/home/test/wo

root     25688  0.0  0.0   2092   732 pts/2    S    Feb24   0:00 rsh -l root

10.100.4.50 ls /  1>/dev/null

test     25689  0.0  0.0   2896   576 pts/2    S    Feb24   0:00 wc -l

test     28197  0.0  0.2  61072  3052 ?        S    Feb19   0:04

/usr/bin/krunner_lock

test     28542  0.0  0.3  53260  3180 ?        S    Feb19   0:00 kblankscrn.kss

-root

root     31261  0.0  0.0      0     0 ?        S    Feb24   0:06 [pdflush]

test     31764  0.0  0.0   2056   840 pts/2    S+   12:33   0:00 telnet

10.100.4.50

 

In this case, the stuck is related to the failure in gdb hardware testing which

has case make kernel reboot. But it looks that it should be happening before the

kernel reboot. Because if it is in u-boot or in the middle of booting up, rsh

should get : No route to host instead of being stuck.

 

 

--- Michael Hennerich                                        2009-02-25 05:12:47

>rsh should get : No route to host instead of being stuck.

 

Well - I don't see this.

my rsh call gets stuck in case I reset and boot into u-boot.

 

Do you still see this only with BF527-EZKIT and BF548-EZKIT?

 

-Michael

 

--- Mingquan Pan                                             2009-02-25 23:33:23

I see the rsh stuck also on bf537 stamp and bf561 ezkit during gdb testing.

 

And there is time kernel stuck happening during fdpic gfortran testing, which

is after gdb testing on bf561 ezkit.And in this testing, after gdb testing,that

is,before the fdpic gcc(including gfortran) testing, the kernel is alive with

good rsh action but having page allocation failure seeing from the

reset_board_log.

 

When stuck, from the console it prints:

 

BusyBJxuv1.13.2 (2009-02-25 16:04:30 CST) built-in shell (msh)mp to NULL

address

Deferred Exception context

CURRENT PROCESS:

COMM=rshd PID=22752

CPU = 0

TEXT = 0x00000000-0x00000000        DATA = 0x00000000-0x00000000

BSS = 0x00000000-0x00000000  USER-STACK = 0x00000000

 

return address: [0x00000000]; contents of:

 

SEQUENCER STATUS:               Not tainted

SEQSTAT: 0006202d  IPEND: 0030  SYSCFG: 0006

  EXCAUSE   : 0x2d

  interrupts disabled

  physical IVG5 asserted : <0xffa00be4> { _evt_ivhw + 0x0 }

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x01c42000> /* kernel dynamic memory */

RETX: <0x00000480> /* Maybe fixed code section */

RETS: <0x00000000> /* Maybe null pointer? */

PC  : <0x00000000> /* Maybe null pointer? */

DCPLB_FAULT_ADDR: <0x01898d78> [ inetd + 0x8d78 ]

ICPLB_FAULT_ADDR: <0x00000000> /* Maybe null pointer? */

 

PROCESSOR STATE:

R0 : ffffffff    R1 : 0078e178    R2 : 0189775c    R3 : 0000002f

R4 : 00000004    R5 : 00000006    R6 : 01897c04    R7 : 00000000

P0 : 0000000b    P1 : 01897c04    P2 : 01897c18    P3 : 0078e178

P4 : 0078e1cd    P5 : 01898f00    FP : 00000000    SP : 01c41f24

LB0: 0189278d    LT0: 0189278a    LC0: 00000000

LB1: 01891de9    LT1: 01891de8    LC1: 00000000

B0 : 00000000    L0 : 00000000    M0 : 00000000    I0 : 018975cb

B1 : 00000000    L1 : 00000000    M1 : 00000000    I1 : 01898f00

B2 : 00000000    L2 : 00000000    M2 : 00000000    I2 : 00000000

B3 : 00000000    L3 : 00000000    M3 : 00000000    I3 : 00000000

A0.w: 00000000   A0.x: 00000000   A1.w: 00000000   A1.x: 00000000

USP : 01898d7c  ASTAT: 02002000

 

Hardware Trace:

   0 Target : <0x00004a58> { _trap_c + 0x0 }

     Source : <0xffa0066a> { _exception_to_level5 + 0x9e } CALL pcrel

   1 Target : <0xffa005cc> { _exception_to_level5 + 0x0 }

     Source : <0xffa0049a> { _bfin_return_from_exception + 0xe } RTX

   2 Target : <0xffa0048c> { _bfin_return_from_exception + 0x0 }

     Source : <0xffa00522> { _ex_trap_c + 0x66 } JUMP.S

   3 Target : <0xffa004bc> { _ex_trap_c + 0x0 }

     Source : <0xffa00704> { _trap + 0x38 } JUMP (P4)

   4 Target : <0xffa006ea> { _trap + 0x1e }

     Source : <0xffa006e6> { _trap + 0x1a } IF !CC JUMP

   5 Target : <0xffa006cc> { _trap + 0x0 }

     Source : <0x01893e6c> [ inetd + 0x3e6c ] RTS

   6 Target : <0x01893e66> [ inetd + 0x3e66 ]

     Source : <0x01893e7a> [ inetd + 0x3e7a ] JUMP.S

   7 Target : <0x01893e72> [ inetd + 0x3e72 ]

     Source : <0x018911a4> [ inetd + 0x11a4 ] RTS

   8 Target : <0x01891194> [ inetd + 0x1194 ]

     Source : <0x01893e6e> [ inetd + 0x3e6e ] CALL pcrel

   9 Target : <0x01893e6e> [ inetd + 0x3e6e ]

     Source : <0x01893e62> [ inetd + 0x3e62 ] IF !CC JUMP

  10 Target : <0x01893e5a> [ inetd + 0x3e5a ]

     Source : <0xffa00be0> { __common_int_entry + 0xd8 } RTI

  11 Target : <0xffa00b7e> { __common_int_entry + 0x76 }

     Source : <0xffa00da4> { _evt_system_call + 0x64 } JUMP.S

  12 Target : <0xffa00da4> { _evt_system_call + 0x64 }

     Source : <0xffa0088c> { _system_call + 0xb8 } RTS

  13 Target : <0xffa00888> { _system_call + 0xb4 }

     Source : <0xffa00878> { _system_call + 0xa4 } IF !CC JUMP

  14 Target : <0xffa00872> { _system_call + 0x9e }

     Source : <0xffa00862> { _system_call + 0x8e } IF !CC JUMP

  15 Target : <0xffa00850> { _system_call + 0x7c }

     Source : <0xffa00870> { _system_call + 0x9c } JUMP.S

Userspace Stack

Stack info:

SP: [0x01898d7c] <0x01898d7c> [ inetd + 0x8d7c ]

FP: (0x01898d9c)

Memory from 0x01898d70 to 01899000

01898d70: 00000000  00000000  00000000 [00000000] 00000000  00000000  0189278a

01891de8

01898d90: 0189278d  01891de9  00062000 (01898db0)<018910d6> 00000400

01898dc4 <0189118c>

01898db0:(01898de0)<01891120> 00000006  00000004  00000000  01898ddc

<018904a4> 00000000

01898dd0: 01898dd8  00000000  00000001  00000000 (01898f2c)<01890b24>

0078e14c  00000007

01898df0: 00000000  00000000  01898f20  01898f00  00000000  00000000  00000000

00000000

01898e10: 00000000  00000000  00000000  00000000  00000000  00000000  00000000

00000000

01898e30: 00000000  00000000  00000000  00000000  00000000  00000000  00000000

00000000

01898e50: 00000000  00000000  00000000  00000000  00000000  00000000  00000000

00000000

01898e70: 00000000  00000000  00000000  00000000  00000070  00000000  00000000

00000000

01898e90: 00000000  00000000  00000000  00000000  00000000  00000000  00000000

00000000

01898eb0: 00000000  00000000  00000000  00000000  00000000  00000000  00000000

00000000

01898ed0: 00000000  00000000  00000000  00000000  00000000  00000000  00000000

00000000

01898ef0: 00000000  00000000  00000000  00000000  fe030002  ae04640a  00000000

00000000

01898f10: 17000002  00000000  00000000  00000000  00000001  00000000  00000010

(01898f5c)

01898f30:<01893e44> 01896084  01897544  01898fa0  00000000  01897c04

01896048  00000001

01898f50: 00000000  00000000  00000000 (00000000)<003c0ab8> 01896084

003d8020  003d8024

01898f70: 00000000  459865ba  ffffffff  00000000 <018908e8> 00000000

00000000  01896048

01898f90: 0189605a  00000000  003e8ad2  00000001  01898fb4  00000000

<01898fc0> 01898fcd

01898fb0: 00000000  6962732f  6e692f6e  00647465  4d524554  6b6e753d  6e776f6e

54415000

01898fd0: 622f3d48  2f3a6e69  2f727375  3a6e6962  6962732f  752f3a6e  732f7273

006e6962

01898ff0: 6962732f  6e692f6e  00647465  00000000  0c443220

Return addresses in stack:

   frame  1 : <0x018910d6> [ inetd + 0x10d6 ]

    address : <0x0189118c> [ inetd + 0x118c ]

   frame  2 : <0x01891120> [ inetd + 0x1120 ]

    address : <0x018904a4> [ inetd + 0x4a4 ]

   frame  3 : <0x01890b24> [ inetd + 0xb24 ]

   frame  4 : <0x01893e44> [ inetd + 0x3e44 ]

   frame  5 : <0x003c0ab8> [ init + 0xab8 ]

    address : <0x018908e8> [ inetd + 0x8e8 ]

    address : <0x01898fc0> [ inetd + 0x8fc0 ]

 

The whole info printed is attached.

 

At this point , on host:

 

root     31702  0.0  0.3   4052  1836 pts/4    S+   Feb25   0:00 bash

test     31731  0.0  0.6   8504  3344 pts/6    Ss+  Feb25   0:00 kermit

test     31783  0.0  0.3   4100  1744 pts/5    S    Feb25   0:01 /bin/sh

./BuildToolChain -t -T 10.100.4.50 -j 1 -w -s

/home/test/work/cruise/checkouts/too

root     21678  0.0  0.0      0     0 ?        S    Feb25   0:05 [pdflush]

test     10017  0.0  0.6   7392  3304 ?        Ss   Feb25   0:00

bfin-uclinux-gdb -nw -nx

test      1364  0.0  0.2   3832  1220 pts/5    S    01:59   0:00 sh

test      1365  0.0  0.5   4900  2984 pts/5    S    01:59   0:00 make -k -j 1

check

test      5246  0.0  0.2   3576  1192 pts/5    S    03:16   0:00 /bin/sh -c

(rootme=`${PWDCMD-pwd}`; export rootme;  srcdir=`cd /home/test/work/cruise/chec

test      5247  0.0  0.2   3836  1272 pts/5    S    03:16   0:00 /bin/sh -c

(rootme=`${PWDCMD-pwd}`; export rootme;  srcdir=`cd /home/test/work/cruise/chec

test      5256  0.0  0.9   5824  4588 pts/5    S    03:16   0:05 expect --

/usr/share/dejagnu/runtest.exp --tool gfortran

root     19357  0.0  0.1   1936   580 pts/5    S    11:51   0:00 rcp

./assign.exe root 10.100.4.50 /tmp/assign.exe.5256

test     19358  0.0  0.0   1468   356 pts/5    S    11:51   0:00 cat

root     19415  0.0  0.4   8144  2232 ?        Ss   11:53   0:00 sshd: test

[priv]

test     19420  0.0  0.4   8144  2340 ?        S    11:54   0:00 sshd:

test@pts/10

test     19421  0.0  0.3   4312  1904 pts/10   Ss+  11:54   0:00 -bash

test     19586  0.0  0.3   4100  1748 pts/5    S    12:03   0:00 /bin/sh

./BuildToolChain -t -T 10.100.4.50 -j 1 -w -s

/home/test/work/cruise/checkouts/too

root     19587  0.0  0.1   1924   568 pts/5    S    12:03   0:00 rsh -l root

10.100.4.50 ps

test     19588  0.0  0.1   2696   692 pts/5    S    12:03   0:00 grep -v  ps

test     19589  0.0  0.1   2696   692 pts/5    S    12:03   0:00 grep -v /rshd

test     19590  0.0  0.1   2696   692 pts/5    S    12:03   0:00 grep -v \[

test     19591  0.0  0.1   2696   692 pts/5    S    12:03   0:00 grep -v

telnetd

test     19592  0.0  0.1   2696   692 pts/5    S    12:03   0:00 grep -v /init

test     19593  0.0  0.1   2696   692 pts/5    S    12:03   0:00 grep -v

inetd

test     19594  0.0  0.1   2696   692 pts/5    S    12:03   0:00 grep -v  PID

test     19595  0.0  0.1   2696   696 pts/5    S    12:03   0:00 grep -v [

/]sh

test     19596  0.0  0.1   2696   692 pts/5    S    12:03   0:00 grep -v

/s*bin/.*d

test     19597  0.0  0.1   2692   672 pts/5    S    12:03   0:00 sed

/^[[:space:]]*$/d

postfix  19941  0.0  0.2   4740  1388 ?        S    12:27   0:00 pickup -l -t

fifo -u

test     20102  0.0  0.1   2692   812 pts/5    R+   12:38   0:00 ps aux

test@43-bf561-toolchain:~/work/cruise/test_scripts/toolchain/toolchain-build>

ping 10.100.4.50

PING 10.100.4.50 (10.100.4.50) 56(84) bytes of data.

64 bytes from 10.100.4.50: icmp_seq=1 ttl=64 time=0.160 ms

64 bytes from 10.100.4.50: icmp_seq=2 ttl=64 time=0.174 ms

64 bytes from 10.100.4.50: icmp_seq=3 ttl=64 time=0.161 ms

 

--- 10.100.4.50 ping statistics ---

3 packets transmitted, 3 received, 0% packet loss, time 2001ms

rtt min/avg/max/mdev = 0.160/0.165/0.174/0.006 ms

test@43-bf561-toolchain:~/work/cruise/test_scripts/toolchain/toolchain-build>

telnet 10.100.4.50

Trying 10.100.4.50...

Connected to 10.100.4.50.

Escape character is '^]'.

                

ping can be through, telnet fails.

 

--- Michael Hennerich                                        2009-02-26 11:08:34

Grace,

 

This is definitely not a networking issue!!

 

The log attached includes the entire repertoire of possible Blackfin

exceptions.

Followed by 00-Memory dumps coming from rcpd, rshd, inted, sh etc.

 

So you see issues that only look like networking bugs on your host machine,

because the rshd, rcpd, and inetd running on Blackfin can’t allocate memory to

handle incoming requests.

 

The failing toolchain test cases are the problem in the first place.

They may allocate memory that isn’t freed after the testcase crashes.

 

So PLEASE assign this bug to whoever is responsible for the toolchain test

cases.

 

-Michael

 

--- Robin Getz                                               2009-02-26 13:06:39

Michael:

 

You said:

>They may allocate memory that isn’t freed after the testcase crashes.

 

that is still a kernel problem - the kernel needs to clean up/free all the

allocated resources when an application quits.

 

-Robin

 

--- Sonic Zhang                                              2009-02-26 23:07:18

It is also possible that the memory fragment issue is the root cause. Assign to

Grace till the toolchain test suite issue is solved.

 

--- Mingquan Pan                                             2010-01-13 04:02:27

This is an old issue that some case make kernel panic and thus stuck the

following cases. We now can log the output on target console and find the failed

case to fix. As to the memory allocate issue, it is something like the issue in

bug [5454], which is solved. Since the testing on trunk head is ok now and not

related to network, so close this bug.

 

 

 

    Files

    Changes

    Commits

    Dependencies

    Duplicates

    Associations

    Tags

 

File Name     File Type     File Size     Posted By

gfortran.log.gz    application/x-gzip    7735    Mingquan Pan

reset_board_log.gz    application/x-gzip    247979    Mingquan Pan

Attachments

Outcomes