[#4623] kernel oops featuring smsc911x_irqhandler when using NFS root

Submitted By: Colin McCabe

Open Date

2008-11-07 18:07:08 Close Date

2008-11-20 08:41:40

Priority:

Medium Assignee:

Nobody

Status:

Closed Fixed In Release:

N/A

Found In Release:

N/A Release:

Category:

Drivers Board:

EZKIT Lite

Processor:

BF548 Silicon Revision:

Is this bug repeatable?:

Yes Resolution:

Duplicate

Uboot version or rev.:

U-Boot 1.1.6-svn1273 Toolchain version or rev.:

uClinux-dist-2008R1.5-RC3

App binary format:

N/A

Summary: kernel oops featuring smsc911x_irqhandler when using NFS root

Details:

Soon after bringing up my BF548 EZ-Kit Lite board, I received a kernel oops which seems to be connected to the smsc911x ethernet driver.

The kernel is at change 73c6d0ac88830de107cb4f468b887e6769f1fa99

(no modifications on my part)

uname -a gives:

Linux blackfin 2.6.28-rc2-ADI-2009R1-pre #2 Fri Nov 7 12:45:24 PST 2008 blackfin

This has happened twice so far...

The oops:

=======================================================

root:/> NULL pointer access

Kernel OOPS in progress

Deferred Exception context

No Valid process in current context

return address: [0x001ac538]; contents of:

0x001ac510: 0c45 5079 030c 1002 5041 5059 030c 1002

0x001ac520: 5041 5051 030c 1002 5041 0000 6040 b1f1

0x001ac530: b230 184a 0000 0000 [e4a8] 015f 4928 0200

0x001ac540: e14d 0029 e10d 33a4 a171 3056 b1b0 6482

SEQUENCER STATUS: Not tainted

SEQSTAT: 00062027 IPEND: c030 SYSCFG: 0006

EXCAUSE : 0x27

physical IVG14 asserted : <0xffa00a0c> { _evt14_softirq + 0x0 }

physical IVG15 asserted : <0xffa00d58> { _evt_system_call + 0x0 }

logical irq 6 mapped : <0xffa0036c> { _timer_interrupt + 0x0 }

logical irq 9 mapped : <0x000edc3c> { _bfin_bf54x_irq_error + 0x0 }

logical irq 14 mapped : <0x00154020> { _bfin_rtc_interrupt + 0x0 }

logical irq 48 mapped : <0x00102bfc> { _bfin_serial_dma_rx_int + 0x0 }

logical irq 49 mapped : <0x00102990> { _bfin_serial_dma_tx_int + 0x0 }

logical irq 52 mapped : <0x00156ecc> { _bfin_twi_interrupt_entry + 0x0 }

logical irq 53 mapped : <0x00156ecc> { _bfin_twi_interrupt_entry + 0x0 }

logical irq 68 mapped : <0x00129bc4> { _bfin_ata_interrupt + 0x0 }

logical irq 74 mapped : <0x0013c058> { _bf5xx_nand_dma_irq + 0x0 }

logical irq 76 mapped : <0x001525f8> { _bfin_kpad_isr + 0x0 }

logical irq 82 mapped : <0x0014d39c> { _blackfin_interrupt + 0x0 }

logical irq 85 mapped : <0x0014fd18> { _dma_controller_irq + 0x0 }

logical irq 175 mapped : <0x0010ad10> { _smsc911x_irqhandler + 0x0 }

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x002b3ce4> /* kernel dynamic memory */

RETX: <0x00000480> /* Maybe fixed code section */

RETS: <0x001ae618> { _tcp_v4_rcv + 0x254 }

PC : <0x001ac538> { _tcp_v4_send_reset + 0xc4 }

DCPLB_FAULT_ADDR: <0x0000015f> /* Maybe null pointer? */

ICPLB_FAULT_ADDR: <0x0000ffff> { _daemonize + 0xc3 }

PROCESSOR STATE:

R0 : 00000008 R1 : 25015780 R2 : 14000600 R3 : 0f00a8c0

R4 : 00000000 R5 : 03a48c10 R6 : 00000002 R7 : 0200a8c0

P0 : 03a97a00 P1 : 03a48c24 P2 : 03a48c10 P3 : 0f00a8c0

P4 : 03a48c24 P5 : 00000000 FP : 00000000 SP : 002b3c08

LB0: 000dee50 LT0: 000dee50 LC0: 00000000

LB1: 000395be LT1: 000395b8 LC1: 00000000

B0 : 000005a0 L0 : 00000000 M0 : 00000000 I0 : 0f00a8c0

B1 : 00000000 L1 : 00000000 M1 : 00000000 I1 : 0200a8c0

B2 : 00000000 L2 : 00000000 M2 : 00000000 I2 : 00000000

B3 : 00000000 L3 : 00000000 M3 : 00000000 I3 : 00000000

A0.w: 00000000 A0.x: 00000000 A1.w: 00000000 A1.x: 00000000

USP : 002b4000 ASTAT: 02002000

Hardware Trace:

0 Target : <0x0000499c> { _trap_c + 0x0 }

Source : <0xffa006a8> { _exception_to_level5 + 0x9c } CALL pcrel

1 Target : <0xffa0060c> { _exception_to_level5 + 0x0 }

Source : <0xffa0049a> { _bfin_return_from_exception + 0x6 } RTX

2 Target : <0xffa00494> { _bfin_return_from_exception + 0x0 }

Source : <0xffa00564> { _ex_trap_c + 0x6c } 0x0046

3 Target : <0xffa004f8> { _ex_trap_c + 0x0 }

Source : <0xffa00730> { _trap + 0x28 } JUMP (P4)

4 Target : <0xffa00708> { _trap + 0x0 }

Source : <0x001ac536> { _tcp_v4_send_reset + 0xc2 } 0x0000

5 Target : <0x001ac52a> { _tcp_v4_send_reset + 0xb6 }

Source : <0x001ac526> { _tcp_v4_send_reset + 0xb2 } IF !CC JUMP

6 Target : <0x001ac522> { _tcp_v4_send_reset + 0xae }

Source : <0x001ac51e> { _tcp_v4_send_reset + 0xaa } IF !CC JUMP

7 Target : <0x001ac51a> { _tcp_v4_send_reset + 0xa6 }

Source : <0x001ac516> { _tcp_v4_send_reset + 0xa2 } IF !CC JUMP

8 Target : <0x001ac4a2> { _tcp_v4_send_reset + 0x2e }

Source : <0x001ac498> { _tcp_v4_send_reset + 0x24 } IF !CC JUMP

9 Target : <0x001ac474> { _tcp_v4_send_reset + 0x0 }

Source : <0x001ae614> { _tcp_v4_rcv + 0x250 } CALL pcrel

10 Target : <0x001ae610> { _tcp_v4_rcv + 0x24c }

Source : <0x001ae602> { _tcp_v4_rcv + 0x23e } IF CC JUMP

11 Target : <0x001ae5ec> { _tcp_v4_rcv + 0x228 }

Source : <0x001ae5dc> { _tcp_v4_rcv + 0x218 } IF !CC JUMP

12 Target : <0x001ae5d0> { _tcp_v4_rcv + 0x20c }

Source : <0x001ae822> { _tcp_v4_rcv + 0x45e } JUMP.S

13 Target : <0x001ae814> { _tcp_v4_rcv + 0x450 }

Source : <0x0019e1f2> { ___inet_lookup_listener + 0xaa } RTS

14 Target : <0x0019e1ec> { ___inet_lookup_listener + 0xa4 }

Source : <0x0019e160> { ___inet_lookup_listener + 0x18 } IF CC JUMP

15 Target : <0x0019e148> { ___inet_lookup_listener + 0x0 }

Source : <0x001ae810> { _tcp_v4_rcv + 0x44c } CALL pcrel

Kernel Stack

Stack info:

SP: [0x002b3f58] <0x002b3f58> /* kernel dynamic memory */

Memory from 0x002b3f50 to 002b4000

002b3f50: 00000000 00000000 [00000000] 00000000 00000000 00000020 002b4000 0029a580

002b3f70: ffa00284 002b2000 0027e6a4 0029a580 002b2000 ffa0092c 0027de04 00000000

002b3f90: 00000000 00000000 00000065 0000001f 00000002 0000ffff 0000ffff ffa0092c

002b3fb0: 00000006 00000000 00000000 00000000 002b56c6 0028a1f0 002d8a14 0027de10

002b3fd0: 00000000 002d8a14 0027d6e8 0000005a 002b50a8 002daa60 002b4000 002ca6be

002b3ff0: 00000000 00000000 00000000 ffb00000 00000000

Return addresses in stack:

Modules linked in:

Kernel panic - not syncing: Kernel exception

=======================================================

.config is attached.

Follow-ups

--- Colin McCabe 2008-11-11 15:59:23

Just a note: we have reproduced this on two EZ-KIT Lite BF548 boards now.

--- Sonic Zhang 2008-11-12 02:22:33

You should built your svn trunk kernel with svn trunk toolchain and boot it by

svn trunk uboot as well.

--- Sonic Zhang 2008-11-12 23:12:10

Can't replicate.

--- Mike Frysinger 2008-11-12 23:21:26

which means "not reproducible", not "rejected"

Colin: can you describe exactly what's going on in the system ? we use the

BF548-EZKIT and networking quite a lot and have yet to see this. so knowing

exactly how you're using the system would help a lot.

--- Colin McCabe 2008-11-13 14:16:52

Hi Mike,

Thanks for responding.

Our root filesystem is mounted over NFS. Have you ever used that

configuration?

Also...we are using

git://sources.blackfin.uclinux.org/git/readonly-mirrors/linux-kernel.git rather

than svn://sources.blackfin.uclinux.org/linux-kernel/trunk linux-kernel. I

believe that the git tree is supposed to be identical to what is in subversion.

However, since switching to using subversion, I haven't seen the problem.

Colin

--- Colin McCabe 2008-11-13 14:21:27

And thanks to Sonic as well. Your idea about svn was a good one, it seems!

--- Colin McCabe 2008-11-13 14:25:04

Update: I spoke too soon.

After several minutes, I got the same oops on the subversion kernel.

root:/> uname -r

2.6.28-rc2-ADI-2009R1-pre-svn5698

root:/> NULL pointer access

Kernel OOPS in progress

Deferred Exception context

--- Mike Frysinger 2008-11-13 15:37:02

git is an exact mirror of svn (just delayed ~1 hour). any behavior changes are

most likely coincidence rather than anything else.

we have tested NFS root before, but not necessarily with the BF548-EZKIT. what

are your exact mount options and what are you doing on the board ? just sitting

idle ?

--- Colin McCabe 2008-11-13 15:46:32

U-Boot settings:

bfin> printenv bootargs

bootargs=root=/dev/nfs rw

nfsroot=192.168.0.2:/opt/target/blackfin-uClinux-dist-2008R1.5-RC3/romfs/,tcp,nfsvers=3

ip=192.168.0.15:192.168.0.2:192.168.0.1:255.255.255.0:1:eth0:off

earlyprintk=serial,uart1,57600 console=tty0 console=ttyBF0,57600

I boot by doing:

bfin> tftp 0x1000000 vmImage

bfin> bootm

And the kernel .config is attached.

The oops doesn't seem to be triggered by any specific behavior. Sometimes it

will happen immediately after I see the motd. Other times the board will be up

for 5 minutes before it goes down.

Colin

--- Mike Frysinger 2008-11-13 15:57:46

sorry, could you also describe the NFS server ? what software versions

(nfs-utils / kernel / etc...) and what is the /etc/exports line for it ?

--- Colin McCabe 2008-11-13 17:17:59

I'm running nfs-kernel-server as the NFS server.

The board is connected via a crossover cable to my laptop's eth1 interface.

The problem has been reproduced with the board connected to other ethernet

interfaces, though.

/etc/exports has /opt

192.168.0.15(rw,no_root_squash,async,subtree_check,insecure)

Some version / configuration information for the NFS server:

============================================================

colin@calvin:~$ uname -r

2.6.24-16-generic

============================================================

colin@calvin:~$ cat /etc/issue

Ubuntu 8.04.1 \n \l

============================================================

colin@calvin:~$ cat /etc/default/nfs-kernel-server

# Number of servers to start up

RPCNFSDCOUNT=8

# Runtime priority of server (see nice(1))

RPCNFSDPRIORITY=0

# Options for rpc.mountd.

# If you have a port-based firewall, you might want to set up

# a fixed port here using the --port option. For more information,

# see rpc.mountd(8) or wiki.debian.org/?SecuringNFS

RPCMOUNTDOPTS=

# Do you want to start the svcgssd daemon? It is only required for Kerberos

# exports. Valid alternatives are "yes" and "no"; the

default is "no".

NEED_SVCGSSD=

# Options for rpc.svcgssd.

RPCSVCGSSDOPTS=

============================================================

colin@calvin:~$ /sbin/portmap -V

portmap version 6.0 - 2007-May-11

============================================================

colin@calvin:~$ dpkg -p nfs-kernel-server

Package: nfs-kernel-server

Priority: optional

Section: net

Installed-Size: 364

Maintainer: Ubuntu Core Developers

<ubuntu-devel-discuss@lists.ubuntu.com>

Architecture: i386

Source: nfs-utils

Version: 1:1.1.2-2ubuntu2.1

Replaces: knfs, nfs-server

Provides: knfs, nfs-server

Depends: libblkid1 (>= 1.39+1.40-WIP-2006.11.14+dfsg-2), libc6 (>= 2.4),

libcomerr2 (>= 1.33-3), libgssglue1, libkrb53 (>= 1.6.dfsg.2),

libldap-2.4-2 (>= 2.4.7), libnfsidmap2, librpcsecgss3, libwrap0, lsb-base

(>= 1.3-9ubuntu3), nfs-common (>= 1:1.0.8-1), ucf

Conflicts: knfs, nfs-server

Size: 150288

Description: support for NFS kernel server

The NFS kernel server is currently the recommended NFS server for use

with Linux, featuring features such as NFSv3 and NFSv4, Kerberos

support via GSS, and much more. It is also significantly faster and

usually more reliable than the user-space NFS servers (from the

unfs3 and nfs-user-server packages). However, it is more difficult to

debug than the user-space servers, and has a slightly different

feature set.

This package contains the user-space support needed to use the

NFS kernel server. Most administrators wishing to set up an NFS server

would want to install this package.

Upstream: SourceForge project "nfs", CVS module nfs-utils.

Homepage: nfs.sourceforge.net/

Original-Maintainer: Anibal Monsalve Salazar <anibal@debian.org>

--- Colin McCabe 2008-11-13 17:59:34

I tried some different options in /etc/exports.

/opt 192.168.0.15(rw,no_root_squash,no_subtree_check)

After restarting the NFS server and the board, however, there was no change

from the behavior under the old /etc/exports.

--- Michael Hennerich 2008-11-14 05:15:30

I've seen this bug in _tcp_v4_send_reset happen on other platforms as well.

It was somehow introduced when we updated our trunk kernel to 2.6.28.

This is what I just got on BF537 with the EMAC.

I also think I saw that someone open a different bug for this as well.

-Michael

NULL pointer access q=2.6 size= 343kB time=8.4 bitrate= 333.1kbits/s

Kernel OOPS in progress

Deferred Exception context

CURRENT PROCESS:

COMM=ffmpeg PID=216

CPU = 0

TEXT = 0x008e0000-0x008f151c DATA = 0x00b4051c-0x00b42828

BSS = 0x00b42828-0x00880000 USER-STACK = 0x0089fe40

return address: [0x00211418]; contents of:

0x002113f0: 5079 030c 1002 5041 5059 030c 1002 5041

0x00211400: 5051 030c 1002 5041 0000 6040 b1f1 b230

0x00211410: 184b 0000 0000 0000 [e4a8] 015f 4928 0200

0x00211420: e14d 002c e10d ae08 a171 3056 b1b0 6482

SEQUENCER STATUS: Not tainted

SEQSTAT: 00060027 IPEND: 4030 SYSCFG: 0006

EXCAUSE : 0x27

physical IVG14 asserted : <0xffa00b40> { _evt14_softirq + 0x0 }

logical irq 6 mapped : <0xffa00460> { _timer_interrupt + 0x0 }

logical irq 10 mapped : <0x001d6760> { _bfin_rtc_interrupt + 0x0 }

logical irq 11 mapped : <0x001c5ea4> { _bcap_ppi_irq + 0x0 }

logical irq 16 mapped : <0x001d8e6c> { _bfin_twi_interrupt_entry +

0x0 }

logical irq 18 mapped : <0x001a29a0> { _bfin_serial_dma_rx_int + 0x0

}

logical irq 19 mapped : <0x001a2710> { _bfin_serial_dma_tx_int + 0x0

}

logical irq 24 mapped : <0x001ab3dc> { _bfin_mac_interrupt + 0x0 }

logical irq 42 mapped : <0x001c5990> { _bcap_ppi_irq_error + 0x0 }

logical irq 54 mapped : <0x001aea30> { _orinoco_interrupt + 0x0 }

RETE: <0x00000000> { _do_one_initcall + 0xfff00000 }

RETN: <0x00d13d34> /* kernel dynamic memory */

RETX: <0x00000480> /* Maybe fixed code section */

RETS: <0x002135aa> { _tcp_v4_rcv + 0x262 }

PC : <0x00211418> { _tcp_v4_send_reset + 0xcc }

DCPLB_FAULT_ADDR: <0x0000015f> /* Maybe null pointer? */

ICPLB_FAULT_ADDR: <0x00211418> { _tcp_v4_send_reset + 0xcc }

PROCESSOR STATE:

R0 : 00000008 R1 : 99045e14 R2 : 14000600 R3 : 2d022c0a

R4 : 00000000 R5 : 00a5e820 R6 : 00000002 R7 : 58022c0a

P0 : 037813c0 P1 : 00a5e834 P2 : 00a5e820 P3 : 2d022c0a

P4 : 00a5e834 P5 : 00000000 FP : 00000000 SP : 00d13c58

LB0: ffa0177c LT0: ffa0177c LC0: 00000000

LB1: 00107507 LT1: 00107506 LC1: 00000000

B0 : 00946240 L0 : 00000000 M0 : 0000000c I0 : 2d022c0a

B1 : ff8000c0 L1 : 00000000 M1 : 00000010 I1 : 58022c0a

B2 : ff8000c0 L2 : 00000000 M2 : 00000080 I2 : 009462c0

B3 : ff900100 L3 : 00000000 M3 : 000000ac I3 : ffffffff

A0.w: 00000000 A0.x: 00000000 A1.w: 00000000 A1.x: 00000000

USP : 00894444 ASTAT: 02002002

Hardware Trace:

0 Target : <0x001039bc> { _trap_c + 0x0 }

Source : <0xffa007ca> { _exception_to_level5 + 0x9e } CALL pcrel

1 Target : <0xffa0072c> { _exception_to_level5 + 0x0 }

Source : <0xffa005be> { _bfin_return_from_exception + 0xe } RTX

2 Target : <0xffa005b0> { _bfin_return_from_exception + 0x0 }

Source : <0xffa00682> { _ex_trap_c + 0x66 } JUMP.S

3 Target : <0xffa0061c> { _ex_trap_c + 0x0 }

Source : <0xffa0049a> { _ex_workaround_261 + 0x22 } JUMP.S

4 Target : <0xffa00478> { _ex_workaround_261 + 0x0 }

Source : <0xffa00864> { _trap + 0x38 } JUMP (P4)

5 Target : <0xffa0084a> { _trap + 0x1e }

Source : <0xffa00846> { _trap + 0x1a } IF !CC JUMP

6 Target : <0xffa0082c> { _trap + 0x0 }

Source : <0xffa005be> { _bfin_return_from_exception + 0xe } RTX

7 Target : <0xffa005b0> { _bfin_return_from_exception + 0x0 }

Source : <0xffa0048a> { _ex_workaround_261 + 0x12 } IF !CC JUMP

8 Target : <0xffa00478> { _ex_workaround_261 + 0x0 }

Source : <0xffa00864> { _trap + 0x38 } JUMP (P4)

9 Target : <0xffa0084a> { _trap + 0x1e }

Source : <0xffa00846> { _trap + 0x1a } IF !CC JUMP

10 Target : <0xffa0082c> { _trap + 0x0 }

Source : <0x00211416> { _tcp_v4_send_reset + 0xca } 0x0000

11 Target : <0x00211408> { _tcp_v4_send_reset + 0xbc }

Source : <0x00211404> { _tcp_v4_send_reset + 0xb8 } IF !CC JUMP

12 Target : <0x00211400> { _tcp_v4_send_reset + 0xb4 }

Source : <0x002113fc> { _tcp_v4_send_reset + 0xb0 } IF !CC JUMP

13 Target : <0x002113f8> { _tcp_v4_send_reset + 0xac }

Source : <0x002113f4> { _tcp_v4_send_reset + 0xa8 } IF !CC JUMP

14 Target : <0x002113c4> { _tcp_v4_send_reset + 0x78 }

Source : <0x002114a4> { _tcp_v4_send_reset + 0x158 } JUMP.S

15 Target : <0x00211458> { _tcp_v4_send_reset + 0x10c }

Source : <0x002113b8> { _tcp_v4_send_reset + 0x6c } IF !CC JUMP

Kernel Stack

Stack info:

SP: [0x00d13f24] <0x00d13f24> /* kernel dynamic memory */

Memory from 0x00d13f20 to 00d14000

00d13f20: 00000480 [00699418] ffffffff 00000000 00000000 00d14000 00d4d018

00699418

00d13f40:<0069933a> 0000000b 02001025 ffa023a5 0069941d ffa0233c

00699412 00000000

00d13f60: 00000033 00000000 00000000 00000000 00000000 ff900100 ff8000c0

ff8000c0

00d13f80: 00946240 00000000 00000000 00000000 00000000 000000ac 00000080

00000010

00d13fa0: 0000000c ffffffff 009462c0 009462c4 00950290 00894444 00946240

0069c216

00d13fc0: 00940010 00010001 0000003f 00950290 0094629a 00010001 00000000

00000001

00d13fe0: 00e0e5d0 00000001 08000800 ffffffff 00000000 00000000 0094629a

00000006

00d14000: 000028c8

Return addresses in stack:

address : <0x0069933a> [ /usr/lib/libavcodec.so.51 + 0x29933a ]

Modules linked in:

Kernel panic - not syncing: Kernel exception

--- Michael Hennerich 2008-11-14 05:18:47

Check out this Bf533 bug opened by Mike:

blackfin.uclinux.org/gf/project/uclinux-dist/tracker/?action=TrackerItemEdit&tracker_item_id=4593

Same thing:

10 Target : <0xffa00768> { _trap + 0x0 }

Source : <0x000d93c2> { _tcp_v4_send_reset + 0xc6 } 0x0000

-Michael

--- Michael Hennerich 2008-11-20 05:33:16

Same issue as:

[#4618] [ltp] ltp test case connect01 crashed in trunk

[#4593] random crash in tcp code

--- Mike Frysinger 2008-11-20 08:41:40

move to LTP bug then

Files

Changes

Commits

Dependencies

Duplicates

Associations