[#4623] kernel oops featuring smsc911x_irqhandler when using NFS root

Document created by Aaronwu Employee on Aug 29, 2013
Version 1Show Document
  • View in full screen mode

[#4623] kernel oops featuring smsc911x_irqhandler when using NFS root

Submitted By: Colin McCabe

Open Date

2008-11-07 18:07:08     Close Date

2008-11-20 08:41:40

Priority:

Medium     Assignee:

Nobody

Status:

Closed     Fixed In Release:

N/A

Found In Release:

N/A     Release:

Category:

Drivers     Board:

EZKIT Lite

Processor:

BF548     Silicon Revision:

Is this bug repeatable?:

Yes     Resolution:

Duplicate

Uboot version or rev.:

U-Boot 1.1.6-svn1273     Toolchain version or rev.:

uClinux-dist-2008R1.5-RC3

App binary format:

N/A     

Summary: kernel oops featuring smsc911x_irqhandler when using NFS root

Details:

 

Soon after bringing up my BF548 EZ-Kit Lite board, I received a kernel oops which seems to be connected to the smsc911x ethernet driver.

 

The kernel is at change 73c6d0ac88830de107cb4f468b887e6769f1fa99

(no modifications on my part)

 

uname -a gives:

Linux blackfin 2.6.28-rc2-ADI-2009R1-pre #2 Fri Nov 7 12:45:24 PST 2008 blackfin

 

This has happened twice so far...

 

The oops:

=======================================================

root:/> NULL pointer access

Kernel OOPS in progress

Deferred Exception context

 

No Valid process in current context

return address: [0x001ac538]; contents of:

0x001ac510:  0c45  5079  030c  1002  5041  5059  030c  1002

0x001ac520:  5041  5051  030c  1002  5041  0000  6040  b1f1

0x001ac530:  b230  184a  0000  0000 [e4a8] 015f  4928  0200

0x001ac540:  e14d  0029  e10d  33a4  a171  3056  b1b0  6482

 

SEQUENCER STATUS:        Not tainted

SEQSTAT: 00062027  IPEND: c030  SYSCFG: 0006

  EXCAUSE   : 0x27

  physical IVG14 asserted : <0xffa00a0c> { _evt14_softirq + 0x0 }

  physical IVG15 asserted : <0xffa00d58> { _evt_system_call + 0x0 }

  logical irq   6 mapped  : <0xffa0036c> { _timer_interrupt + 0x0 }

  logical irq   9 mapped  : <0x000edc3c> { _bfin_bf54x_irq_error + 0x0 }

  logical irq  14 mapped  : <0x00154020> { _bfin_rtc_interrupt + 0x0 }

  logical irq  48 mapped  : <0x00102bfc> { _bfin_serial_dma_rx_int + 0x0 }

  logical irq  49 mapped  : <0x00102990> { _bfin_serial_dma_tx_int + 0x0 }

  logical irq  52 mapped  : <0x00156ecc> { _bfin_twi_interrupt_entry + 0x0 }

  logical irq  53 mapped  : <0x00156ecc> { _bfin_twi_interrupt_entry + 0x0 }

  logical irq  68 mapped  : <0x00129bc4> { _bfin_ata_interrupt + 0x0 }

  logical irq  74 mapped  : <0x0013c058> { _bf5xx_nand_dma_irq + 0x0 }

  logical irq  76 mapped  : <0x001525f8> { _bfin_kpad_isr + 0x0 }

  logical irq  82 mapped  : <0x0014d39c> { _blackfin_interrupt + 0x0 }

  logical irq  85 mapped  : <0x0014fd18> { _dma_controller_irq + 0x0 }

  logical irq 175 mapped  : <0x0010ad10> { _smsc911x_irqhandler + 0x0 }

RETE: <0x00000000> /* Maybe null pointer? */

RETN: <0x002b3ce4> /* kernel dynamic memory */

RETX: <0x00000480> /* Maybe fixed code section */

RETS: <0x001ae618> { _tcp_v4_rcv + 0x254 }

PC  : <0x001ac538> { _tcp_v4_send_reset + 0xc4 }

DCPLB_FAULT_ADDR: <0x0000015f> /* Maybe null pointer? */

ICPLB_FAULT_ADDR: <0x0000ffff> { _daemonize + 0xc3 }

 

PROCESSOR STATE:

R0 : 00000008    R1 : 25015780    R2 : 14000600    R3 : 0f00a8c0

R4 : 00000000    R5 : 03a48c10    R6 : 00000002    R7 : 0200a8c0

P0 : 03a97a00    P1 : 03a48c24    P2 : 03a48c10    P3 : 0f00a8c0

P4 : 03a48c24    P5 : 00000000    FP : 00000000    SP : 002b3c08

LB0: 000dee50    LT0: 000dee50    LC0: 00000000

LB1: 000395be    LT1: 000395b8    LC1: 00000000

B0 : 000005a0    L0 : 00000000    M0 : 00000000    I0 : 0f00a8c0

B1 : 00000000    L1 : 00000000    M1 : 00000000    I1 : 0200a8c0

B2 : 00000000    L2 : 00000000    M2 : 00000000    I2 : 00000000

B3 : 00000000    L3 : 00000000    M3 : 00000000    I3 : 00000000

A0.w: 00000000   A0.x: 00000000   A1.w: 00000000   A1.x: 00000000

USP : 002b4000  ASTAT: 02002000

 

Hardware Trace:

   0 Target : <0x0000499c> { _trap_c + 0x0 }

     Source : <0xffa006a8> { _exception_to_level5 + 0x9c } CALL pcrel

   1 Target : <0xffa0060c> { _exception_to_level5 + 0x0 }

     Source : <0xffa0049a> { _bfin_return_from_exception + 0x6 } RTX

   2 Target : <0xffa00494> { _bfin_return_from_exception + 0x0 }

     Source : <0xffa00564> { _ex_trap_c + 0x6c } 0x0046

   3 Target : <0xffa004f8> { _ex_trap_c + 0x0 }

     Source : <0xffa00730> { _trap + 0x28 } JUMP (P4)

   4 Target : <0xffa00708> { _trap + 0x0 }

     Source : <0x001ac536> { _tcp_v4_send_reset + 0xc2 } 0x0000

   5 Target : <0x001ac52a> { _tcp_v4_send_reset + 0xb6 }

     Source : <0x001ac526> { _tcp_v4_send_reset + 0xb2 } IF !CC JUMP

   6 Target : <0x001ac522> { _tcp_v4_send_reset + 0xae }

     Source : <0x001ac51e> { _tcp_v4_send_reset + 0xaa } IF !CC JUMP

   7 Target : <0x001ac51a> { _tcp_v4_send_reset + 0xa6 }

     Source : <0x001ac516> { _tcp_v4_send_reset + 0xa2 } IF !CC JUMP

   8 Target : <0x001ac4a2> { _tcp_v4_send_reset + 0x2e }

     Source : <0x001ac498> { _tcp_v4_send_reset + 0x24 } IF !CC JUMP

   9 Target : <0x001ac474> { _tcp_v4_send_reset + 0x0 }

     Source : <0x001ae614> { _tcp_v4_rcv + 0x250 } CALL pcrel

  10 Target : <0x001ae610> { _tcp_v4_rcv + 0x24c }

     Source : <0x001ae602> { _tcp_v4_rcv + 0x23e } IF CC JUMP

  11 Target : <0x001ae5ec> { _tcp_v4_rcv + 0x228 }

     Source : <0x001ae5dc> { _tcp_v4_rcv + 0x218 } IF !CC JUMP

  12 Target : <0x001ae5d0> { _tcp_v4_rcv + 0x20c }

     Source : <0x001ae822> { _tcp_v4_rcv + 0x45e } JUMP.S

  13 Target : <0x001ae814> { _tcp_v4_rcv + 0x450 }

     Source : <0x0019e1f2> { ___inet_lookup_listener + 0xaa } RTS

  14 Target : <0x0019e1ec> { ___inet_lookup_listener + 0xa4 }

     Source : <0x0019e160> { ___inet_lookup_listener + 0x18 } IF CC JUMP

  15 Target : <0x0019e148> { ___inet_lookup_listener + 0x0 }

     Source : <0x001ae810> { _tcp_v4_rcv + 0x44c } CALL pcrel

 

Kernel Stack

Stack info:

SP: [0x002b3f58] <0x002b3f58> /* kernel dynamic memory */

Memory from 0x002b3f50 to 002b4000

002b3f50: 00000000  00000000 [00000000] 00000000  00000000  00000020  002b4000  0029a580

002b3f70: ffa00284  002b2000  0027e6a4  0029a580  002b2000  ffa0092c  0027de04  00000000

002b3f90: 00000000  00000000  00000065  0000001f  00000002  0000ffff  0000ffff  ffa0092c

002b3fb0: 00000006  00000000  00000000  00000000  002b56c6  0028a1f0  002d8a14  0027de10

002b3fd0: 00000000  002d8a14  0027d6e8  0000005a  002b50a8  002daa60  002b4000  002ca6be

002b3ff0: 00000000  00000000  00000000  ffb00000  00000000

Return addresses in stack:

Modules linked in:

Kernel panic - not syncing: Kernel exception

=======================================================

 

.config is attached.

 

Follow-ups

 

--- Colin McCabe                                             2008-11-11 15:59:23

Just a note: we have reproduced this on two EZ-KIT Lite BF548 boards now.

 

--- Sonic Zhang                                              2008-11-12 02:22:33

You should built your svn trunk kernel with svn trunk toolchain and boot it by

svn trunk uboot as well.

 

--- Sonic Zhang                                              2008-11-12 23:12:10

Can't replicate.

 

--- Mike Frysinger                                           2008-11-12 23:21:26

which means "not reproducible", not "rejected"

 

Colin: can you describe exactly what's going on in the system ?  we use the

BF548-EZKIT and networking quite a lot and have yet to see this.  so knowing

exactly how you're using the system would help a lot.

 

--- Colin McCabe                                             2008-11-13 14:16:52

Hi Mike,

 

Thanks for responding.

 

Our root filesystem is mounted over NFS. Have you ever used that

configuration?

 

Also...we are using

git://sources.blackfin.uclinux.org/git/readonly-mirrors/linux-kernel.git rather

than svn://sources.blackfin.uclinux.org/linux-kernel/trunk linux-kernel. I

believe that the git tree is supposed to be identical to what is in subversion.

However, since switching to using subversion, I haven't seen the problem.

 

Colin

 

--- Colin McCabe                                             2008-11-13 14:21:27

And thanks to Sonic as well. Your idea about svn was a good one, it seems!

 

--- Colin McCabe                                             2008-11-13 14:25:04

Update: I spoke too soon.

After several minutes, I got the same oops on the subversion kernel.

 

root:/> uname -r

2.6.28-rc2-ADI-2009R1-pre-svn5698

 

root:/> NULL pointer access

Kernel OOPS in progress

Deferred Exception context

 

--- Mike Frysinger                                           2008-11-13 15:37:02

git is an exact mirror of svn (just delayed ~1 hour).  any behavior changes are

most likely coincidence rather than anything else.

 

we have tested NFS root before, but not necessarily with the BF548-EZKIT.  what

are your exact mount options and what are you doing on the board ?  just sitting

idle ?

 

--- Colin McCabe                                             2008-11-13 15:46:32

U-Boot settings:

 

bfin> printenv bootargs

bootargs=root=/dev/nfs rw

nfsroot=192.168.0.2:/opt/target/blackfin-uClinux-dist-2008R1.5-RC3/romfs/,tcp,nfsvers=3

ip=192.168.0.15:192.168.0.2:192.168.0.1:255.255.255.0:1:eth0:off

earlyprintk=serial,uart1,57600 console=tty0 console=ttyBF0,57600

 

I boot by doing:

bfin> tftp 0x1000000 vmImage

bfin> bootm

 

And the kernel .config is attached.

 

The oops doesn't seem to be triggered by any specific behavior. Sometimes it

will happen immediately after I see the motd. Other times the board will be up

for 5 minutes before it goes down.

 

Colin

 

--- Mike Frysinger                                           2008-11-13 15:57:46

sorry, could you also describe the NFS server ?  what software versions

(nfs-utils / kernel / etc...) and what is the /etc/exports line for it ?

 

--- Colin McCabe                                             2008-11-13 17:17:59

I'm running nfs-kernel-server as the NFS server.

The board is connected via a crossover cable to my laptop's eth1 interface.

The problem has been reproduced with the board connected to other ethernet

interfaces, though.

 

/etc/exports has /opt

192.168.0.15(rw,no_root_squash,async,subtree_check,insecure)

 

Some version / configuration information for the NFS server:

============================================================

colin@calvin:~$ uname -r

2.6.24-16-generic

============================================================

colin@calvin:~$ cat /etc/issue

Ubuntu 8.04.1 \n \l

============================================================

colin@calvin:~$ cat /etc/default/nfs-kernel-server

# Number of servers to start up

RPCNFSDCOUNT=8

 

# Runtime priority of server (see nice(1))

RPCNFSDPRIORITY=0

 

# Options for rpc.mountd.

# If you have a port-based firewall, you might want to set up

# a fixed port here using the --port option. For more information,

# see rpc.mountd(8) or   wiki.debian.org/?SecuringNFS

RPCMOUNTDOPTS=

 

# Do you want to start the svcgssd daemon? It is only required for Kerberos

# exports. Valid alternatives are "yes" and "no"; the

default is "no".

NEED_SVCGSSD=

 

# Options for rpc.svcgssd.

RPCSVCGSSDOPTS=

============================================================

colin@calvin:~$ /sbin/portmap -V

portmap version 6.0 - 2007-May-11

============================================================

colin@calvin:~$ dpkg -p  nfs-kernel-server

Package: nfs-kernel-server

Priority: optional

Section: net

Installed-Size: 364

Maintainer: Ubuntu Core Developers

<ubuntu-devel-discuss@lists.ubuntu.com>

Architecture: i386

Source: nfs-utils

Version: 1:1.1.2-2ubuntu2.1

Replaces: knfs, nfs-server

Provides: knfs, nfs-server

Depends: libblkid1 (>= 1.39+1.40-WIP-2006.11.14+dfsg-2), libc6 (>= 2.4),

libcomerr2 (>= 1.33-3), libgssglue1, libkrb53 (>= 1.6.dfsg.2),

libldap-2.4-2 (>= 2.4.7), libnfsidmap2, librpcsecgss3, libwrap0, lsb-base

(>= 1.3-9ubuntu3), nfs-common (>= 1:1.0.8-1), ucf

Conflicts: knfs, nfs-server

Size: 150288

Description: support for NFS kernel server

The NFS kernel server is currently the recommended NFS server for use

with Linux, featuring features such as NFSv3 and NFSv4, Kerberos

support via GSS, and much more. It is also significantly faster and

usually more reliable than the user-space NFS servers (from the

unfs3 and nfs-user-server packages). However, it is more difficult to

debug than the user-space servers, and has a slightly different

feature set.

.

This package contains the user-space support needed to use the

NFS kernel server. Most administrators wishing to set up an NFS server

would want to install this package.

.

Upstream: SourceForge project "nfs", CVS module nfs-utils.

Homepage:   nfs.sourceforge.net/

Original-Maintainer: Anibal Monsalve Salazar <anibal@debian.org>

 

--- Colin McCabe                                             2008-11-13 17:59:34

I tried some different options in /etc/exports.

/opt 192.168.0.15(rw,no_root_squash,no_subtree_check)

 

After restarting the NFS server and the board, however, there was no change

from the behavior under the old /etc/exports.

 

--- Michael Hennerich                                        2008-11-14 05:15:30

I've seen this bug in _tcp_v4_send_reset happen on other platforms as well.

It was somehow introduced when we updated our trunk kernel to 2.6.28.

 

This is what I just got on BF537 with the EMAC.

I also think I saw that someone open a different bug for this as well.

 

-Michael

 

NULL pointer access q=2.6 size=     343kB time=8.4 bitrate= 333.1kbits/s

Kernel OOPS in progress

Deferred Exception context

CURRENT PROCESS:

COMM=ffmpeg PID=216

CPU = 0

TEXT = 0x008e0000-0x008f151c        DATA = 0x00b4051c-0x00b42828

BSS = 0x00b42828-0x00880000  USER-STACK = 0x0089fe40

 

return address: [0x00211418]; contents of:

0x002113f0:  5079  030c  1002  5041  5059  030c  1002  5041

0x00211400:  5051  030c  1002  5041  0000  6040  b1f1  b230

0x00211410:  184b  0000  0000  0000 [e4a8] 015f  4928  0200

0x00211420:  e14d  002c  e10d  ae08  a171  3056  b1b0  6482

 

SEQUENCER STATUS:               Not tainted

SEQSTAT: 00060027  IPEND: 4030  SYSCFG: 0006

  EXCAUSE   : 0x27

  physical IVG14 asserted : <0xffa00b40> { _evt14_softirq + 0x0 }

  logical irq   6 mapped  : <0xffa00460> { _timer_interrupt + 0x0 }

  logical irq  10 mapped  : <0x001d6760> { _bfin_rtc_interrupt + 0x0 }

  logical irq  11 mapped  : <0x001c5ea4> { _bcap_ppi_irq + 0x0 }

  logical irq  16 mapped  : <0x001d8e6c> { _bfin_twi_interrupt_entry +

0x0 }

  logical irq  18 mapped  : <0x001a29a0> { _bfin_serial_dma_rx_int + 0x0

}

  logical irq  19 mapped  : <0x001a2710> { _bfin_serial_dma_tx_int + 0x0

}

  logical irq  24 mapped  : <0x001ab3dc> { _bfin_mac_interrupt + 0x0 }

  logical irq  42 mapped  : <0x001c5990> { _bcap_ppi_irq_error + 0x0 }

  logical irq  54 mapped  : <0x001aea30> { _orinoco_interrupt + 0x0 }

RETE: <0x00000000> { _do_one_initcall + 0xfff00000 }

RETN: <0x00d13d34> /* kernel dynamic memory */

RETX: <0x00000480> /* Maybe fixed code section */

RETS: <0x002135aa> { _tcp_v4_rcv + 0x262 }

PC  : <0x00211418> { _tcp_v4_send_reset + 0xcc }

DCPLB_FAULT_ADDR: <0x0000015f> /* Maybe null pointer? */

ICPLB_FAULT_ADDR: <0x00211418> { _tcp_v4_send_reset + 0xcc }

 

PROCESSOR STATE:

R0 : 00000008    R1 : 99045e14    R2 : 14000600    R3 : 2d022c0a

R4 : 00000000    R5 : 00a5e820    R6 : 00000002    R7 : 58022c0a

P0 : 037813c0    P1 : 00a5e834    P2 : 00a5e820    P3 : 2d022c0a

P4 : 00a5e834    P5 : 00000000    FP : 00000000    SP : 00d13c58

LB0: ffa0177c    LT0: ffa0177c    LC0: 00000000

LB1: 00107507    LT1: 00107506    LC1: 00000000

B0 : 00946240    L0 : 00000000    M0 : 0000000c    I0 : 2d022c0a

B1 : ff8000c0    L1 : 00000000    M1 : 00000010    I1 : 58022c0a

B2 : ff8000c0    L2 : 00000000    M2 : 00000080    I2 : 009462c0

B3 : ff900100    L3 : 00000000    M3 : 000000ac    I3 : ffffffff

A0.w: 00000000   A0.x: 00000000   A1.w: 00000000   A1.x: 00000000

USP : 00894444  ASTAT: 02002002

 

Hardware Trace:

   0 Target : <0x001039bc> { _trap_c + 0x0 }

     Source : <0xffa007ca> { _exception_to_level5 + 0x9e } CALL pcrel

   1 Target : <0xffa0072c> { _exception_to_level5 + 0x0 }

     Source : <0xffa005be> { _bfin_return_from_exception + 0xe } RTX

   2 Target : <0xffa005b0> { _bfin_return_from_exception + 0x0 }

     Source : <0xffa00682> { _ex_trap_c + 0x66 } JUMP.S

   3 Target : <0xffa0061c> { _ex_trap_c + 0x0 }

     Source : <0xffa0049a> { _ex_workaround_261 + 0x22 } JUMP.S

   4 Target : <0xffa00478> { _ex_workaround_261 + 0x0 }

     Source : <0xffa00864> { _trap + 0x38 } JUMP (P4)

   5 Target : <0xffa0084a> { _trap + 0x1e }

     Source : <0xffa00846> { _trap + 0x1a } IF !CC JUMP

   6 Target : <0xffa0082c> { _trap + 0x0 }

     Source : <0xffa005be> { _bfin_return_from_exception + 0xe } RTX

   7 Target : <0xffa005b0> { _bfin_return_from_exception + 0x0 }

     Source : <0xffa0048a> { _ex_workaround_261 + 0x12 } IF !CC JUMP

   8 Target : <0xffa00478> { _ex_workaround_261 + 0x0 }

     Source : <0xffa00864> { _trap + 0x38 } JUMP (P4)

   9 Target : <0xffa0084a> { _trap + 0x1e }

     Source : <0xffa00846> { _trap + 0x1a } IF !CC JUMP

  10 Target : <0xffa0082c> { _trap + 0x0 }

     Source : <0x00211416> { _tcp_v4_send_reset + 0xca } 0x0000

  11 Target : <0x00211408> { _tcp_v4_send_reset + 0xbc }

     Source : <0x00211404> { _tcp_v4_send_reset + 0xb8 } IF !CC JUMP

  12 Target : <0x00211400> { _tcp_v4_send_reset + 0xb4 }

     Source : <0x002113fc> { _tcp_v4_send_reset + 0xb0 } IF !CC JUMP

  13 Target : <0x002113f8> { _tcp_v4_send_reset + 0xac }

     Source : <0x002113f4> { _tcp_v4_send_reset + 0xa8 } IF !CC JUMP

  14 Target : <0x002113c4> { _tcp_v4_send_reset + 0x78 }

     Source : <0x002114a4> { _tcp_v4_send_reset + 0x158 } JUMP.S

  15 Target : <0x00211458> { _tcp_v4_send_reset + 0x10c }

     Source : <0x002113b8> { _tcp_v4_send_reset + 0x6c } IF !CC JUMP

 

Kernel Stack

Stack info:

SP: [0x00d13f24] <0x00d13f24> /* kernel dynamic memory */

Memory from 0x00d13f20 to 00d14000

00d13f20: 00000480 [00699418] ffffffff  00000000  00000000  00d14000  00d4d018

00699418

00d13f40:<0069933a> 0000000b  02001025  ffa023a5  0069941d  ffa0233c

00699412  00000000

00d13f60: 00000033  00000000  00000000  00000000  00000000  ff900100  ff8000c0

ff8000c0

00d13f80: 00946240  00000000  00000000  00000000  00000000  000000ac  00000080

00000010

00d13fa0: 0000000c  ffffffff  009462c0  009462c4  00950290  00894444  00946240

0069c216

00d13fc0: 00940010  00010001  0000003f  00950290  0094629a  00010001  00000000

00000001

00d13fe0: 00e0e5d0  00000001  08000800  ffffffff  00000000  00000000  0094629a

00000006

00d14000: 000028c8

Return addresses in stack:

    address : <0x0069933a> [ /usr/lib/libavcodec.so.51 + 0x29933a ]

Modules linked in:

Kernel panic - not syncing: Kernel exception                                  

                                                                               

            

 

--- Michael Hennerich                                        2008-11-14 05:18:47

Check out this Bf533 bug opened by Mike:

 

  blackfin.uclinux.org/gf/project/uclinux-dist/tracker/?action=TrackerItemEdit&tracker_item_id=4593

 

Same thing:

 

  10 Target : <0xffa00768> { _trap + 0x0 }

     Source : <0x000d93c2> { _tcp_v4_send_reset + 0xc6 } 0x0000

 

-Michael

 

--- Michael Hennerich                                        2008-11-20 05:33:16

Same issue as:

 

[#4618] [ltp] ltp test case connect01 crashed in trunk

[#4593] random crash in tcp code

 

--- Mike Frysinger                                           2008-11-20 08:41:40

move to LTP bug then

 

 

 

    Files

    Changes

    Commits

    Dependencies

    Duplicates

    Associations

    Tags

 

File Name     File Type     File Size     Posted By

config    application/octet-stream    40937    Colin McCabe

Attachments

Outcomes