2009-03-31 17:20:36     application dies in the kernal on user space violation

2009-03-31 17:20:36     application dies in the kernal on user space violation

Doug Bailey (UNITED STATES)

Message: 71910   

I have a multi-threaded application that induces the following error when it

runs under periods of high loads.  I can trap this error under gdb/gdbserver but

it appears that the error is occurring in the kernel during a system call.

(Which call, I do not know.)  I see nothing via the gdb debugger that indicates

that there is a problem with the data variables being passed.

The implications of this error are a little beyond me at this time.  The return

address that is flagged in the error maps to _evt_system_call which I would not

expect to see in user space.  Is there something that I am missing that I need

to be looking at?

(The system I am running is running an old kernel (2.6.16.27) so this issue may

have already been addressed but I need to determine what is happening on my

production system.)

console output:

- Attempted to use a Supervisor register or instruction from User mode.

Supervisor resources are registers and instructions that are reserved

for Supervisor use: Supervisor only registers, all MMRs, and Supervisor

only instructions.

CURRENT PROCESS:

COMM=asterisk PID=2696

TEXT = 0x002d8000-0x002dcaec  DATA = 0x04300924-0x0433e3e8

BSS = 0x0433e3e8-0x01a20000   USER-STACK = 0x01a3feb0

return address: 0x000099c0; contents of [PC-16...PC+8]:

0140 05c0 6058 2eb4 017a 0148 0140 05c0

6060 2eae 017a 0148 0140 05c0 6068 2ea8 X

017a 0148 0140 05c0 014f 0178 0150 0151

RETE:  00000000  RETN: 0509e000  RETX: 000099c0  RETS: 014b6cac

IPEND: 0030  SYSCFG: 0036

SEQSTAT: 0006002e    SP: 0509df24

R0: fffffffc    R1: 00000008    R2: 00000008    R3: 054cb3b0

R4: 00000000    R5: 05c00004    R6: 00000000    R7: 0144cb80

P0: 000000b3    P1: 003f1228    P2: 054cb3b0    P3: 01f9adc8

P4: 054cbe24    P5: 054cb3b0    FP: 054cb398

A0.w: 00000000    A0.x: 00000000    A1.w: 00000000    A1.x: 00000000

LB0: 003ed9df  LT0: 003ed9de  LC0: 00000000

LB1: 00c31f77  LT1: 00c31f76  LC1: 00000000

B0: fffff052  L0: 00000000  M0: 04a0c0c0  I0: 054ca7a0

B1: ffffec24  L1: 00000000  M1: 00000004  I1: 00001f54

B2: 00000000  L2: 00000000  M2: 00000001  I2: ffffec24

B3: 054ca508  L3: 00000000  M3: 054ca50f  I3: fffff052

USP: 054cb38c   ASTAT: 02001024

DCPLB_FAULT_ADDR=06539ee0

ICPLB_FAULT_ADDR=0000b0d0

Hardware Trace:

0 Target : <0x00004f48> { _trap_c + 0x0 }

   Source : <0x000091f0> { _exception_to_level5 + 0xb4 }

1 Target : <0x0000913c> { _exception_to_level5 + 0x0 }

   Source : <0x0000913a> { _ex_trap_c + 0x4e }

2 Target : <0x000090ec> { _ex_trap_c + 0x0 }

   Source : <0x00009290> { _trap + 0x28 }

3 Target : <0x00009268> { _trap + 0x0 }

   Source : <0x000097e6> { __common_int_entry + 0xd8 }

4 Target : <0x00009784> { __common_int_entry + 0x76 }

   Source : <0x00009a24> { _evt_system_call + 0x64 }

5 Target : <0x00009a24> { _evt_system_call + 0x64 }

   Source : <0x00009418> { _system_call + 0xb8 }

6 Target : <0x00009414> { _system_call + 0xb4 }

   Source : <0x00009404> { _system_call + 0xa4 }

7 Target : <0x000093fe> { _system_call + 0x9e }

   Source : <0x000093ee> { _system_call + 0x8e }

8 Target : <0x000093c8> { _system_call + 0x68 }

   Source : <0x00001a6c> { _sys_sigreturn + 0x10 }

9 Target : <0x00001a68> { _sys_sigreturn + 0xc }

   Source : <0x00002ed2> { _do_sigreturn + 0x432 }

10 Target : <0x00002ec8> { _do_sigreturn + 0x428 }

   Source : <0x00002dea> { _do_sigreturn + 0x34a }

11 Target : <0x00002cac> { _do_sigreturn + 0x20c }

   Source : <0x0009369a> { _memcpy + 0x52 }

12 Target : <0x0009367a> { _memcpy + 0x32 }

   Source : <0x0009366a> { _memcpy + 0x22 }

13 Target : <0x0009365c> { _memcpy + 0x14 }

   Source : <0x00093654> { _memcpy + 0xc }

14 Target : <0x00093648> { _memcpy + 0x0 }

   Source : <0x00002ca8> { _do_sigreturn + 0x208 }

15 Target : <0x00002c92> { _do_sigreturn + 0x1f2 }

   Source : <0x00002f78> { _do_sigreturn + 0x4d8 }

Stack from 0509de80:

        fffff052 00000004 00000000 00030005 000099c0 00000000 00000000 00000000

        00000000 fffff052 ffffec24 00000000 054ca508 00000000 00000000 00000000

        00000000 02001024 014b6cac 00000000 00000000 003ed9de 00c31f76 003ed9df

        00c31f77 00060026 003f123c 000099c0 00001a68 00001a5c 00000077 0509c000

        000000b3 000091f4 001627f0 001627f0 001627e8 05c00004 054cb3b0 00000000

        054cb2de 000099c0 00000030 0006002e 00000000 0509e000 000099c0 000099c0

Call Trace:

[<00001f54>] _arch_ptrace+0x50/0x49c

=========================================

bfin-uclinux-gdb output:

(gdb) bt

#0  0x000099c0 in ?? ()

#1  0x014b6cac in __pthread_wait_for_restart_signal () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libpthread.so.0

#2  0x014b6cac in __pthread_wait_for_restart_signal () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libpthread.so.0

#3  0x014b7b64 in sem_wait () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libpthread.so.0

#4  0x00c207ea in _UseMem (context=0x5c00004) at commetrex/tt38src/dscrtapi.c:77

#5  0x00c213fa in DSC_Wait_for_Absolute_Time (context=0x54cb3b0) at commetrex/tt38src/dscrtdll.c:299

#6  0x00c21668 in RT_Queue_Mgr (param=0x5c00004) at commetrex/tt38src/dscrtdll.c:507

#7  0x014b5366 in pthread_start_thread () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libpthread.so.0

#8  0x014b53b2 in pthread_start_thread_event () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libpthread.so.0

#9  0x003eeaa8 in clone () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

(gdb) info threads

  38 thread 2876  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  37 thread 2875  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  36 thread 2874  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  35 thread 2873  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  34 thread 2872  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  33 thread 2871  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  32 thread 2870  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  31 thread 2869  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  30 thread 2868  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  29 thread 2867  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  28 thread 2695  0x003f030c in nanosleep () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  27 thread 2694  0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  26 thread 1677  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  25 thread 1676  0x003f030c in nanosleep () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  24 thread 1675  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  23 thread 1674  0x003f030c in nanosleep () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  22 thread 1673  0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  21 thread 1672  0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  20 thread 1671  0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  19 thread 1670  0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  18 thread 1669  0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  17 thread 1668  0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  16 thread 1667  0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  15 thread 1666  0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  14 thread 1665  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  13 thread 1664  0x003f030c in nanosleep () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  12 thread 1663  0x003f030c in nanosleep () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  11 thread 1662  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  10 thread 1660  0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  9 thread 1659  0x003f0a40 in select () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  8 thread 1658  0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  7 thread 1657  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  6 thread 1656  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  5 thread 1655  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  4 thread 1654  0x003f0756 in read () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

* 3 thread 2696  0x000099c0 in ?? ()

  2 thread 2359  0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

  1 thread 1661  0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0

QuoteReplyEditDelete

2009-03-31 17:59:53     Re: application dies in the kernal on user space violation

Mike Frysinger (UNITED STATES)

Message: 71914   

2.6.16 is like 2007R1 isnt it ?  we have fixed code in the kernel related to syscall return paths that only occurred under high loads ...

QuoteReplyEditDelete

2009-04-01 16:50:31     Re: application dies in the kernal on user space violation

Doug Bailey (UNITED STATES)

Message: 71979   

I realize that the answer to this issue is to upgrade the kernel.  However, I would be interested to see where these changes were made.  Do you have a time frame/commit # when the changes were made or source files that I should look at?

- Doug

QuoteReplyEditDelete

2009-04-02 06:16:07     Re: application dies in the kernal on user space violation

Mike Frysinger (UNITED STATES)

Message: 72034   

it's been a while at this point, but i'd say look at mach-common/entry.S and mach-common/interrupt.S for changes related to anomalies