2009-03-31 17:20:36 application dies in the kernal on user space violation
Doug Bailey (UNITED STATES)
Message: 71910
I have a multi-threaded application that induces the following error when it
runs under periods of high loads. I can trap this error under gdb/gdbserver but
it appears that the error is occurring in the kernel during a system call.
(Which call, I do not know.) I see nothing via the gdb debugger that indicates
that there is a problem with the data variables being passed.
The implications of this error are a little beyond me at this time. The return
address that is flagged in the error maps to _evt_system_call which I would not
expect to see in user space. Is there something that I am missing that I need
to be looking at?
(The system I am running is running an old kernel (2.6.16.27) so this issue may
have already been addressed but I need to determine what is happening on my
production system.)
console output:
- Attempted to use a Supervisor register or instruction from User mode.
Supervisor resources are registers and instructions that are reserved
for Supervisor use: Supervisor only registers, all MMRs, and Supervisor
only instructions.
CURRENT PROCESS:
COMM=asterisk PID=2696
TEXT = 0x002d8000-0x002dcaec DATA = 0x04300924-0x0433e3e8
BSS = 0x0433e3e8-0x01a20000 USER-STACK = 0x01a3feb0
return address: 0x000099c0; contents of [PC-16...PC+8]:
0140 05c0 6058 2eb4 017a 0148 0140 05c0
6060 2eae 017a 0148 0140 05c0 6068 2ea8 X
017a 0148 0140 05c0 014f 0178 0150 0151
RETE: 00000000 RETN: 0509e000 RETX: 000099c0 RETS: 014b6cac
IPEND: 0030 SYSCFG: 0036
SEQSTAT: 0006002e SP: 0509df24
R0: fffffffc R1: 00000008 R2: 00000008 R3: 054cb3b0
R4: 00000000 R5: 05c00004 R6: 00000000 R7: 0144cb80
P0: 000000b3 P1: 003f1228 P2: 054cb3b0 P3: 01f9adc8
P4: 054cbe24 P5: 054cb3b0 FP: 054cb398
A0.w: 00000000 A0.x: 00000000 A1.w: 00000000 A1.x: 00000000
LB0: 003ed9df LT0: 003ed9de LC0: 00000000
LB1: 00c31f77 LT1: 00c31f76 LC1: 00000000
B0: fffff052 L0: 00000000 M0: 04a0c0c0 I0: 054ca7a0
B1: ffffec24 L1: 00000000 M1: 00000004 I1: 00001f54
B2: 00000000 L2: 00000000 M2: 00000001 I2: ffffec24
B3: 054ca508 L3: 00000000 M3: 054ca50f I3: fffff052
USP: 054cb38c ASTAT: 02001024
DCPLB_FAULT_ADDR=06539ee0
ICPLB_FAULT_ADDR=0000b0d0
Hardware Trace:
0 Target : <0x00004f48> { _trap_c + 0x0 }
Source : <0x000091f0> { _exception_to_level5 + 0xb4 }
1 Target : <0x0000913c> { _exception_to_level5 + 0x0 }
Source : <0x0000913a> { _ex_trap_c + 0x4e }
2 Target : <0x000090ec> { _ex_trap_c + 0x0 }
Source : <0x00009290> { _trap + 0x28 }
3 Target : <0x00009268> { _trap + 0x0 }
Source : <0x000097e6> { __common_int_entry + 0xd8 }
4 Target : <0x00009784> { __common_int_entry + 0x76 }
Source : <0x00009a24> { _evt_system_call + 0x64 }
5 Target : <0x00009a24> { _evt_system_call + 0x64 }
Source : <0x00009418> { _system_call + 0xb8 }
6 Target : <0x00009414> { _system_call + 0xb4 }
Source : <0x00009404> { _system_call + 0xa4 }
7 Target : <0x000093fe> { _system_call + 0x9e }
Source : <0x000093ee> { _system_call + 0x8e }
8 Target : <0x000093c8> { _system_call + 0x68 }
Source : <0x00001a6c> { _sys_sigreturn + 0x10 }
9 Target : <0x00001a68> { _sys_sigreturn + 0xc }
Source : <0x00002ed2> { _do_sigreturn + 0x432 }
10 Target : <0x00002ec8> { _do_sigreturn + 0x428 }
Source : <0x00002dea> { _do_sigreturn + 0x34a }
11 Target : <0x00002cac> { _do_sigreturn + 0x20c }
Source : <0x0009369a> { _memcpy + 0x52 }
12 Target : <0x0009367a> { _memcpy + 0x32 }
Source : <0x0009366a> { _memcpy + 0x22 }
13 Target : <0x0009365c> { _memcpy + 0x14 }
Source : <0x00093654> { _memcpy + 0xc }
14 Target : <0x00093648> { _memcpy + 0x0 }
Source : <0x00002ca8> { _do_sigreturn + 0x208 }
15 Target : <0x00002c92> { _do_sigreturn + 0x1f2 }
Source : <0x00002f78> { _do_sigreturn + 0x4d8 }
Stack from 0509de80:
fffff052 00000004 00000000 00030005 000099c0 00000000 00000000 00000000
00000000 fffff052 ffffec24 00000000 054ca508 00000000 00000000 00000000
00000000 02001024 014b6cac 00000000 00000000 003ed9de 00c31f76 003ed9df
00c31f77 00060026 003f123c 000099c0 00001a68 00001a5c 00000077 0509c000
000000b3 000091f4 001627f0 001627f0 001627e8 05c00004 054cb3b0 00000000
054cb2de 000099c0 00000030 0006002e 00000000 0509e000 000099c0 000099c0
Call Trace:
[<00001f54>] _arch_ptrace+0x50/0x49c
=========================================
bfin-uclinux-gdb output:
(gdb) bt
#0 0x000099c0 in ?? ()
#1 0x014b6cac in __pthread_wait_for_restart_signal () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libpthread.so.0
#2 0x014b6cac in __pthread_wait_for_restart_signal () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libpthread.so.0
#3 0x014b7b64 in sem_wait () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libpthread.so.0
#4 0x00c207ea in _UseMem (context=0x5c00004) at commetrex/tt38src/dscrtapi.c:77
#5 0x00c213fa in DSC_Wait_for_Absolute_Time (context=0x54cb3b0) at commetrex/tt38src/dscrtdll.c:299
#6 0x00c21668 in RT_Queue_Mgr (param=0x5c00004) at commetrex/tt38src/dscrtdll.c:507
#7 0x014b5366 in pthread_start_thread () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libpthread.so.0
#8 0x014b53b2 in pthread_start_thread_event () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libpthread.so.0
#9 0x003eeaa8 in clone () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
(gdb) info threads
38 thread 2876 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
37 thread 2875 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
36 thread 2874 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
35 thread 2873 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
34 thread 2872 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
33 thread 2871 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
32 thread 2870 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
31 thread 2869 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
30 thread 2868 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
29 thread 2867 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
28 thread 2695 0x003f030c in nanosleep () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
27 thread 2694 0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
26 thread 1677 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
25 thread 1676 0x003f030c in nanosleep () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
24 thread 1675 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
23 thread 1674 0x003f030c in nanosleep () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
22 thread 1673 0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
21 thread 1672 0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
20 thread 1671 0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
19 thread 1670 0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
18 thread 1669 0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
17 thread 1668 0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
16 thread 1667 0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
15 thread 1666 0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
14 thread 1665 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
13 thread 1664 0x003f030c in nanosleep () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
12 thread 1663 0x003f030c in nanosleep () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
11 thread 1662 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
10 thread 1660 0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
9 thread 1659 0x003f0a40 in select () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
8 thread 1658 0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
7 thread 1657 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
6 thread 1656 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
5 thread 1655 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
4 thread 1654 0x003f0756 in read () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
* 3 thread 2696 0x000099c0 in ?? ()
2 thread 2359 0x003f123c in __rt_sigsuspend () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
1 thread 1661 0x003f04e4 in poll () from /usr/src/dahdifun/toolchain/buildscript/out-uclinux/bfin-uclinux/runtime/lib/libc.so.0
QuoteReplyEditDelete
2009-03-31 17:59:53 Re: application dies in the kernal on user space violation
Mike Frysinger (UNITED STATES)
Message: 71914
2.6.16 is like 2007R1 isnt it ? we have fixed code in the kernel related to syscall return paths that only occurred under high loads ...
QuoteReplyEditDelete
2009-04-01 16:50:31 Re: application dies in the kernal on user space violation
Doug Bailey (UNITED STATES)
Message: 71979
I realize that the answer to this issue is to upgrade the kernel. However, I would be interested to see where these changes were made. Do you have a time frame/commit # when the changes were made or source files that I should look at?
- Doug
QuoteReplyEditDelete
2009-04-02 06:16:07 Re: application dies in the kernal on user space violation
Mike Frysinger (UNITED STATES)
Message: 72034
it's been a while at this point, but i'd say look at mach-common/entry.S and mach-common/interrupt.S for changes related to anomalies