2010-11-14 00:17:37     httpd crash

Document created by Aaronwu Employee on Aug 23, 2013
Version 1Show Document
  • View in full screen mode

2010-11-14 00:17:37     httpd crash

Wojtek Skulski (UNITED STATES)

Message: 95824   

 

Hi:

 

I am testing my own version of ndso. It basically works, but from time to time the httpd server is crashing. It happens without any special CGI activity. I mean, after some plotting and testing I go to do some stuff unrelated to uClinux, and when I come back I see the crash on the serial console. Here is the example. Process "183" was the one started with "ndso &", that is the httpd server.

 

A couple questions. (1) What is going on and how to fix it? (2) Why does ndso have its own httpd rather than "newest greatest official" httpd? Is it normal that every web applications carries its own private version of httpd?

 

root:/home/httpd/cgi-bin> Allocation of length 271081472 from process 183 failed

 

DMA per-cpu:

CPU    0: hi:   18, btch:   3 usd:   9

Active_anon:0 active_file:496 inactive_anon:0

inactive_file:2906 dirty:0 writeback:0 unstable:0

free:10669 slab:395 mapped:0 pagetables:0 bounce:0

DMA free:42676kB min:4096kB low:5120kB high:6144kB active_anon:0kB inactive_anon

:0kB active_file:1984kB inactive_file:11624kB present:64004kB pages_scanned:0 al

l_unreclaimable? no

lowmem_reserve[]: 0 0 0

DMA: 21*4kB 6*8kB 7*16kB 2*32kB 0*64kB 1*128kB 1*256kB 2*512kB 2*1024kB 1*2048kB

1*4096kB 0*8192kB 2*16384kB 0*32768kB = 42676kB

3405 total pagecache pages

QuoteReplyEditDelete

 

 

2010-11-14 14:43:19     Re: httpd crash

Mike Frysinger (UNITED STATES)

Message: 95834   

 

your system doesnt have 271MB of memory to allocate, so dont try to allocate that much.  also make sure you arent smashing the stack:

  docs.blackfin.uclinux.org/doku.php?id=uclinux-dist:debugging_applications#stack_checking

QuoteReplyEditDelete

 

 

2010-11-15 03:58:07     httpd crash

Michael Hennerich (GERMANY)

Message: 95845    (1) for me this smells like a stack overflow.

The cgi process forked by thttpd inherits the thttpd stack.

So try to increase it.

 

(2) That's not normal - many years ago, when ndso was done as a example.

Thttpd from uclinux-dist/user required modifications, I don't remember exactly.

But I think they were unacceptable to put into uclinux-dist/user/thttpd.

QuoteReplyEditDelete

 

 

2010-11-15 11:26:48     httpd crash

Wojtek Skulski (UNITED STATES)

Message: 95859    Michael:

thank you for the answer. I left running the board overnight and when I

came back to day morning, httpd was crashed despite of lack of any

activity overnight. It means that the cgi process has not been called, and

yet the httpd crashed.

 

root:/home/httpd/cgi-bin> NULL pointer access

Deferred Exception context

CURRENT PROCESS:

COMM=ndso PID%7

CPU = 0

 

ad 1) Could you please suggest how to increase the stack? I am not fluent

with the compiler options and make. Could you please suggest what to put

into the Makefile to increase the stack?

 

ad 2) The httpd included with ndso prints the date 1998. Looks pretty old

to me. Perhaps I am dealing with some obscure bug which has been fixed

long ago. I can try to use the more recent httpd which I think is provided

in "user" directory. I can take the newer httpd binary and just copy it

over to /bin to replace the faulty httpd. Does it make sense to you?

 

Thank you -- Wojtek

QuoteReplyEditDelete

 

 

2010-11-15 14:49:02     Re: httpd crash

Mike Frysinger (UNITED STATES)

Message: 95864   

 

i already pointed out the documentation that covers stack management

QuoteReplyEditDelete

 

 

2010-11-16 18:24:12     Re: httpd crash

Wojtek Skulski (UNITED STATES)

Message: 95882   

 

Mike:

 

thank you for the wiki pointer. I added the stack options to the Makefile

 

CFLAGS += -mstack-check-l1 -elf2flt="-s 8192"

 

And indeed it told me more than just a random crash at random time. The exception happened immediately while calculating an FFT. So I now know a likely culprit.

 

root:/home/httpd/cgi-bin> Application stack overflow

- Please increase the stack size of the application using elf2flt -s option,

   and/or reduce the stack use of the application.

Deferred Exception context

CURRENT PROCESS:

COMM=ndso.cgi PID=392

CPU = 0

QuoteReplyEditDelete

 

 

2010-11-16 19:51:39     Re: httpd crash

Wojtek Skulski (UNITED STATES)

Message: 95883    After increasing the stack the FFT does not crash anymore, but the random

"memory allocation" crash still happens. Like Michael said, it smells like

stack overflow, but now the stack is guarded with -mstack-check-l1. So it

must be something else.

 

If not the stack, then perhaps the heap. I got rid of allocating the

sample buffer, but the file names are still generated on the heap and then

freed before the CGI exits. I wonder how robust is the "free" and what

happens if not all memory is freed before the program exits.

 

root:/home/httpd/cgi-bin> Allocation of length 277307392 from process 346

failed

 

DMA per-cpu:

CPU 0: hi: 18, btch: 3 usd: 10

Active_anon:0 active_file:628 inactive_anon:0

inactive_file:2799 dirty:0 writeback:0 unstable:0

free:10681 slab:393 mapped:0 pagetables:0 bounce:0

DMA free:42724kB min:4096kB low:5120kB high:6144kB active_anon:0kB

inactive_anon

:0kB active_file:2512kB inactive_file:11196kB present:64004kB

pages_scanned:0 al

l_unreclaimable? no

lowmem_reserve[]: 0 0 0

DMA: 17*4kB 16*8kB 8*16kB 3*32kB 1*64kB 0*128kB 1*256kB 2*512kB 2*1024kB

1*2048k

B 1*4096kB 0*8192kB 2*16384kB 0*32768kB = 42724kB

3431 total pagecache pages

QuoteReplyEditDelete

 

 

2010-11-16 21:00:55     Re: httpd crash

Wojtek Skulski (UNITED STATES)

Message: 95884   

 

It turns out it is gnuplot. I added the stack check to gnuplot and it detected stack overflow. Then I removed the check and it plots. But it is a time bomb. At some point it corrupts memory and kills the httpd.

 

root:/home/httpd/cgi-bin> Application stack overflow

- Please increase the stack size of the application using elf2flt -s option,

   and/or reduce the stack use of the application.

Deferred Exception context

CURRENT PROCESS:

COMM=gnuplot PID=80

CPU = 0

QuoteReplyEditDelete

 

 

2010-11-16 23:14:25     Re: httpd crash

Wojtek Skulski (UNITED STATES)

Message: 95885   

 

I increased the stack to 2 megabytes using  elf2flt -s option and gnuplot would immediately trap on stack overflow. Then I did the same for httpd server and the CGI executables, and they would start crashing as well. Then I removed elf2flt -s option and I used the option FLTFLAGS and the crashing stopped. This seems to be in contradiction to the Wiki page   docs.blackfin.uclinux.org/doku.php?id=uclinux-dist:debugging_applications

 

Specifically, adding the following lines caused an immediate execution trap on stack check, no matter what was the stack size (I tried several choices for the 0x...... number):

 

CFLAGS += -mstack-check-l1

CFLAGS += -elf2flt="-s 0x800000"

 

The following lines produced executables which are not immediately trapping on stack overflow:

 

CFLAGS += -mstack-check-l1

FLTFLAGS = -s 0x80000

 

As far as I understand the Wiki, the above ways of specifying the stack size should be equivalent. But they are not. Morever, using the elf2flt -s option is recommended. But in my case it produces executables which trap as soon as they are executed.

QuoteReplyEditDelete

 

 

2010-11-17 01:04:49     Re: httpd crash

Mike Frysinger (UNITED STATES)

Message: 95886   

 

you can use flthdr to examine the result to make sure the settings are as you expect

Attachments

    Outcomes