2008-07-11 15:40:36     Page Cache Question

2008-07-11 15:40:36     Page Cache Question

David Kasper (UNITED STATES)

Message: 58700   

I am now running the 2008 uClinux kernel and streaming data to CompactFlash via the PCMCIA driver (actually my test program writes 5MB files, 500 bytes at a time, closes and performs sync between each save).  Concurrently I run top and watch the cached level rise until around 43,500K the system encounters a page allocation failure (see output below) apparently when sync is performed.  How can this issue be solved?  Is there a recommended limit that I should place on the page cache size?  Any other advice is appreciated.

David Kasper

Mem: 50736K used, 1252K free, 0K shrd, 144K buff, 44132K cached

Load average: 0.17 0.08 0.02

[7m  PID USER     STATUS   VSZ  PPID %CPU %MEM COMMAND [0m

   25 root     SW<        0     5  2.6  0.0 kblockd/0

  349 root     R        524   220  1.9  1.0 top

   39 root     SW         0     5  1.2  0.0 pdflush

  321 root     S       2346   220  0.5  4.4 raw

  329 root     S       2346   324  0.5  4.4 raw

   40 root     SW<        0     5  0.2  0.0 kswapd0

  325 root     R       2346   324  0.0  4.4 raw

  328 root     S       2346   324  0.0  4.4 raw

  327 root     S       2346   324  0.0  4.4 raw

  326 root     R       2346   324  0.0  4.4 raw

  324 root     S       2346   321  0.0  4.4 raw

  220 root     S        656     1  0.0  1.2 sh

    1 root     S        132     0  0.0  0.2 init

  174 root     S         68     1  0.0  0.1 inetd

  133 root     SW         0     1  0.0  0.0 mtdblockd

  147 root     SW<        0     5  0.0  0.0 pccardd

    4 root     SW<        0     1  0.0  0.0 khelper

    3 root     SW<        0     1  0.0  0.0 events/0

  178 root     SWN        0     1  0.0  0.0 jffs2_gcd_mtd2

  153 root     SW<        0     5  0.0  0.0 kpsmoused

   26 root     SW<        0     5  0.0  0.0 kseriod

Closing file

"Next is a printf performed by my application indicating return status from close"

Starting file close

File closed with status =

sync: page allocation failure. order:7, mode:0x40d0

Hardware Trace:

0 Target : <0x00004610> { _dump_stack + 0x0 }

   Source : <0x00030526> { ___alloc_pages + 0x17a }

1 Target : <0x00030526> { ___alloc_pages + 0x17a }

   Source : <0x0000c7fa> { _printk + 0x16 }

2 Target : <0x0000c7f6> { _printk + 0x12 }

   Source : <0x0000c684> { _vprintk + 0x1b0 }

3 Target : <0x0000c678> { _vprintk + 0x1a4 }

   Source : <0xffa0124e> { __common_int_entry + 0xd8 }

4 Target : <0xffa011ec> { __common_int_entry + 0x76 }

   Source : <0xffa00fb4> { _return_from_int + 0x58 }

5 Target : <0xffa00fb4> { _return_from_int + 0x58 }

   Source : <0xffa00f8a> { _return_from_int + 0x2e }

6 Target : <0xffa00f5c> { _return_from_int + 0x0 }

   Source : <0xffa011e8> { __common_int_entry + 0x72 }

7 Target : <0xffa011e6> { __common_int_entry + 0x70 }

   Source : <0xffa0057e> { _asm_do_IRQ + 0x7a }

8 Target : <0xffa00576> { _asm_do_IRQ + 0x72 }

   Source : <0x000106aa> { __local_bh_enable + 0x3e }

9 Target : <0x0001066c> { __local_bh_enable + 0x0 }

   Source : <0x00010924> { ___do_softirq + 0x94 }

10 Target : <0x0001091c> { ___do_softirq + 0x8c }

   Source : <0x000108fc> { ___do_softirq + 0x6c }

11 Target : <0x000108ee> { ___do_softirq + 0x5e }

   Source : <0x00010a18> { _tasklet_action + 0x7c }

12 Target : <0x00010a12> { _tasklet_action + 0x76 }

   Source : <0x000109ee> { _tasklet_action + 0x52 }

13 Target : <0x000109ec> { _tasklet_action + 0x50 }

   Source : <0x0001ab26> { ___rcu_process_callbacks + 0xb6 }

14 Target : <0x0001ab20> { ___rcu_process_callbacks + 0xb0 }

   Source : <0x0001aab6> { ___rcu_process_callbacks + 0x46 }

15 Target : <0x0001aab2> { ___rcu_process_callbacks + 0x42 }

   Source : <0x0001ab4e> { ___rcu_process_callbacks + 0xde }

Stack from 032e9cdc:

        00000001 0003052a 00000000 00000001 000040d0 011ca46c 00000007 000040d0

        00000000 00000080 00000000 000240d0 00000000 032e8000 032e8000 032e8000

        00000010 00000000 001bacc0 00031542 001a8f74 00000000 0025e8a0 00000020

        000040d0 00000001 00000010 00264440 00000000 00000000 000040d0 0025e8a0

        004e6640 000317f2 0004cfc4 0317f300 00000077 0000ffff 00000002 00000000

        00000000 0015e6c0 0002ee8e 0004cfc4 0002f2ba 00000000 0015e554 001703b8

Call Trace:

[<0005a1ec>] _load_flat_binary+0x60c/0xd10

[<00042d90>] _sys_getcwd+0x74/0x1d0

[<00004970>] _trap_c+0x2a4/0x2bc

[<0000a234>] _schedule_tail+0x50/0x9c

[<0000a230>] _schedule_tail+0x4c/0x9c

[<0000e130>] _do_wait+0x40/0x938

[<0003040a>] ___alloc_pages+0x5e/0x1fc

[<0001ffa5>] _do_adjtimex+0x539/0x554

[<0004cfc4>] _sys_tee+0x150/0x320

[<0000288c>] _do_rt_sigreturn+0x60/0x850

[<00034c60>] _fs_may_remount_ro+0x24/0x50

[<00042d90>] _sys_getcwd+0x74/0x1d0

[<00034c60>] _fs_may_remount_ro+0x24/0x50

[<00009803>] _sched_setscheduler+0x4f/0x270

[<000200d2>] _second_overflow+0x112/0x1e4

[<00004e93>] _set_gpio_inen+0x2b/0x70

[<000371c8>] _search_binary_handler+0x6c/0x20c

[<000384c8>] _do_execve+0xe8/0x1bc

[<00001c1e>] _sys_execve+0x2e/0x54

[<00001bf0>] _sys_execve+0x0/0x54

[<0015e520>] _inode_init_early+0x58/0x90

[<0015e380>] _vfs_caches_init_early+0x40/0x94

[<00008000>] _bfin_generic_error_mask_irq+0x3c/0x44

[<00004227>] _dump_bfin_regs+0x27f/0x33c

[<00016000>] ___kill_pgrp_info+0x48/0x54

Mem-info:

DMA per-cpu:

CPU    0: Hot: hi:   18, btch:   3 usd:   1   Cold: hi:    6, btch:   1 usd:   5

Active:7448 inactive:3408 dirty:91 writeback:0 unstable:0 free:362 slab:1708 mapped:0 pagetables:0

DMA free:1448kB min:928kB low:1160kB high:1392kB active:29792kB inactive:13632kB present:53848kB pages_scanned:0 all_unreclaimable? no

lowmem_reserve[]: 0 0

DMA: 6*4kB 6*8kB 4*16kB 3*32kB 7*64kB 2*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 1448kB

13568 pages of RAM

396 free pages

571 reserved pages

10552 pages shared

0 pages swap cached

Allocation of length 315332 from process 357 failed

DMA per-cpu:

CPU    0: Hot: hi:   18, btch:   3 usd:   1   Cold: hi:    6, btch:   1 usd:   5

Active:7448 inactive:3408 dirty:91 writeback:0 unstable:0 free:362 slab:1708 mapped:0 pagetables:0

DMA free:1448kB min:928kB low:1160kB high:1392kB active:29792kB inactive:13632kB present:53848kB pages_scanned:0 all_unreclaimable? no

lowmem_reserve[]: 0 0

DMA: 6*4kB 6*8kB 4*16kB 3*32kB 7*64kB 2*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB 0*8192kB 0*16384kB 0*32768kB = 1448kB

Unable to allocate RAM for process text/data, errno 12

Instruction fetch misaligned address violation

- Attempted misaligned instruction cache fetch. On a misaligned instruction fetch exception,

the return address provided in RETX is the destination address which is misaligned, rather than the address of the offending instruction.

CURRENT PROCESS:

COMM=sync PID=357

TEXT = 0x00000000-0x00000000  DATA = 0x00000000-0x00000000

BSS = 0x00000000-0x00000000   USER-STACK = 0x00000000

return address: 0x01f0142c; contents of [PC-16...PC+8]:

00a0 3038 e120 f000 0a07 1c07 e300 0a12

3210 43b8 63ff 9310 e801 0000 3007 0530 X

0010 0000 e800 0003 e300 0a04 3210 6130

RETE:  00000000  RETN: 032ea000  RETX: 01f0142c  RETS: 00000001

IPEND: 0030  SYSCFG: 0036

SEQSTAT: 0000002a    SP: 032e9f24

R0: ffffffff    R1: 005602a0    R2: 0056023c    R3: 0056023c

R4: 0056029c    R5: 0056028c    R6: 01f3bef1    R7: 01f258a8

P0: 0000000b    P1: 00560291    P2: 01f4341c    P3: 01f42898

P4: 005602a0    P5: 01f7ffcb    FP: 00000000

A0.w: 00004227    A0.x: 00000000    A1.w: 0000009f    A1.x: 00000000

LB0: 01f09755  LT0: 01f09752  LC0: 00000000

LB1: 01f2459d  LT1: 01f2459c  LC1: 00000000

B0: 0000001c  L0: 00000000  M0: 00000000  I0: 005601f4

B1: 00000000  L1: 00000000  M1: 00000000  I1: 00016000

B2: 00000000  L2: 00000000  M2: 00000000  I2: 00000000

B3: 00000000  L3: 00000000  M3: 00000000  I3: 0301fa08

USP: 01f7fb84   ASTAT: 02002000

DCPLB_FAULT_ADDR=01f7fb80

ICPLB_FAULT_ADDR=01f0142c

Hardware Trace:

0 Target : <0x000046cc> { _trap_c + 0x0 }

   Source : <0xffa00c48> { _exception_to_level5 + 0xb4 }

1 Target : <0xffa00b94> { _exception_to_level5 + 0x0 }

   Source : <0xffa00b92> { _ex_trap_c + 0x4e }

2 Target : <0xffa00b44> { _ex_trap_c + 0x0 }

   Source : <0xffa00ce8> { _trap + 0x28 }

3 Target : <0xffa00cc0> { _trap + 0x0 }

   Source : <0x01f0142a> [ sh + 0x142a ]

4 Target : <0x01f0141c> [ sh + 0x141c ]

   Source : <0x01f0284c> [ sh + 0x284c ]

5 Target : <0x01f0283c> [ sh + 0x283c ]

   Source : <0x01f01418> [ sh + 0x1418 ]

6 Target : <0x01f0140e> [ sh + 0x140e ]

   Source : <0xffa0124e> { __common_int_entry + 0xd8 }

7 Target : <0xffa011ec> { __common_int_entry + 0x76 }

   Source : <0xffa0148c> { _evt_system_call + 0x64 }

8 Target : <0xffa0148c> { _evt_system_call + 0x64 }

   Source : <0xffa00e70> { _system_call + 0xb8 }

9 Target : <0xffa00e6c> { _system_call + 0xb4 }

   Source : <0xffa00e5c> { _system_call + 0xa4 }

10 Target : <0xffa00e56> { _system_call + 0x9e }

   Source : <0xffa00e46> { _system_call + 0x8e }

11 Target : <0xffa00e34> { _system_call + 0x7c }

   Source : <0xffa00e54> { _system_call + 0x9c }

12 Target : <0xffa00e52> { _system_call + 0x9a }

   Source : <0xffa01a1e> { _schedule + 0x31e }

13 Target : <0xffa019cc> { _schedule + 0x2cc }

   Source : <0xffa00eea> { _new_old_task + 0x6 }

14 Target : <0xffa00ee4> { _new_old_task + 0x0 }

   Source : <0xffa00ee2> { _resume + 0x2e }

15 Target : <0xffa00eb4> { _resume + 0x0 }

   Source : <0xffa019c8> { _schedule + 0x2c8 }

Stack from 032e9f04:

        01f42898 ffa00c4c 0016c710 0016c710 0016c70c 0056023c 0056029c 0064d4ec

        01f0142c 00000030 0000002a 00000000 032ea000 01f0142c 01f0142c 00000001

        ffffffff 02002000 01f2459d 01f09755 01f2459c 01f09752 00000000 00000000

        0000009f 00000000 00004227 00000000 00000000 00000000 00000000 0000001c

        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

        0301fa08 00000000 00016000 005601f4 01f7fb84 00000000 01f7ffcb 005602a0

Call Trace:

357: Bus error

"Next is a printf performed by my application indicating return status from sync (0xFF00)"

0, invoking sync

Sync complete, status = 65280

Saving file /mnt/cf/test/Ccp-44 Id-A.tst, Size = 5000000.000000, Blk = 500

[H [JMem: 41764K used, 10224K free, 0K shrd, 136K buff, 34656K cached

Load average: 0.24 0.10 0.02

[7m  PID USER     STATUS   VSZ  PPID %CPU %MEM COMMAND [0m

  349 root     R        524   220  4.3  1.0 top

   40 root     DW<        0     5  3.2  0.0 kswapd0

  133 root     SW         0     1  0.7  0.0 mtdblockd

  329 root     S       2346   324  0.3  4.4 raw

  321 root     R       2346   220  0.1  4.4 raw

  327 root     S       2346   324  0.1  4.4 raw

  326 root     S       2346   324  0.1  4.4 raw

  328 root     S       2346   324  0.0  4.4 raw

  325 root     S       2346   324  0.0  4.4 raw

  324 root     S       2346   321  0.0  4.4 raw

  220 root     S        656     1  0.0  1.2 sh

    1 root     S        132     0  0.0  0.2 init

  174 root     S         68     1  0.0  0.1 inetd

   25 root     SW<        0     5  0.0  0.0 kblockd/0

   39 root     SW         0     5  0.0  0.0 pdflush

  147 root     SW<        0     5  0.0  0.0 pccardd

    4 root     SW<        0     1  0.0  0.0 khelper

    3 root     SW<        0     1  0.0  0.0 events/0

  178 root     SWN        0     1  0.0  0.0 jffs2_gcd_mtd2

  153 root     SW<        0     5  0.0  0.0 kpsmoused

   26 root     SW<        0     5  0.0  0.0 kseriod

[H [JMem: 29200K used, 22788K free, 0K shrd, 104K buff, 22252K cached

Load average: 0.30 0.11 0.03

QuoteReplyEditDelete

2008-07-11 16:46:19     Re: Page Cache Question

Robin Getz (UNITED STATES)

Message: 58702   

David:

Sonic had made some tweaks to the mem/page cache system on the branch to help with this - are you using the live svn branch, or the release snapshot?

-Robin

QuoteReplyEditDelete

2008-07-11 17:00:28     Re: Page Cache Question

David Kasper (UNITED STATES)

Message: 58703   

I am using the release.  I am new to SVN, tell me what should I select:

    Trunk

    branches/2008R1

    Or something else

Thanks,

David Kasper

QuoteReplyEditDelete

2008-07-12 01:12:01     Re: Page Cache Question

Robin Getz (UNITED STATES)

Message: 58708   

David:

Use the 2008R1 branch.

-Robin

QuoteReplyEditDelete

2008-07-13 23:40:44     Re: Page Cache Question

Sonic Zhang (CHINA)

Message: 58740   

Could you try "dd if=/dev/zero of=/mnt/cf bs=500 size=10000" on your CF card? Repeat 20 times. Can you replicate your bug?

QuoteReplyEditDelete

2008-07-16 18:06:32     Re: Page Cache Question

David Kasper (UNITED STATES)

Message: 58975   

Sonic,

I made a mistake as I was running the 2007 kernel when I experienced the page allocation failure.  So far the 2008 kernel with PCMCIA CompactFlash driver seems to be working fine (I tested 800 5MB files without any errors).

Thank you,

David Kasper