2011-08-17 09:39:43     BUG() with xenomai __ipipe_sync_root

Document created by Aaronwu Employee on Aug 27, 2013
Version 1Show Document
  • View in full screen mode

2011-08-17 09:39:43     BUG() with xenomai __ipipe_sync_root

Thorsten Pohlmann (GERMANY)

Message: 102967   

 

Hi!

 

We experience this BUG under heavy interrupt load in xenomai 2.4.7 (2009R1RC6) and the one from 2010R1 on our custom BF537 board.

 

Heavy means: two sports (DMA) interrupting every ~125us with slightly different external clocks, a uart rx/txing and some eth. The 125us irq triggers a higest prio rt-task which works for about 40us, so the chance for pending irqs is quite high.

Hardware Trace:

   0 Target : <0x00004830> { _dump_bfin_trace_buffer + 0x0 }

     Source : <0x00007230> { ___ipipe_sync_root + 0x78 } CALL pcrel

   1 Target : <0x00007230> { ___ipipe_sync_root + 0x78 }

     Source : <0x000071d4> { ___ipipe_sync_root + 0x1c } IF !CC JUMP

   2 Target : <0x000071b8> { ___ipipe_sync_root + 0x0 }

     Source : <0xffa008d8> { _system_call + 0xb4 }

   3 Target : <0xffa008aa> { _system_call + 0x86 }

     Source : <0x000209e0> { _sys_clock_gettime + 0x68 } RTS

   4 Target : <0x000209d8> { _sys_clock_gettime + 0x60 }

     Source : <0xffa015ce> { _memcpy + 0x2e }

   5 Target : <0xffa015b4> { _memcpy + 0x14 }

     Source : <0xffa015ac> { _memcpy + 0xc }

   6 Target : <0xffa015a0> { _memcpy + 0x0 }

     Source : <0x000209d4> { _sys_clock_gettime + 0x5c } CALL pcrel

   7 Target : <0x000209ca> { _sys_clock_gettime + 0x52 }

     Source : <0xffa00226> { __access_ok + 0xc6 }

   8 Target : <0xffa00212> { __access_ok + 0xb2 }

     Source : <0xffa0018c> { __access_ok + 0x2c }

   9 Target : <0xffa00160> { __access_ok + 0x0 }

     Source : <0x000209c6> { _sys_clock_gettime + 0x4e } CALL pcrel

  10 Target : <0x000209be> { _sys_clock_gettime + 0x46 }

     Source : <0x0002046c> { _posix_ktime_get_ts + 0x10 } RTS

  11 Target : <0x00020466> { _posix_ktime_get_ts + 0xa }

     Source : <0x000246ce> { _ktime_get_ts + 0x4a } RTS

  12 Target : <0x000246c8> { _ktime_get_ts + 0x44 }

     Source : <0x0001481e> { _set_normalized_timespec + 0x46 } RTS

  13 Target : <0x00014816> { _set_normalized_timespec + 0x3e }

     Source : <0x00014804> { _set_normalized_timespec + 0x2c } IF !CC JUMP

  14 Target : <0x000147d8> { _set_normalized_timespec + 0x0 }

     Source : <0x000246c4> { _ktime_get_ts + 0x40 } CALL pcrel

  15 Target : <0x000246ac> { _ktime_get_ts + 0x28 }

     Source : <0x00026c1e> { _getnstimeofday + 0xde } RTS

BUG: failure at arch/blackfin/kernel/ipipe.c:319/__ipipe_sync_root()!

Kernel panic - not syncing: BUG!

 

Its the BUG_ON(irqs_disabled()) at the beginnig of ipipe.c:asmlinkage void __ipipe_sync_root(void).

 

This BUG is reproducable within 15min to 2 hours under this load and always shows this call-stack.

 

We applied the "blackfin/ipipe: fix deferred pipeline sync for the root stage" patch, but we think that is not the whole thing?!

 

In ipipe.c, function "_system_call" there is the BUG-triggering call:

 

 

.Lsyscall_resched:

#ifdef CONFIG_IPIPE

    cc = BITTST(r7, TIF_IRQ_SYNC);

    if !cc jump .Lsyscall_no_irqsync;

    [--sp] = reti;

    r0 = [sp++];

    SP += -12;

    call ___ipipe_sync_root;

    SP += 12;

    jump .Lresume_userspace_1;

.Lsyscall_no_irqsync:

#endif

    cc = BITTST(r7, TIF_NEED_RESCHED);

 

 

 

We do not quite understand how the program flow can get there with disabled irqs...

 

Any suggestions?

 

 

 

Regards, Thorsten

 

 

TranslateQuoteReplyEditDelete

 

 

2011-08-18 01:47:55     Re: BUG() with xenomai __ipipe_sync_root

Simon Brewer (AUSTRALIA)

Message: 102971   

 

Hi Thorsten,

 

very strange.  Can you please send over a copy of all the registers when the BUG() happens?  And the value in the IMASK MMR.

 

Simon

QuoteReplyEditDelete

 

 

2011-08-18 05:46:48     Re: BUG() with xenomai __ipipe_sync_root

Thorsten Pohlmann (GERMANY)

Message: 102993   

 

Hi!

 

Thanks for your response!

 

I modified ipie.c this way:

 

 

 

...

 

static unsigned long s_ipend, s_imask, s_ilat;

 

asmlinkage void __ipipe_sync_root(void)

{

    void (*irq_tail_hook)(void) = (void (*)(void))__ipipe_irq_tail_hook;

    unsigned long flags;

 

    unsigned long ipend = 0, imask = 0, ilat = 0;

 

    ipend = bfin_read_IPEND();

    ilat = bfin_read_ILAT();

    imask = bfin_read_IMASK();

 

    if(unlikely(irqs_disabled()))

    {

        printk(KERN_EMERG "former BUG()\n");

        printk("ipend %08lx, imask %08lx, ilat %08lx\n",ipend,imask,ilat);

        dump_bfin_trace_buffer();

        dump_stack();

        panic("BUG");

    }

 

    //BUG_ON(irqs_disabled());

 

    local_irq_save_hw(flags);

 

    if(ipend != s_ipend || imask != s_imask || ilat != s_ilat)

    {

        int c = s_ilat != ilat;

 

        s_ipend = ipend;

        s_imask = imask;

        s_ilat = ilat;

 

        if(c)

        {

            printk("__ipipe_sync_root context change: ipend %08lx, imask %08lx, ilat %08lx\n",ipend,imask,ilat);

        }

    }

 

...

 

 

 

So you can see changes in ILAT during normal work.

 

The result:

 

...

 

__ipipe_sync_root context change: ipend 00008000, imask 0000001f, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

__ipipe_sync_root context change: ipend 00008000, imask 0000001f, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

__ipipe_sync_root context change: ipend 00008000, imask 0000001f, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

__ipipe_sync_root context change: ipend 00008000, imask 0000001f, ilat 00000040

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

former BUG()

ipend 00008000, imask 0000ffff, ilat 00000000

Hardware Trace:

   0 Target : <0x00004830> { _dump_bfin_trace_buffer + 0x0 }

     Source : <0x000072cc> { ___ipipe_sync_root + 0x114 } CALL pcrel

   1 Target : <0x000072cc> { ___ipipe_sync_root + 0x114 }

     Source : <0x000114c6> { _printk + 0x66 } RTS

   2 Target : <0x000114bc> { _printk + 0x5c }

     Source : <0x00011ef4> { _vprintk + 0x148 } RTS

   3 Target : <0x00011eec> { _vprintk + 0x140 }

     Source : <0x00033a2c> { ___ipipe_restore_root + 0x30 } RTS

   4 Target : <0x000339fc> { ___ipipe_restore_root + 0x0 }

     Source : <0x00011ee8> { _vprintk + 0x13c } CALL pcrel

   5 Target : <0x00011ee0> { _vprintk + 0x134 }

     Source : <0x0001203a> { _vprintk + 0x28e } JUMP.S

   6 Target : <0x0001203a> { _vprintk + 0x28e }

     Source : <0x000118c0> { _release_console_sem + 0x1d8 } RTS

   7 Target : <0x000118ba> { _release_console_sem + 0x1d2 }

     Source : <0x00010f52> { _wake_up_klogd + 0x22 } RTS

   8 Target : <0x00010f4e> { _wake_up_klogd + 0x1e }

     Source : <0x00010f40> { _wake_up_klogd + 0x10 } IF !CC JUMP

   9 Target : <0x00010f30> { _wake_up_klogd + 0x0 }

     Source : <0x000118b6> { _release_console_sem + 0x1ce } CALL pcrel

  10 Target : <0x000118b6> { _release_console_sem + 0x1ce }

     Source : <0x000118ac> { _release_console_sem + 0x1c4 } IF !CC JUMP

  11 Target : <0x000118aa> { _release_console_sem + 0x1c2 }

     Source : <0x00033a2c> { ___ipipe_restore_root + 0x30 } RTS

  12 Target : <0x000339fc> { ___ipipe_restore_root + 0x0 }

     Source : <0x000118a6> { _release_console_sem + 0x1be } CALL pcrel

  13 Target : <0x0001189e> { _release_console_sem + 0x1b6 }

     Source : <0x000251d4> { _up + 0x54 } RTS

  14 Target : <0x000251ce> { _up + 0x4e }

     Source : <0x00033a2c> { ___ipipe_restore_root + 0x30 } RTS

  15 Target : <0x000339fc> { ___ipipe_restore_root + 0x0 }

     Source : <0x000251ca> { _up + 0x4a } CALL pcrel

Hardware Trace:

   0 Target : <0x00004b1c> { _dump_stack + 0x0 }

     Source : <0x000072d0> { ___ipipe_sync_root + 0x118 } CALL pcrel

   1 Target : <0x000072d0> { ___ipipe_sync_root + 0x118 }

     Source : <0x000049f4> { _dump_bfin_trace_buffer + 0x1c4 } RTS

Stack info:

BUG: sleeping function called from invalid context at kernel/fork.c:466

in_atomic(): 0, irqs_disabled(): 1, pid: 218, name: uDaksAlert

Hardware Trace:

Stack info:

SP: [0x0360bbdc] <0x0360bbdc> /* kernel dynamic memory */

FP: (0x0360bbe0)

Memory from 0x0360bbd0 to 0360c000

0360bbd0: 0360bbf4  0360bbdc <0000bae6>[007ecde0](0360bc04)<0000baea> 007ecde0  0360a000

0360bbf0: 000000a0  00000000  00000001  000000da  0114aea8 (0360bc1c)<0000f31c> 0360bed4

0360bc10: 00000001  0000001f  0360bc28 (0360bdd8)<00003c2a> 0315e4e0  0365fc38  0360bc74

0360bc30: 002afe42  0360bdcc  0360bd4c  ffc00414  002d5c44  0000000c  0000001f  0000000c

0360bc50: 002d5c44  ffffc000  0000001e  00000000  00000000  00000000  00000000  0000000c

0360bc70: 00000006  0360bc8c <00179f00> 0000000c  00000001  0000000c  0360bc9c  0360bcbc

0360bc90:<0017d7d4> 002f4518  002ded10  002ded0c  00000001  0360bcbc <0017d7da> 00000000

0360bcb0: 00000001  0360bcd4  0017d5e4  0360bcd4 <00010e0c> 0000000c  0360bd24  0000314c

0360bcd0: 002ded34  0360bd00 <00010e72> 002ded0c  002ded34  002e22e0  071a719b  071a719b

0360bcf0: 0360bd04 <000251ce> 071a7149  00003fff  0360bd24  0360bd24  0360bd24 <000118ba>

0360bd10: 002b4cb8  0360bd28 <00025230> 00000400  0000001f  0360bda8 <0001203a> 002b4cb8

0360bd30: 002e2dc4  002b4cc7  00000005  0000000f  0360bda8 <00011eec> 001608d8  00a62300

0360bd50: 001608c8  0360be00  0360bd60  0000001f  00000000  00000000  00000000  00000000

0360bd70: 00000000  00000004  00000000  0000001b  00000000  00000000  00000000  00000000

0360bd90: 00000000  00000000  00000000  00000001  0021f384  00000000  0360bde0 <000114bc>

0360bdb0: 002e22d0  002e2dc4  0342cfc8  00253ac4 <00008000> 0360c000  0000fffe  00000001

0360bdd0: 00000013 <00008000>(0360bea4)<000045d6> 0360be0c  00000004  0342cfc8  00000013

0360bdf0:<00008000> 0360c000  0000fffe  0360be20  0360bed4  00000001  3078303c  34303030

0360be10: 3e346639  5f207b20  706d7564  6966625f  72745f6e  5f656361  66667562  2b207265

0360be30: 31783020  7d203463  2b207000  34783020  007d2061  5f65726f  746f6f72  30202b20

0360be50: 7d203078  00000000  0360c000  00a4fa2e  0360be74  002ba704  00000000  00000000

0360be70: 0360a000  0360a000 <0006efde> 00a62366  00000008  00000000  00003f67  0360bec4

0360be90:<0006f550> 0360becc <000114bc> 002e22d0  002e0010 (0360bed8)<00004b48> ffe06000

0360beb0: 00000004  0342cfc8  00000013 <00008000> 0000ffff  0000fffe <000072d0> 0360bed4

0360bed0: 00000004  0342cfc8 (0360bef8)<000072d4> 0003cb48  00000000  0360beec <00008000>

0360bef0: 0000ffff  00000000 (00000000) ffa008dc  0360a000  00000004  00000080  ffffe000

0360bf10: 02959770  0000fffe  0365fccc  0000000c  00033daa  00a4fa2e <00008000> 00000000

0360bf30: 00000000  0360c000  00a4fa2e  00a4fa2e <00538914> ffa00eba  02003025  022b833f

0360bf50: 00a62367  022b8332  00a62366  00000008  00000000  00003f67  00000000  04d00000

0360bf70: 00000000  00000000  00000000  00000000  00000000  00000000  00000000  00000000

0360bf90: 00000000  00000000  00000000  00000000  ffffffe8  00000000  00000000  00a7aba4

0360bfb0: 00000000  0365fc38  0365fc44  0290f5e0  02912e98  0342cfc8  0365fe24  00a4fa24

0360bfd0: 00000004  03421388  00000023  02959770  0000000c  0365fccc  0000000c  02959770

0360bff0: 0000000c  00000023  00000004  00000006

Return addresses in stack:

    address : <0x0000bae6> { ___might_sleep + 0xba }

   frame  1 : <0x0000baea> { ___might_sleep + 0xbe }

   frame  2 : <0x0000f31c> { _mmput + 0x18 }

   frame  3 : <0x00003c2a> { _decode_address + 0x136 }

    address : <0x00179f00> { _uart_console_write + 0x44 }

    address : <0x0017d7d4> { _bfin_serial_console_write + 0x6c }

    address : <0x0017d7da> { _bfin_serial_console_write + 0x72 }

    address : <0x00010e0c> { ___call_console_drivers + 0x48 }

    address : <0x00010e72> { __call_console_drivers + 0x56 }

    address : <0x000251ce> { _up + 0x4e }

    address : <0x000118ba> { _release_console_sem + 0x1d2 }

    address : <0x00025230> { _down_trylock + 0x40 }

    address : <0x0001203a> { _vprintk + 0x28e }

    address : <0x00011eec> { _vprintk + 0x140 }

    address : <0x000114bc> { _printk + 0x5c }

    address : <0x00008000> { _module_arch_cleanup + 0x30 }

    address : <0x00008000> { _module_arch_cleanup + 0x30 }

   frame  4 : <0x000045d6> { _show_stack + 0x32 }

    address : <0x00008000> { _module_arch_cleanup + 0x30 }

    address : <0x0006efde> { _rw_verify_area + 0x6a }

    address : <0x0006f550> { _vfs_write + 0xac }

    address : <0x000114bc> { _printk + 0x5c }

   frame  5 : <0x00004b48> { _dump_stack + 0x2c }

    address : <0x00008000> { _module_arch_cleanup + 0x30 }

    address : <0x000072d0> { ___ipipe_sync_root + 0x118 }

   frame  6 : <0x000072d4> { ___ipipe_sync_root + 0x11c }

    address : <0x00008000> { _module_arch_cleanup + 0x30 }

    address : <0x00008000> { _module_arch_cleanup + 0x30 }

    address : <0x00538914> [ /lib/libpthread-0.9.29.so + 0x8914 ]

SP: [0x0360bed4] <0x0360bed4> /* kernel dynamic memory */

FP: (0x0360bed8)

Memory from 0x0360bed0 to 0360c000

0360bed0: 00000004 [0342cfc8](0360bef8)<000072d4> 0003cb48  00000000  0360beec <00008000>

0360bef0: 0000ffff  00000000 (00000000) ffa008dc  0360a000  00000004  00000080  ffffe000

0360bf10: 02959770  0000fffe  0365fccc  0000000c  00033daa  00a4fa2e <00008000> 00000000

0360bf30: 00000000  0360c000  00a4fa2e  00a4fa2e <00538914> ffa00eba  02003025  022b833f

0360bf50: 00a62367  022b8332  00a62366  00000008  00000000  00003f67  00000000  04d00000

0360bf70: 00000000  00000000  00000000  00000000  00000000  00000000  00000000  00000000

0360bf90: 00000000  00000000  00000000  00000000  ffffffe8  00000000  00000000  00a7aba4

0360bfb0: 00000000  0365fc38  0365fc44  0290f5e0  02912e98  0342cfc8  0365fe24  00a4fa24

0360bfd0: 00000004  03421388  00000023  02959770  0000000c  0365fccc  0000000c  02959770

0360bff0: 0000000c  00000023  00000004  00000006

Return addresses in stack:

   frame  1 : <0x000072d4> { ___ipipe_sync_root + 0x11c }

    address : <0x00008000> { _module_arch_cleanup + 0x30 }

    address : <0x00008000> { _module_arch_cleanup + 0x30 }

    address : <0x00538914> [ /lib/libpthread-0.9.29.so + 0x8914 ]

Kernel panic - not syncing: BUG

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

__ipipe_sync_root context change: ipend 00008000, imask 0000001f, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

 

 

 

Regards, Thorsten

TranslateQuoteReplyEditDelete

 

 

2011-08-18 07:06:16     Re: BUG() with xenomai __ipipe_sync_root

Simon Brewer (AUSTRALIA)

Message: 102995   

 

Can you also dump out IMASK as shown below?

 

imask = bfin_read_IMASK();

 

    if(unlikely(irqs_disabled()))

    {

        printk(KERN_EMERG "former BUG()\n");

        printk("ipend %08lx, imask %08lx, ilat %08lx\n",ipend,imask,ilat);

 

        imask = bfin_read_IMASK();  // add this

        printk("ipend %08lx, imask2 %08lx, ilat %08lx\n",ipend,imask,ilat); // add this

 

        dump_bfin_trace_buffer();

        dump_stack();

        panic("BUG");

QuoteReplyEditDelete

 

 

2011-08-18 08:27:49     Re: BUG() with xenomai __ipipe_sync_root

Thorsten Pohlmann (GERMANY)

Message: 102997   

 

Hi!

 

 

 

I additionally did that:

 

imask = bfin_read_IMASK();

 

    if(unlikely(irqs_disabled()))

    {

        unsigned long imask2 = bfin_read_IMASK();

 

        printk(KERN_EMERG "former BUG()\n");

        printk("ipend %08lx, imask %08lx/%08lx, ilat %08lx\n",ipend,imask,imask2,ilat);

 

        imask = bfin_read_IMASK();  // add this

        printk("ipend %08lx, imask2 %08lx, ilat %08lx\n",ipend,imask,ilat); // add this

 

The next thing i do is another trace with the dump_trace() and dump_stack() in front of the printks,

 

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

__ipipe_sync_root context change: ipend 00008000, imask 0000001f, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

__ipipe_sync_root context change: ipend 00008000, imask 0000001f, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

__ipipe_sync_root context change: ipend 00008000, imask 0000001f, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

former BUG()

ipend 00008000, imask 0000ffff/0000ffff, ilat 00000000

ipend 00008000, imask2 0000ffff, ilat 00000000

Hardware Trace:

   0 Target : <0x00004830> { _dump_bfin_trace_buffer + 0x0 }

     Source : <0x000072f0> { ___ipipe_sync_root + 0x138 } CALL pcrel

   1 Target : <0x000072f0> { ___ipipe_sync_root + 0x138 }

     Source : <0x000114ea> { _printk + 0x66 } RTS

   2 Target : <0x000114e0> { _printk + 0x5c }

     Source : <0x00011f18> { _vprintk + 0x148 } RTS

   3 Target : <0x00011f10> { _vprintk + 0x140 }

     Source : <0x00033a50> { ___ipipe_restore_root + 0x30 } RTS

   4 Target : <0x00033a20> { ___ipipe_restore_root + 0x0 }

     Source : <0x00011f0c> { _vprintk + 0x13c } CALL pcrel

   5 Target : <0x00011f04> { _vprintk + 0x134 }

     Source : <0x0001205e> { _vprintk + 0x28e } JUMP.S

   6 Target : <0x0001205e> { _vprintk + 0x28e }

     Source : <0x000118e4> { _release_console_sem + 0x1d8 } RTS

   7 Target : <0x000118de> { _release_console_sem + 0x1d2 }

     Source : <0x00010f76> { _wake_up_klogd + 0x22 } RTS

   8 Target : <0x00010f72> { _wake_up_klogd + 0x1e }

     Source : <0x00010f64> { _wake_up_klogd + 0x10 } IF !CC JUMP

   9 Target : <0x00010f54> { _wake_up_klogd + 0x0 }

     Source : <0x000118da> { _release_console_sem + 0x1ce } CALL pcrel

  10 Target : <0x000118da> { _release_console_sem + 0x1ce }

     Source : <0x000118d0> { _release_console_sem + 0x1c4 } IF !CC JUMP

  11 Target : <0x000118ce> { _release_console_sem + 0x1c2 }

     Source : <0x00033a50> { ___ipipe_restore_root + 0x30 } RTS

  12 Target : <0x00033a20> { ___ipipe_restore_root + 0x0 }

     Source : <0x000118ca> { _release_console_sem + 0x1be } CALL pcrel

  13 Target : <0x000118c2> { _release_console_sem + 0x1b6 }

     Source : <0x000251f8> { _up + 0x54 } RTS

  14 Target : <0x000251f2> { _up + 0x4e }

     Source : <0x00033a50> { ___ipipe_restore_root + 0x30 } RTS

  15 Target : <0x00033a20> { ___ipipe_restore_root + 0x0 }

     Source : <0x000251ee> { _up + 0x4a } CALL pcrel

Hardware Trace:

   0 Target : <0x00004b1c> { _dump_stack + 0x0 }

     Source : <0x000072f4> { ___ipipe_sync_root + 0x13c } CALL pcrel

   1 Target : <0x000072f4> { ___ipipe_sync_root + 0x13c }

     Source : <0x000049f4> { _dump_bfin_trace_buffer + 0x1c4 } RTS

Stack info:

BUG: sleeping function called from invalid context at kernel/fork.c:466

in_atomic(): 0, irqs_disabled(): 1, pid: 218, name: uDaksAlert

Hardware Trace:

Stack info:

SP: [0x0360dbd8] <0x0360dbd8> /* kernel dynamic memory */

FP: (0x0360dbdc)

Memory from 0x0360dbd0 to 0360e000

0360dbd0: 0360dbd8 <0000bb0a>[007ecde0](0360dc00)<0000bb0e> 007ecde0  0360c000  0360dd48

0360dbf0: 00000000  00000001  000000da  03608ea8 (0360dc18)<0000f340> 0360ded0 <00003b2a>

0360dc10: 0360de08  0003cb6c (0360ddd4)<00003c2a> 0315e4e0  0365fc38  0360dc70  002b0562

0360dc30: 0360ddc8  0360dd48  ffc00400  002d5c44  0000000c  0000001f  0000000c  002d5c44

0360dc50: ffffc000  0000001e  00000000  00000000  00000000  00000000  0000000c  00000006

0360dc70: 0360dc88 <00179f24> 0000000c  00a4d284  00012000 <00011e54> 0360dcb8 <0017d7f8>

0360dc90: 002f4518  002ded10  002ded0c  00000001  0360dcb8 <0017d7fe> 00000000  00000000

0360dcb0: 00000000  0017d608  0360dcd0 <00010e30> 0000000c  0000001b  00000000  00000000

0360dcd0: 0360dcfc <00010e96> 002ded0c  002ded34  002e22e0  0807f8bb  0807f8bb  0360dd00

0360dcf0:<000251f2> 00000003  0365fc38  0360dd20  0360dd20  0360dd20 <000118de> 0360dd5c

0360dd10: 0360dd24 <00025254> 00000004  0000001f  0360dda4 <0001205e> 002b4cb8  002e2dc4

0360dd30: 002b4cc7  00000005  0000000f  0360dda4 <00011f10><0001205e> 6d756400  66625f70

0360dd50: 0360ddfc  0360dd5c  0000001f  00726566 <00011f10> 001608fc  0070755f  65706970

0360dd70: 0360de1c  0360dd7c  0000001f  00000074  00000000  00000000  00000000  00000000

0360dd90: 00000000  00000000  0000001b  00000000  00000000  0360dddc <000114e0> 002e22d0

0360ddb0: 002e2dc4  0342cfc8  00253aa8  00000000  0360e000  0000ffff  0027db54  000002ec

0360ddd0: 0360dde4 (0360dea0)<000045d6> 0360de08  0003cb6c  0342cfc8  00000013  00000000

0360ddf0: 0360e000  0000ffff  0360de1c  0360ded0  00000001  3078303c  34303030  3e346639

0360de10: 5f207b20  706d7564  6966625f  72745f6e  5f656361  66667562  2b207265  31783020

0360de30: 7d203463  2b207000  34783020  007d2061  5f65726f  746f6f72  30202b20  7d203078

0360de50: 79000000  00000090  002ba108  7952ac03  00000090  0360de94 <00033aa2> 0360de94

0360de70: 0360de88 <00026c2a> 00000000  002e2dc0  002b7ca0  00000000  0360deb8 <0002474c>

0360de90: 0360dec8 <000114e0> 002e22d0  002e0010 (0360ded4)<00004b48> ffe06000  0003cb6c

0360deb0: 0342cfc8  00000013  00000000  00008000  0000ffff <000072f4> 0360ded0  0003cb6c

0360ded0: 0342cfc8 (0360def8)<000072f8> ffe02104  0000ffff <00020a6a> 00008000  0000ffff

0360def0: 00000000  00000000 (00000000) ffa008dc  0360c000  0000010a  00000080  ffffe000

0360df10: 0365fc94  0000fffe  00000001  20100805  00a4d284  00a4d284  00008000  00002000

0360df30: 00000000  0360e000  00a4d284  00a4d284 <022bd8dc> ffa00eba  02003004  022b833f

0360df50: 00a62367  022b8332  00a62366  00000008  00000000  0000209b  00000000  04100000

0360df70: 00000000  00000000  00000000  00000000  00000000  00000000  00000000  00000000

0360df90: 00000000  00000000  00000000  00000000  ffffffe8  00000000  00000000  00a7aba4

0360dfb0: 00000000  0365fc38  0365fc44  0365fc94  02912e98  0342cfc8  03640020  00a4d278

0360dfd0: 0000010a  00000009  0295a768  0365fc94  20100805  00000001  0000097a  0365fc5c

0360dff0: 00000000  00000001  0000010a  00000006

Return addresses in stack:

    address : <0x0000bb0a> { ___might_sleep + 0xba }

   frame  1 : <0x0000bb0e> { ___might_sleep + 0xbe }

   frame  2 : <0x0000f340> { _mmput + 0x18 }

    address : <0x00003b2a> { _decode_address + 0x36 }

   frame  3 : <0x00003c2a> { _decode_address + 0x136 }

    address : <0x00179f24> { _uart_console_write + 0x44 }

    address : <0x00011e54> { _vprintk + 0x84 }

    address : <0x0017d7f8> { _bfin_serial_console_write + 0x6c }

    address : <0x0017d7fe> { _bfin_serial_console_write + 0x72 }

    address : <0x00010e30> { ___call_console_drivers + 0x48 }

    address : <0x00010e96> { __call_console_drivers + 0x56 }

    address : <0x000251f2> { _up + 0x4e }

    address : <0x000118de> { _release_console_sem + 0x1d2 }

    address : <0x00025254> { _down_trylock + 0x40 }

    address : <0x0001205e> { _vprintk + 0x28e }

    address : <0x00011f10> { _vprintk + 0x140 }

    address : <0x0001205e> { _vprintk + 0x28e }

    address : <0x00011f10> { _vprintk + 0x140 }

    address : <0x000114e0> { _printk + 0x5c }

   frame  4 : <0x000045d6> { _show_stack + 0x32 }

    address : <0x00033aa2> { ___ipipe_dispatch_wired_nocheck + 0x46 }

    address : <0x00026c2a> { _getnstimeofday + 0x4a }

    address : <0x0002474c> { _ktime_get_ts + 0x28 }

    address : <0x000114e0> { _printk + 0x5c }

   frame  5 : <0x00004b48> { _dump_stack + 0x2c }

    address : <0x000072f4> { ___ipipe_sync_root + 0x13c }

   frame  6 : <0x000072f8> { ___ipipe_sync_root + 0x140 }

    address : <0x00020a6a> { _sys_clock_gettime + 0x52 }

    address : <0x022bd8dc> [ /tetronik/uDaksAlert + 0x2bd8dc ]

SP: [0x0360ded0] <0x0360ded0> /* kernel dynamic memory */

FP: (0x0360ded4)

Memory from 0x0360ded0 to 0360e000

0360ded0:[0342cfc8](0360def8)<000072f8> ffe02104  0000ffff <00020a6a> 00008000  0000ffff

0360def0: 00000000  00000000 (00000000) ffa008dc  0360c000  0000010a  00000080  ffffe000

0360df10: 0365fc94  0000fffe  00000001  20100805  00a4d284  00a4d284  00008000  00002000

0360df30: 00000000  0360e000  00a4d284  00a4d284 <022bd8dc> ffa00eba  02003004  022b833f

0360df50: 00a62367  022b8332  00a62366  00000008  00000000  0000209b  00000000  04100000

0360df70: 00000000  00000000  00000000  00000000  00000000  00000000  00000000  00000000

0360df90: 00000000  00000000  00000000  00000000  ffffffe8  00000000  00000000  00a7aba4

0360dfb0: 00000000  0365fc38  0365fc44  0365fc94  02912e98  0342cfc8  03640020  00a4d278

0360dfd0: 0000010a  00000009  0295a768  0365fc94  20100805  00000001  0000097a  0365fc5c

0360dff0: 00000000  00000001  0000010a  00000006

Return addresses in stack:

   frame  1 : <0x000072f8> { ___ipipe_sync_root + 0x140 }

    address : <0x00020a6a> { _sys_clock_gettime + 0x52 }

    address : <0x022bd8dc> [ /tetronik/uDaksAlert + 0x2bd8dc ]

Kernel panic - not syncing: BUG

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000200

__ipipe_sync_root context change: ipend 00008000, imask 0000ffff, ilat 00000000

__ipipe_sync_root context change: ipend 00008000, imask 0000001f, ilat 00000200

TranslateQuoteReplyEditDelete

 

 

2011-08-18 08:59:03     Re: BUG() with xenomai __ipipe_sync_root

Thorsten Pohlmann (GERMANY)

Message: 102998   

 

PS:

 

Since IMASK is completely open:

 

The definitions from our kernel:

 

#define irqs_disabled()        __ipipe_test_root()

 

static inline unsigned long __ipipe_test_root(void)

{

    const unsigned long *p = &__ipipe_root_status;

    return test_bit(0, p);

}

TranslateQuoteReplyEditDelete

 

 

2011-08-19 00:15:02     Re: BUG() with xenomai __ipipe_sync_root

Simon Brewer (AUSTRALIA)

Message: 103017   

 

Hi Thorsten,

 

which release are you using?  I can't find that code in my tree.

 

Simon

QuoteReplyEditDelete

 

 

2011-08-19 03:28:35     Re: BUG() with xenomai __ipipe_sync_root

Thorsten Pohlmann (GERMANY)

Message: 103026   

 

These test are done with 2009R1 RC6 Kernel 2.6.28.10-ADI-2009R1

 

Xenomai 2.4.7 (Andalusia)

 

I-Pipe 1.10.00

 

BF537 0.3

TranslateQuoteReplyEditDelete

 

 

2011-08-19 03:28:37     Re: BUG() with xenomai __ipipe_sync_root

Thorsten Pohlmann (GERMANY)

Message: 103027   

 

These test are done with 2009R1 RC6 Kernel 2.6.28.10-ADI-2009R1

 

Xenomai 2.4.7 (Andalusia)

 

I-Pipe 1.10.00

 

BF537 0.3

TranslateQuoteReplyEditDelete

 

 

2011-08-19 04:05:27     Re: BUG() with xenomai __ipipe_sync_root

Aaron Wu (CHINA)

Message: 103029   

 

is there a way for us to reproduce this issue on our side with an ADI board, without much dependence on your specific hardware so we can have a try on our side?

QuoteReplyEditDelete

 

 

2011-08-19 09:28:13     Re: BUG() with xenomai __ipipe_sync_root

Thorsten Pohlmann (GERMANY)

Message: 103041   

 

Ok, i try to create an easy module which causes this bug.

 

 

 

regards, T

TranslateQuoteReplyEditDelete

 

 

2011-08-22 10:45:24     Re: BUG() with xenomai __ipipe_sync_root

Thorsten Pohlmann (GERMANY)

Message: 103080   

 

Hi!

 

I tried hard today to build a simple bug reproducing app, but i did not succed (yet). It seems its caused by a very specific history or simultanously occcuring irqs or whatsoever.

 

Do you have any hints for me where i can have a look at? How can adeos/xenomai get into a state where prior to falling back to usermode scheduling is locked?

 

 

 

Our system roughly works like this:

 

A rtdm-module registers Irqs for both sport-dmas and a gpio and registers a rtdm-device . The iqr-handlers set a rt-event (rtdm_event_signal())

 

A Xenomai user-appl calls ioctls in the rtdm-module which wait for the irq corresponding events.

 

The xenomai user appl may then write some data into a rt_pipe, which is read by a "ordinary" Linux-appl.

 

 

 

Any idea would be great!

 

 

 

regards, Thorsten

TranslateQuoteReplyEditDelete

 

 

2011-08-25 10:01:48     Re: BUG() with xenomai __ipipe_sync_root

Thorsten Pohlmann (GERMANY)

Message: 103120   

 

Does nobody have a hint?

 

 

 

regards,

TranslateQuoteReplyEditDelete

 

 

2011-08-26 00:51:45     Re: BUG() with xenomai __ipipe_sync_root

Sonic Zhang (CHINA)

Message: 103126   

 

Look like the IMASK is not cleared properly in some interrupt handler.

 

Do you call hard_local_irq_disable() or hard_local_irq_save() in any of your drivers?

QuoteReplyEditDelete

 

 

2011-08-29 03:03:54     Re: BUG() with xenomai __ipipe_sync_root

Thorsten Pohlmann (GERMANY)

Message: 103178   

 

Hi!

 

No, our handlers are quite simple:

 

 

 

Two identical handlers for Sport0/1 like this:

 

static int sport1_handler(rtdm_irq_t *irq_handle)

{

    unsigned int rx_stat;

  

  rx_stat = get_dma_curr_irqstat(CH_SPORT1_RX);

 

  if((rx_stat & 3) == 0)

      return(RTDM_IRQ_NONE);

 

    clear_dma_irqstat(CH_SPORT1_RX);

  

    sport1_frame ^= 1;

  

    rtdm_event_signal(&sport1_evt);

  

    return RTDM_IRQ_HANDLED;

}

 

Same for a GPIO:

 

 

static int pg14_rirq(rtdm_irq_t *irq_handle)

{

    pg14_disabled = 1;

  

    rtdm_event_signal(&pg14_evt);

 

    return(XN_ISR_NOENABLE);

}

 

The only "evil" part is in a rtdm_ioctl() to work around BUG 05000416 for a fifo located in AMS-space :

 

#define CLI(x) __asm__ __volatile__ ("cli %0; NOP; NOP; SSYNC;" : "=d"(x):)

#define STI(x) __asm__ __volatile__ ("sti %0" : "=d"(x):)

 

...

 

        case READ_ADDR8:

        {

            unsigned long context;

 

            CLI(context);

 

            ret = (int)*(unsigned char*)arg;

          

            STI(context);

        }

        break;

 

...

 

 

 

I  played around with the kernel settings meanwhile, gave each sport, uart and int_mask_a a separate irq-level. Now it seems the ipipe-bug does not occur anymore, but soon it comes to a state, where both EMAC_RX and TX irqs are masked in SIC_IMASK preventing network traffic. Re-enabling them via ice and everything works again for a while...

 

It seems to me there is a missing lock somewhere in the ipipe or linux interrupt dispatcher...

 

regards,

TranslateQuoteReplyEditDelete

 

 

2011-08-29 05:06:37     Re: BUG() with xenomai __ipipe_sync_root

Sonic Zhang (CHINA)

Message: 103179   

 

Maybe you can try to disable hardware error debugging in kernel hacking config.

 

BTW: which kernel preemption model do you use? Volentary Preemption?

QuoteReplyEditDelete

 

 

2011-08-29 05:31:07     Re: BUG() with xenomai __ipipe_sync_root

Thorsten Pohlmann (GERMANY)

Message: 103180   

 

Yes, its voluntary preemption.

 

HWERR is not configurable (and disabled) due to CONFIG_IPIPE.

 

 

 

regards, Thorsten

TranslateQuoteReplyEditDelete

 

 

2011-09-07 05:05:27     Re: BUG() with xenomai __ipipe_sync_root

Thorsten Pohlmann (GERMANY)

Message: 103319   

 

The ipipe-bug still occurs, between 8 to 45h of contiuously performig the same tasks...

 

Could this happen due to switching between primary/secondary mode somewhere interrupted?

 

May the "interrupt shield" configuration help?

 

 

 

regards

Attachments

    Outcomes