2008-11-07 05:53:43     Infinite trace buffer dumps

2008-11-07 05:53:43     Infinite trace buffer dumps

Michael McTernan (UNITED KINGDOM)

Message: 64821   

Hi,

I'm using 2008R1.5 and found that when I have "Trace user apps (turn off hwtrace in kernel)" enabled the traces go on forever, literally until the platform is reset.  This is on a BF537-0.3.

The other configs are set like this:

[*] Turn on Blackfin's Hardware Trace                    

  │ │      Omit loop Tracing (Trace all Loops)  --->

  │ │[ ]   Expand Trace Buffer greater than 16 entries                 

  │ │[*]   Trace user apps (turn off hwtrace in kernel)   

I modified dump_bfin_trace_buffer() with the following:

printk(KERN_NOTICE "Hardware Trace TBUFCTL=%x:\n", tflags);

In both the case that the trace is infinite (trace user apps ON) or where the trace is correct (trace user apps OFF), the value printed is '1' which looks correct.

It would be nice to be able to use trace user apps - is there known fix or any suggestions?

Regards,

Mike

QuoteReplyEditDelete

2008-11-07 05:57:36     Re: Infinite trace buffer dumps

Mike Frysinger (UNITED STATES)

Message: 64822   

please read the help description on the option.  it doesnt do what you think it does.

are you using the release or svn branch ?  istr there being bugs in the trace dump code that we've fixed since the release ...

QuoteReplyEditDelete

2008-11-07 07:59:44     Re: Infinite trace buffer dumps

Michael McTernan (UNITED KINGDOM)

Message: 64832   

So I'm hoping that the option cuts out some of the kernel stuff so we may get a little more depth on some suspected faulty user space code without the overhead of the expanded trace buffer.  Isn't that what this does?

> are you using the release or svn branch ?  istr there being bugs in the trace dump code that we've fixed since the release ...

I'm at top of svn://firewall-sources.blackfin.uclinux.org:80/svn/linux-kernel/branches/2008R1 as of this morning.  I've compiled the traps_test and can see the infinite output with the exceptions raised by that.

QuoteReplyEditDelete

2008-11-07 08:09:33     Re: Infinite trace buffer dumps

Mike Frysinger (UNITED STATES)

Message: 64835   

ok, some people think that option allows them to only trace user apps rather than disable tracing in some "jumpy" kernel locations

there's probably some changes we didnt commit to the branch from the trunk as we test the traps thing on trunk semi-regularly ...

what hardware are you using ?

QuoteReplyEditDelete

2008-11-07 08:45:20     Re: Infinite trace buffer dumps

Michael McTernan (UNITED KINGDOM)

Message: 64837   

Right now it's a custom board similar to the BF537 STAMP design with the same parts selections.  I'm pretty sure I could make this happen on the BF537 EZKIT as well though.

FWIW, the traps output is pretty good on the 2008R1 branch (so long as trace user apps is off so it can complete

root:/> /bin/traps_test -1

Running test 0 : Data access misaligned address violation

... PASS (test failed as expected by signal 7: Bus error)

Running test 1 : Data access CPLB miss

... PASS (test failed as expected by signal 7: Bus error)

Running test 2 : Data access multiple CPLB hits/Null Pointer

... PASS (test failed as expected by signal 11: Segmentation fault)

Running test 3 : Instruction fetch misaligned address violation

... PASS (test failed as expected by signal 7: Bus error)

Running test 4 : Instruction fetch CPLB miss

... PASS (test failed as expected by signal 7: Bus error)

Running test 5 : l1_instruction_access

... PASS (test failed as expected by signal 7: Bus error)

Running test 6 : Illegal use of supervisor resource - Instruction

... PASS (test failed as expected by signal 4: Illegal instruction)

Running test 7 : Illegal use of supervisor resource - MMR

... PASS (test failed as expected by signal 7: Bus error)

Running test 8 : Instruction fetch multiple CPLB hits - Jump to zero

... FAIL (test failed, but not with the right signal)

        (We expected 11 'Segmentation fault' but instead we got 7 'Bus error')

Running test 9 : RAISE 5 instruction

... PASS (test failed as expected by signal 4: Illegal instruction)

Running test 10 : Invalid Opcode

... PASS (test failed as expected by signal 4: Illegal instruction)

QuoteReplyEditDelete

2008-11-07 11:07:58     Re: Infinite trace buffer dumps

Mike Frysinger (UNITED STATES)

Message: 64844   

i was mostly interested in the processor you were using ... there's currently an issue open about BF533, but it wasnt a prob on the BF537

QuoteReplyEditDelete

2008-11-07 15:21:48     Re: Infinite trace buffer dumps

Michael McTernan (UNITED KINGDOM)

Message: 64861   

I think there are a couple of bugs in the trace setup.  In trace.h from the 2008R1 branch, the following is defined:

#define BFIN_TRACE_INIT ((CONFIG_DEBUG_BFIN_HWTRACE_COMPRESSION << 4) | 0x03)

Looking in the Blackfin programmers reference manual rev 1.3, page 795, CMPLP[1:0] in TBUFCTL occupies bits 4 and 3, so the compression level should only be shifted by 3.  Additionally it also says:

"The Trace Unit is enabled by two control bits in the Trace Buffer Control

register (TBUFCTL) register. First, the Trace Unit must be activated by setting

the TBUFPWR bit. If TBUFPWR = 1, then setting TBUFEN to 1 enables the

Trace Unit."

I'm interpreting this to mean that BFIN_TRACE_INIT should only set TBUFPWR, and a call to trace_buffer_init() should be followed by trace_buffer_start() to correcltly initialise and then start tracing.  So I think we need:

#define BFIN_TRACE_INIT ((CONFIG_DEBUG_BFIN_HWTRACE_COMPRESSION << 3) | 0x01)

Possibly the init() calls in the platform head.S files should be ammended with a start() to get it going at boot, if that's desired?

Finally, Listing 21-1 "Recreating the Execution Trace in Memory" shows how to read out the trace buffer.  In the example it loops 32 times and reads the data, rather than relying on the value returned by TBUFSTAT.  Since TBUFSTAT isn't reliable in some cases, would it be sensible to use a bounded loop of 32 reads as per ADI's example in dump_bfin_trace_buffer().

I'm not sure this will correctly fix the infinite dump I'm seeing (it's the weekend so I have no access to a platform), but I think it maybe a bit better after these changes?

QuoteReplyEditDelete

2008-11-10 10:32:04     Re: Infinite trace buffer dumps

Michael McTernan (UNITED KINGDOM)

Message: 64946   

I've tested these changes on a platform, and tracing works as before, but still with infinite stack dumps when "trace user apps" is switched on.  Should I open a bug/patch for this, as the compression level looks definately wrong.

So I've modified the trace dump function to print TBUFCTL on each loop:

    printk(KERN_NOTICE "Hardware Trace:\n");

    if (likely(bfin_read_TBUFSTAT() & TBUFCNT)) {

        for (; (bfin_read_TBUFSTAT() & TBUFCNT) && i < 16; i++) {

            printk(KERN_NOTICE "ctl=%x\n", bfin_read_TBUFCTL());

            decode_address(buf, (unsigned long)bfin_read_TBUF());

            printk(KERN_NOTICE "%4i Target : %s\n", i, buf);

            decode_address(buf, (unsigned long)bfin_read_TBUF());

            printk(KERN_NOTICE "     Source : %s\n", buf);

        }

    }

It looks like the buffer is getting re-enabled during the dump:

Hardware Trace:

ctl=9

   0 Target : <0x00004344> /* unknown address */

     Source : <0x0000ddc2> /* unknown address */

ctl=b

   1 Target : <0x0000df32> /* unknown address */

     Source : <0x0000ddc2> /* unknown address */

ctl=b

   2 Target : <0x0000df32> /* unknown address */

....

So I figured that DEBUG_START_HWTRACE must somehow be getting called during the trace dump (as this only gets defined when CONFIG_DEBUG_BFIN_NO_KERN_HWTRACE is on, but can't see how.  I tried disabling interrupts during the dump incase something was getting in, but no luck - still infinite.

Is there some other way in which the trace could be getting re-enabled during the buffer dumping?

QuoteReplyEditDelete

2008-11-10 13:40:39     Re: Infinite trace buffer dumps

Michael McTernan (UNITED KINGDOM)

Message: 64966   

I think I've got this now.

I found the problem only happens when MPU protection is on, and then not always but most of the time.  I what's happening is that during the address decode of the trace buffer values there is a CPLB miss, which calls:

ENTRY(_ex_dcplb_viol)

ENTRY(_ex_dcplb_miss)

ENTRY(_ex_icplb_miss)

    (R7:6,P5:4) = [sp++];

    ASTAT = [sp++];

    SAVE_ALL_SYS

#ifdef CONFIG_MPU

    ....

#else

    call __cplb_hdr;

#endif

    DEBUG_START_HWTRACE(p5, r7)

    RESTORE_ALL_SYS

    SP = EX_SCRATCH_REG;

    rtx;

ENDPROC(_ex_icplb_miss)

And so the trace is re-enabled during the dump in the case that CONFIG_DEBUG_BFIN_NO_KERN_HWTRACE is defined.

I'm not sure how this is best fixed - my thoughts are that the DEBUG_START_HWTRACE() call could be changed to a save/restore such that the trace isn't changed if not already disabled, or that the buffer dump function could just grab the addresses from the buffer and then decode and display the lines in a second pass so as to lessen the risk of a CPLB miss.

QuoteReplyEditDelete

2008-11-11 05:30:11     Re: Infinite trace buffer dumps

Michael McTernan (UNITED KINGDOM)

Message: 65016   

I've submitted patch #4431 for this.