2008-02-25 13:56:15     Anyone experience a problems with pthreads & DCACHE/dma?

Document created by Aaronwu Employee on Aug 5, 2013
Version 1Show Document
  • View in full screen mode

2008-02-25 13:56:15     Anyone experience a problems with pthreads & DCACHE/dma?

Rob D (UNITED STATES)

Message: 51611    2008r1-rc1  on bf548-ezkit (I will try rc2 later).

 

I have this funky issue that I can't really explain.

 

I have a basic producer/consumer example that reads from SPORT1 (producer) and writes the data out to disk(consumer).  Now when I use threads, I get corrupt data (I'm pretty sure I have the basic locking working correctly - see below).

 

The buffer in the producer would have some 0xFF's in there after the read() (which is not possible).  There is a timing issues of some sort.

 

There are a few things that I have done that causes the issue to go away:

1) Turn off DCACHE in the kernel

-- Everything works as expected, no buffer corruption

2) instead of using malloc() for my data buffer, I use sram_alloc()

-- Everything works with DCACHE enabled

 

I'm just pretty confused.  I do not experience any problems when i read from /dev/zero or /dev/urandom - it's just /dev/sportX.  There appears to be a timing issues with invalidate_dcache_range() and the dma transfer or something... *shrug*

 

Just wondering if anyone else has run into this...

 

Thanks!

 

SPORT is setup like this:

  config.fsync = 1;

  config.late_fsync = 1;

  config.lsb_first = 1;

  config.word_len = 32;

  config.dma_enabled = 1;

 

Producer thread looks like this:

  while( !quit )

    {

      currentBuffer = GetNextBuffer(ctx, currentBuffer);

      WriterLock(currentBuffer);

      /* WORK */

      memset(currentBuffer->data, 0xff, SPORT_BUFFER_LEN);

      len = read( sport_fd, currentBuffer->data, SPORT_BUFFER_LEN );

      WriterUnLock(currentBuffer);

    }

 

Consumer thread looks like this:

  while( !quit )

    {

      currentBuffer = GetNextBuffer(ctx, currentBuffer);

      ReaderLock(currentBuffer);

      /* WORK */

      write(out_fd, currentBuffer->data, SPORT_BUFFER_LEN);

      ReaderUnLock(currentBuffer);

    }

QuoteReplyEditDelete

 

 

2008-02-26 02:31:15     Re: Anyone experience a problems with pthreads & DCACHE/dma?

Yi Li (CHINA)

Message: 51646    What if the dma is disabled? And what if not using two threads (read data from SPORT first, then write to disk)?

QuoteReplyEditDelete

 

 

2008-02-26 14:35:43     Re: Anyone experience a problems with pthreads & DCACHE/dma?

Rob D (UNITED STATES)

Message: 51673    Hi Yi,

 

If DMA is disabled, it all works fine.

 

If it's sequential (no threads) it all works fine (with DMA).

 

If I add printk statements to the beginning sport_read() in the kernel, it all works fine (with DMA).

 

If I use sram_alloc() instead of malloc(), it works fine (with DMA)- which is what I'm doing now.  But I'm not happy with this solution because I don't know what the underlying problem is.

 

If I use '/dev/zero' instead of '/dev/sport1', it all works fine (no data corruption)

 

There is definitely something going on with the DMA portion of the sport.  I don't know if it's a Linux buffering thing, or a compiler thing...  or what.

 

Any thoughts?

 

Thanks!

QuoteReplyEditDelete

 

 

2008-02-26 14:40:50     Re: Anyone experience a problems with pthreads & DCACHE/dma?

Rob D (UNITED STATES)

Message: 51674    Actually, here is my buffer.[ch] with the locking stuff in there...  it's really basic, and I don't think there is anything wrong with it.  You'll notice in InitBufferPool() I have an #ifdef for using sram_alloc() vs malloc().

 

-Rob

buffer.c

buffer.h

QuoteReplyEditDelete

 

 

2008-02-26 21:50:32     Re: Anyone experience a problems with pthreads & DCACHE/dma?

Yi Li (CHINA)

Message: 51687    It is not easy for me to make conclusion that this is a driver issue or application issue, without running a test case on board. But it looks GetNextBuffer() is not thread safe, two threads work on a global "currentBuffer" without lock.

 

 

"

 

SAFE_BUFFER *GetNextBuffer( SAFE_BUFFER_POOL_CTX *ctx, SAFE_BUFFER *current )

{

 

  SAFE_BUFFER *retVal = NULL;

 

  if( (current==NULL) ||

      (current==ctx->last) )

    {

      // return the 1st buffer

      retVal = ctx->bufPool;

    }

  else

    {

      // return the next buffer

      retVal = (current+1);

    }

 

  return retVal;

}

 

"

QuoteReplyEditDelete

 

 

2008-02-27 22:49:36     Re: Anyone experience a problems with pthreads & DCACHE/dma?

Rob D (UNITED STATES)

Message: 51756    Hi Yi!

 

No, that's not a problem, currentBuffer is local to the producer and the consumer, the only thing that is shared is the 'ctx'.

 

I've uploaded a sample user project that demonstrates this bug... when the bug happens it prints out:

 

root:/> test_main                                                                    

Should _never_ happen (24)                                                           

[0000]   FF FF FF FF FF FF FF FF   04 00 00 00 0D F5 FF FF   ........ ........       

[0010]   05 01 00 00 0E FF FF FF                             ........             

 

see those FF's in a row?  The device I'm talking to should never print out more then three 0xFFs in a row.  And this only happens when I use malloc instead of sram_alloc() with DCACHE enabled... I don't get it.

 

If you could take a quick look at this for me, I'd really appreciate it.  As I said before, I'm pretty sure my buffer mutex locking is fine, as it works perfectly using /dev/zero etc...  It appears to be a DCACHE issue... but perhaps I'm doing something silly :/

 

Anyway, thanks for the support... you guys are great! 

 

-Rob

test_dcache_bug.tar.gz

QuoteReplyEditDelete

 

 

2008-02-29 04:44:49     Re: Anyone experience a problems with pthreads & DCACHE/dma?

Yi Li (CHINA)

Message: 51891    I can build and run your test case using the char/bfin_sport.c driver. But I am using an ad-73311 sound board for testing.  It may not be possible to reproduce your issue exactly without your device. But I will look into the driver.

QuoteReplyEditDelete

 

 

2008-02-29 13:21:00     Re: Anyone experience a problems with pthreads & DCACHE/dma?

Rob D (UNITED STATES)

Message: 51912    Yi,

 

FYI, my device is spitting out a 32bit word about every 260us.  The bits are clocked in at ~380kHz.      I read 6 of them at a time before passing the buffer off to the consumer thread.

 

I'm not sure if something similar will be possible using the ad-73311, but I am hopeful that my test app fails in the same way in your setup.

 

Thanks,

Rob

Attachments

Outcomes