2008-02-25 13:56:15 Anyone experience a problems with pthreads & DCACHE/dma?
Rob D (UNITED STATES)
Message: 51611 2008r1-rc1 on bf548-ezkit (I will try rc2 later).
I have this funky issue that I can't really explain.
I have a basic producer/consumer example that reads from SPORT1 (producer) and writes the data out to disk(consumer). Now when I use threads, I get corrupt data (I'm pretty sure I have the basic locking working correctly - see below).
The buffer in the producer would have some 0xFF's in there after the read() (which is not possible). There is a timing issues of some sort.
There are a few things that I have done that causes the issue to go away:
1) Turn off DCACHE in the kernel
-- Everything works as expected, no buffer corruption
2) instead of using malloc() for my data buffer, I use sram_alloc()
-- Everything works with DCACHE enabled
I'm just pretty confused. I do not experience any problems when i read from /dev/zero or /dev/urandom - it's just /dev/sportX. There appears to be a timing issues with invalidate_dcache_range() and the dma transfer or something... *shrug*
Just wondering if anyone else has run into this...
Thanks!
SPORT is setup like this:
config.fsync = 1;
config.late_fsync = 1;
config.lsb_first = 1;
config.word_len = 32;
config.dma_enabled = 1;
Producer thread looks like this:
while( !quit )
{
currentBuffer = GetNextBuffer(ctx, currentBuffer);
WriterLock(currentBuffer);
/* WORK */
memset(currentBuffer->data, 0xff, SPORT_BUFFER_LEN);
len = read( sport_fd, currentBuffer->data, SPORT_BUFFER_LEN );
WriterUnLock(currentBuffer);
}
Consumer thread looks like this:
while( !quit )
{
currentBuffer = GetNextBuffer(ctx, currentBuffer);
ReaderLock(currentBuffer);
/* WORK */
write(out_fd, currentBuffer->data, SPORT_BUFFER_LEN);
ReaderUnLock(currentBuffer);
}
QuoteReplyEditDelete
2008-02-26 02:31:15 Re: Anyone experience a problems with pthreads & DCACHE/dma?
Yi Li (CHINA)
Message: 51646 What if the dma is disabled? And what if not using two threads (read data from SPORT first, then write to disk)?
QuoteReplyEditDelete
2008-02-26 14:35:43 Re: Anyone experience a problems with pthreads & DCACHE/dma?
Rob D (UNITED STATES)
Message: 51673 Hi Yi,
If DMA is disabled, it all works fine.
If it's sequential (no threads) it all works fine (with DMA).
If I add printk statements to the beginning sport_read() in the kernel, it all works fine (with DMA).
If I use sram_alloc() instead of malloc(), it works fine (with DMA)- which is what I'm doing now. But I'm not happy with this solution because I don't know what the underlying problem is.
If I use '/dev/zero' instead of '/dev/sport1', it all works fine (no data corruption)
There is definitely something going on with the DMA portion of the sport. I don't know if it's a Linux buffering thing, or a compiler thing... or what.
Any thoughts?
Thanks!
QuoteReplyEditDelete
2008-02-26 14:40:50 Re: Anyone experience a problems with pthreads & DCACHE/dma?
Rob D (UNITED STATES)
Message: 51674 Actually, here is my buffer.[ch] with the locking stuff in there... it's really basic, and I don't think there is anything wrong with it. You'll notice in InitBufferPool() I have an #ifdef for using sram_alloc() vs malloc().
-Rob
buffer.c
buffer.h
QuoteReplyEditDelete
2008-02-26 21:50:32 Re: Anyone experience a problems with pthreads & DCACHE/dma?
Yi Li (CHINA)
Message: 51687 It is not easy for me to make conclusion that this is a driver issue or application issue, without running a test case on board. But it looks GetNextBuffer() is not thread safe, two threads work on a global "currentBuffer" without lock.
"
SAFE_BUFFER *GetNextBuffer( SAFE_BUFFER_POOL_CTX *ctx, SAFE_BUFFER *current )
{
SAFE_BUFFER *retVal = NULL;
if( (current==NULL) ||
(current==ctx->last) )
{
// return the 1st buffer
retVal = ctx->bufPool;
}
else
{
// return the next buffer
retVal = (current+1);
}
return retVal;
}
"
QuoteReplyEditDelete
2008-02-27 22:49:36 Re: Anyone experience a problems with pthreads & DCACHE/dma?
Rob D (UNITED STATES)
Message: 51756 Hi Yi!
No, that's not a problem, currentBuffer is local to the producer and the consumer, the only thing that is shared is the 'ctx'.
I've uploaded a sample user project that demonstrates this bug... when the bug happens it prints out:
root:/> test_main
Should _never_ happen (24)
[0000] FF FF FF FF FF FF FF FF 04 00 00 00 0D F5 FF FF ........ ........
[0010] 05 01 00 00 0E FF FF FF ........
see those FF's in a row? The device I'm talking to should never print out more then three 0xFFs in a row. And this only happens when I use malloc instead of sram_alloc() with DCACHE enabled... I don't get it.
If you could take a quick look at this for me, I'd really appreciate it. As I said before, I'm pretty sure my buffer mutex locking is fine, as it works perfectly using /dev/zero etc... It appears to be a DCACHE issue... but perhaps I'm doing something silly :/
Anyway, thanks for the support... you guys are great!
-Rob
test_dcache_bug.tar.gz
QuoteReplyEditDelete
2008-02-29 04:44:49 Re: Anyone experience a problems with pthreads & DCACHE/dma?
Yi Li (CHINA)
Message: 51891 I can build and run your test case using the char/bfin_sport.c driver. But I am using an ad-73311 sound board for testing. It may not be possible to reproduce your issue exactly without your device. But I will look into the driver.
QuoteReplyEditDelete
2008-02-29 13:21:00 Re: Anyone experience a problems with pthreads & DCACHE/dma?
Rob D (UNITED STATES)
Message: 51912 Yi,
FYI, my device is spitting out a 32bit word about every 260us. The bits are clocked in at ~380kHz. I read 6 of them at a time before passing the buffer off to the consumer thread.
I'm not sure if something similar will be possible using the ad-73311, but I am hopeful that my test app fails in the same way in your setup.
Thanks,
Rob