2010-04-26 23:23:43 Write file to sd card can be block in bf518

Bill Xie (CHINA)

Message: 88921

Hi,

I have feeded back this problem to Yi Li at Blackfin Training in Guangzhou China. Last months I offered this probem:

blackfin.uclinux.org/gf/project/uclinux-dist/forum/?_forum_action=ForumMessageBrowse&thread_id=38524&action=ForumBrowse&forum_id=39

Using writing thread according to Wolfgang Muees suggest aslo lost packets because of the writing thread

not having enough time to write data to sd card.

In this training one participant telled me that he use spi inteface to connect sd card in bf561 for writing image

file to sd card. It is not block in his project. But 518 have sdio interface is faster than spi interface to connect sd card and packet (we use sd card to write audio data) size is smaller than image packet.

I use bonnie++ to test Sandisk 8GB class 2.The result is:

Version 1.94 ------Sequential Output------ --Sequential Input- --Random-

Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP

blackfin 300M 43 98 1251 18 986 14 104 99 3563 15 58.6 4

Latency 552ms 995ms 548ms 88000us 671ms 9044ms

Version 1.94 ------Sequential Create------ --------Random Create--------

blackfin -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP

16 5 99 5025 100 69 93 7 99 6095 100 16 97

Latency 696ms 8000us 503ms 580ms 4000us 1859ms

1.93c,1.94,blackfin,1,1167609719,300M,,43,98,1251,18,986,14,104,99,3563,15,58.6,

4,16,,,,,5,99,5025,100,69,93,7,99,6095,100,16,97,552ms,995ms,548ms,88000us,671ms

,9044ms,696ms,8000us,503ms,580ms,4000us,1859ms

Best Regards

-Bill

QuoteReplyEditDelete

2010-04-27 00:07:17 Re: Write file to sd card can be block in bf518

Yi Li (CHINA)

Message: 88926

Bill,

What is date rate (Kbytes/second) you expect to write to the SD card?

I assume you are using a data capture thread writing data into the data buffer, and another writing thread transfer data from the buffer to SD card. How about increase size of data buffer?

-Yi

QuoteReplyEditDelete

2010-04-27 00:13:01 Re: Write file to sd card can be block in bf518

Yi Li (CHINA)

Message: 88928

Bill,

Also, how about increase the priority of your writing thread?

-Yi

QuoteReplyEditDelete

2010-04-27 02:27:11 Re: Write file to sd card can be block in bf518

Bill Xie (CHINA)

Message: 88933

Yi,

Thank you. We need the writing data rate above 500 Kbytes/s. One thread capture 8 channels data to 8 channels data buffer.Writing thread transfer 8 channel data to SD card.I test buffer size from 1k bytes to 320K.It is also have

the same problem. Not set priority of thread.

-Bill

QuoteReplyEditDelete

2010-04-27 02:56:40 Re: Write file to sd card can be block in bf518

Wolfgang Muees (GERMANY)

Message: 88935

Post your code here. It's not possible to give you a good advice without looking at the code.

Minimum is:

- write thread

- data buffers between aquisition and write

- aquisition thread (only the main loop and data buffer sync).

regards

Wolfgang

TranslateQuoteReplyEditDelete

2010-05-04 06:15:38 Re: Write file to sd card can be block in bf518

Bill Xie (CHINA)

Message: 89109

Wolfgang,

Thank you. I'm very sorry that it took so long time to post the code and result.

Main thread get data to fifo every 20ms(not accurate),Writing thread inquiry fifo size more than 1s buffer size to

write data to sd card. Total fifo size is loaded 4s data.The time spent to write 8 files record to bf518_sd.lua. Testing 12 hours not using fsync,The result with spent time more than average 10 times is to result.txt file.

test count : 35989

total timelen : 2735(s)

total_time_gettimelen : 2685(s)

max timelen : 2004412(us)

min timelen : 10305(us)

average/unit : 76002(us)

total size : 4393(M Bytes)

average write rate : 1.606131(M Bytes/s)

Total test : 35989 numbers

More than average 5 times: 2081 numbers

More than average 10 times: 1614 numbers

More than average 20 times: 71 numbers

If it is many number need much more time in a short time,The fifo will be full to lead to lost packets. If the cpu deal with other events in application, the results will be more poor.

If using fsync to write data to sd card, the result is very bad.

-Bill

simulate_write_sd-3.tar.gz

result.txt

bf518_sd.lua

QuoteReplyEditDelete

2010-05-05 04:14:36 Re: Write file to sd card can be block in bf518

Wolfgang Muees (GERMANY)

Message: 89149

Bill,

I have checked your program. You write the data in increments of (320 x 50) bytes = 16000. This is not much. I think you will have better results if you are using increments which are a power of 2, and bigger size (p.e. 65536 Bytes). Then the fsync() will be much better (not so time-consuming).

But the problem of the uneven distribution of time will remain: there are buffers in the linux file system, so most of the writes will go only in the file system memory buffer, and from time to time, all dirty buffers are flushed to SD card by the file system layer at once. This is the long delay which you see. Using fsync(), you will be able to force the file system to make many small flushes instead of few large flushes, and so you will get a better distribution over time.

But hey, why do you worry at all? You get a data rate of 1.6 MBytes/Sec to SD card, and this is more than sufficient if you need only 500 KBytes/Sec. So all you need is a big buffer, and you will have no data loss.

Your fear that one day, you will have several long-lasting file writes at once, one after another, until your buffer overflows, will not happen. If the file system has done a flush to the SD card, the next writes will go fast again: the file system cache is empty.

Have you checked more than 1 SD card (different chips, different vendors, differenz sizes)?. If there are big differences among the cards, and your data aquisition programs runs out of CPU processing power if using a slow SD card, we might do some changes to the mmc framework in /drivers/mmc/card/block.c. There is a busy waiting loop after a WRITE to SD card.

regards

Wolfgang

TranslateQuoteReplyEditDelete

2010-05-06 06:10:42 Re: Write file to sd card can be block in bf518

Bill Xie (CHINA)

Message: 89175

Wolfgang,

Thank you.

The buffer size is 16000 byte with async way write data to sd with 4G size.The result is follow:

channel = 8

bufsize = 16000 bytes

test count = 30851