2010-04-26 23:23:43     Write file to sd card can be block in bf518

Document created by Aaronwu Employee on Aug 21, 2013
Version 1Show Document
  • View in full screen mode

2010-04-26 23:23:43     Write file to sd card can be block in bf518

Bill Xie (CHINA)

Message: 88921   

 

Hi,

 

   I  have feeded back this problem to Yi Li at Blackfin Training in Guangzhou  China. Last months I offered this probem:

 

  blackfin.uclinux.org/gf/project/uclinux-dist/forum/?_forum_action=ForumMessageBrowse&thread_id=38524&action=ForumBrowse&forum_id=39

 

  Using writing thread according to  Wolfgang Muees suggest aslo lost packets because of the writing thread

 

not having enough time to write data to sd card.

 

    In this training one participant telled me that he use  spi inteface to connect  sd card in bf561 for writing image

 

file to sd card. It is not block in his project. But 518 have sdio interface  is faster than spi interface to connect sd  card and  packet (we use sd card to write audio data) size is smaller than image packet.

 

   I use bonnie++ to test Sandisk 8GB class 2.The result is:

 

Version  1.94       ------Sequential Output------ --Sequential Input- --Random-

Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--

Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP

blackfin       300M    43  98  1251  18   986  14   104  99  3563  15  58.6   4

Latency               552ms     995ms     548ms   88000us     671ms    9044ms

Version  1.94       ------Sequential Create------ --------Random Create--------

blackfin            -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--

              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP

                 16     5  99  5025 100    69  93     7  99  6095 100    16  97

Latency               696ms    8000us     503ms     580ms    4000us    1859ms

1.93c,1.94,blackfin,1,1167609719,300M,,43,98,1251,18,986,14,104,99,3563,15,58.6,

4,16,,,,,5,99,5025,100,69,93,7,99,6095,100,16,97,552ms,995ms,548ms,88000us,671ms

,9044ms,696ms,8000us,503ms,580ms,4000us,1859ms

 

 

 

Best Regards

 

-Bill

QuoteReplyEditDelete

 

 

2010-04-27 00:07:17     Re: Write file to sd card can be block in bf518

Yi Li (CHINA)

Message: 88926   

 

Bill,

 

What is date rate (Kbytes/second) you expect to write to the SD card?

 

I assume you are using a data capture thread writing data into the data buffer, and another writing thread transfer data from the buffer to SD card.  How about increase size of data buffer?

 

-Yi

QuoteReplyEditDelete

 

 

2010-04-27 00:13:01     Re: Write file to sd card can be block in bf518

Yi Li (CHINA)

Message: 88928   

 

Bill,

 

Also, how about increase the priority of your writing thread?

 

-Yi

QuoteReplyEditDelete

 

 

2010-04-27 02:27:11     Re: Write file to sd card can be block in bf518

Bill Xie (CHINA)

Message: 88933   

 

Yi,

 

   Thank you. We need the writing data rate above 500 Kbytes/s. One thread capture 8 channels data to 8 channels data buffer.Writing thread transfer 8 channel data to SD card.I test buffer size from 1k bytes to 320K.It is also have

 

the same problem. Not set priority of thread.

 

-Bill

QuoteReplyEditDelete

 

 

2010-04-27 02:56:40     Re: Write file to sd card can be block in bf518

Wolfgang Muees (GERMANY)

Message: 88935   

 

Post your code here. It's not possible to give you a good advice without looking at the code.

 

Minimum is:

 

- write thread

 

- data buffers between aquisition and write

 

- aquisition thread (only the main loop and data buffer sync).

 

regards

 

Wolfgang

TranslateQuoteReplyEditDelete

 

 

2010-05-04 06:15:38     Re: Write file to sd card can be block in bf518

Bill Xie (CHINA)

Message: 89109   

 

Wolfgang,

 

     Thank you.   I'm very sorry   that it took so long  time to post the code and result.

 

  Main thread get  data to fifo every 20ms(not accurate),Writing thread inquiry fifo size more than 1s buffer size to

 

write data to sd card. Total fifo size is loaded 4s data.The time spent to write 8 files record to bf518_sd.lua. Testing 12 hours not using fsync,The result with spent time more than average 10 times  is to result.txt file.

 

                   test count : 35989

                   total timelen : 2735(s)

           total_time_gettimelen : 2685(s)

                     max timelen : 2004412(us)

                     min timelen : 10305(us)

                    average/unit : 76002(us)

                      total size : 4393(M Bytes)

              average write rate : 1.606131(M Bytes/s)

 

Total test :                              35989  numbers

 

More than average 5 times:    2081    numbers

 

More than average 10 times:    1614   numbers

 

More than average 20 times:    71    numbers

 

 

 

If it is many number need  much more time in a short time,The fifo will be full to lead to lost packets. If the cpu deal with other events in application, the  results will be more poor.

 

 

If using fsync to write data to sd card, the result is very bad.

 

-Bill

 

simulate_write_sd-3.tar.gz

result.txt

bf518_sd.lua

QuoteReplyEditDelete

 

 

2010-05-05 04:14:36     Re: Write file to sd card can be block in bf518

Wolfgang Muees (GERMANY)

Message: 89149   

 

Bill,

 

I have checked your program. You write the data in increments of (320 x 50) bytes = 16000. This is not much. I think you will have better results if you are using increments which are a power of 2, and bigger size (p.e. 65536 Bytes). Then the fsync() will be much better (not so time-consuming).

 

But the problem of the uneven distribution of time will remain: there are buffers in the linux file system, so most of the writes will go only in the file system memory buffer, and from time to time, all dirty buffers are flushed to SD card by the file system layer at once. This is the long delay which you see. Using fsync(), you will be able to force the file system to make many small flushes instead of few large flushes, and so you will get a better distribution over time.

 

But hey, why do you worry at all? You get a data rate of 1.6 MBytes/Sec to SD card, and this is more than sufficient if you need only 500 KBytes/Sec. So all you need is a big buffer, and you will have no data loss.

 

Your fear that one day, you will have several long-lasting file writes at once, one after another, until your buffer overflows, will not happen. If the file system has done a flush to the SD card, the next writes will go fast again: the file system cache is empty.

 

Have you checked more than 1 SD card (different chips, different vendors, differenz sizes)?. If there are big differences among the cards, and your data aquisition programs runs out of CPU processing power if using a slow SD card, we might do some changes to the mmc framework in /drivers/mmc/card/block.c. There is a busy waiting loop after a WRITE to SD card.

 

regards

 

Wolfgang

TranslateQuoteReplyEditDelete

 

 

2010-05-06 06:10:42     Re: Write file to sd card can be block in bf518

Bill Xie (CHINA)

Message: 89175   

 

Wolfgang,

 

    Thank you.

 

    The buffer size is 16000 byte with async way write data to sd with 4G size.The result is follow:

 

          channel  = 8

 

          bufsize =      16000 bytes

             test count =      30851

            totle time = 2672640000 us, 2672 S

       totle time cycles = 2526850628 us, 2526 S

               max time =   15844000 us

              min time =       4000 us

           average time =      86630 us

             total size =       3765 Mbytes

 

               rate = 1442.000000 kbytes/S

 

 

      The average rate can be up to 1.4 Mbytes/s,but it  lost packet when some time testing.

 

 

     Using the same test program,Only using fsync() and setting every bufsize size is 32768 bytes.The result is follow:

 

        channel  = 8

         bufsize =      32768 bytes

      test count =        254

      totle time =  610556000 us, 610 S

    totle time cycles =  610512838 us, 610 S

        max time =    4808000 us

        min time =     548000 us

    average time =    2403763 us

      total size =         63 Mbytes

        rate = 106.000000 k bytes/S

 

  I am very confused that the result is not the same as you said .

 

Best Regard

 

Bill

Outcomes