2009-08-19 09:21:44     SPI slave running at 2MHz and interrupt latency

Document created by Aaronwu Employee on Aug 18, 2013
Version 1Show Document
  • View in full screen mode

2009-08-19 09:21:44     SPI slave running at 2MHz and interrupt latency

Ulisses Montenegro (BRAZIL)

Message: 79055   

 

We have implemented a SPI slave driver for the Blackfin (we are using the BF527) where the protocol implements a byte-oriented request/response model (i.e., at every cycle I receive a request byte from the master and send a response byte in the next cycle). This means we cannot use DMA, because we cannot predict what will the MISO next byte we will be sending until we have read the current MOSI byte. This needs to work, however over a 2MHz SPI bus, and interrupt latency is extremely erratic in our tests.

 

We have a window of about 3µs between each byte, and most of the time our code runs between 1.5µs and 2µs after the clock has stopped, which is enough for us to read the RX register and write our payload to the TX, but sometimes our interrupt handler is called while the next byte from the master is already being received. We have no control on when the master starts transmission, so having it wait for us is not an option.

 

We have also observed that, while most of the times our interrupt handler will run under 1µs, sometimes it can take almost 30x that time, meaning we lose a lot of bytes, for no apparent reason (code flow inside the handler is the same every time, which sould mean it is not our code that is causing the different execution intervals).

 

Both behaviours have been observed under all kernel preemption models (CONFIG_PREEMPT_NONE, CONFIG_PREEMPT_VOLUNTARY and CONFIG_PREEMPT), and we do disable interrupts inside our handler.

 

Anyone got any experience on operating the Blackfin SPI at MHz speeds without DMA support? Is it even possible at all?

QuoteReplyEditDelete

 

 

2009-08-19 18:48:48     Re: SPI slave running at 2MHz and interrupt latency

Mike Frysinger (UNITED STATES)

Message: 79075   

 

you want the Blackfin SPI peripheral to be a slave on the SPI bus and something else is the master ?  that means you've implemented your own slave SPI driver rather than trying to re-use the existing one right ?

 

what you've described sounds accurate, but i dont think any of the preempt settings will make any difference at all.  those really only apply to userspace scheduling, not hardware interrupts.

 

have you tried raising the SPI priority in the kernel configuration menu ?

 

how are you talking to the board ?  is it via the serial console ?  what if you dont read/write to the serial console during the tests ?

QuoteReplyEditDelete

 

 

2009-08-20 09:47:23     Re: SPI slave running at 2MHz and interrupt latency

Ulisses Montenegro (BRAZIL)

Message: 79084   

 

Thanks for the answer Mike, here's a rundown of our status:

 

    Yes, we did write our own driver (I have reduced it to the simples, shortest piece of code we could find, it still fails occasionally and inconsistently);

    I thought the preemption modes would protect us from context switches inside our interrupt handler, but now that you say it is only meaningful for userspace latency then I can understand why it did not make much of a difference in our results;

    I have raise the SPI priority to the highest level, which made things a little better, but it sometimes still takes way longer than it should to execute, losing bytes;

    We are monitoring performance through a GPIO pin which is raised when our interrupt handler code starts, and lowered when it finishes. I am aware that using the serial console can add latency because it's much slower than the 2MHz SPI traffic we need to handle -- we are not using the serial during transfer, and no userland code producing output is being run during the tests, meaning the serial output should not be affecting our results.

 

Once we simplified our code to a simple read RX -> write !RX to TX handler, our latency did improve. Our real code will not be so simple, though (we use workqueues for communication with higher level protocol handlers), but even when our queues were not being processed (being consumed only) latency and interrupt handler duration still suffered.

 

I believe our timing requirements might be a little too much for a non-realtime OS to handle, but I need to make sure we have exhausted our options with Linux because our userland code does require it.

 

I have attached our source code, if anyone care to have a quick look and check whether we are doing anything obviously wrong. Once again, thanks again.

 

slave_spi.c

QuoteReplyEditDelete

 

 

2009-08-20 10:52:30     Re: SPI slave running at 2MHz and interrupt latency

Robin Getz (UNITED STATES)

Message: 79086   

 

Ulisses:

 

What kernel version are you using? It is kind of concerning that you seea 30x variation...

 

-Robin

QuoteReplyEditDelete

 

 

2009-08-20 14:11:55     Re: SPI slave running at 2MHz and interrupt latency

Mike Frysinger (UNITED STATES)

Message: 79089   

 

can you look at /proc/interrupts to see what other interrupts are firing ?

QuoteReplyEditDelete

 

 

2009-08-20 14:43:44     Re: SPI slave running at 2MHz and interrupt latency

Ulisses Montenegro (BRAZIL)

Message: 79091   

 

Robin

 

We are running 2.6.28 (this is actually uClinux-trunk revision 7856, at the time we started our project 2009R1 was not out yet).

 

Mike, our /proc/interrupts:

 

root:/> cat /proc/interrupts

  6:    1564923      CORE  Blackfin Core Timer

13:          1      INTN  PPI ERROR

21:          1      INTN  rtc-bfin

28:       5469      INTN  SPI_IRQ

31:          0      INTN  BFIN_UART_RX

32:        147      INTN  BFIN_UART_TX

NMI:          0      CORE  Non Maskable Interrupt

Err:          0

 

Even with the minimalistic code I attached to a previous post, we are still seeing some significant differences in interrupt latency and interrupt handler duration, as seen in the attached screenshots, which displays two bytes (first signal is our interrupt handler GPIO pin, second signal is the SCLK).

 

    First byte has a 1.8us interrupt latency (print_08.png), 1.4us interrupt handler duration (print_09.png)

    Second byte has a 840ns interrupt latency (print_10.png), 120ns interrupt handler duration ((print_11.png)

 

Not as obvious as the 30x differences we were experience before, but still rather baffling to us.

 

print_11.png

print_09.png

print_08.png

print_10.png

QuoteReplyEditDelete

 

 

2009-08-20 14:58:46     Re: SPI slave running at 2MHz and interrupt latency

Mike Frysinger (UNITED STATES)

Message: 79092   

 

i guess you need to be clear as to what your "30x" is measuring.  if one test case took 100ns and other took 3us, that is a 30x difference, but not something to be concerned with.  we look at things in terms of "typical interrupt latency should be below XXX us".

QuoteReplyEditDelete

 

 

2009-08-20 15:14:13     Re: SPI slave running at 2MHz and interrupt latency

Ulisses Montenegro (BRAZIL)

Message: 79093   

 

Sorry if I did not make myself clear -- the 30x were not associated with the interrupt latency, but rather with the interrupt handler execution duration. The fact that we cannot guarantee when our code starts running, or how long it takes to run even when there are no alternative code paths in our handler is what concerns us, because when the real code replaces it there will be certainly a lot more points for preemption by the kernel, meaning it will probably take even longer and possibly miss the start of the next byte.

QuoteReplyEditDelete

 

 

2009-08-20 15:30:41     Re: SPI slave running at 2MHz and interrupt latency

Mike Frysinger (UNITED STATES)

Message: 79094   

 

presumably you're seeing cache differences

 

what if you add __attribute__((l1_text)) to your irq handler ?  and make sure you have gpiolib disabled in your kernel .config

QuoteReplyEditDelete

 

 

2009-08-20 17:18:56     Re: SPI slave running at 2MHz and interrupt latency

Ulisses Montenegro (BRAZIL)

Message: 79095   

 

Disabled gpiolib, added __attribute__((l1_text)) and things have improved quite a bit, though latency still resides solidly in the 1.7us while doing absolutely nothing else on the system. If generate some network traffic the EMAC RX (I know, it was disabled in the results I sent before, but we need the ethernet for this project), it quickly jumps to around 4.5us when the traffic gets heavier, even with the SPI IRQ configured for maximum priority.

 

I also disabled CONFIG_BFIN_INS_LOWOVERHEAD, but it did not change much of anything. I am aware that these are pretty impressive numbers for a non-realtime OS, but I am starting to think it is next to impossible for us to achieve the 99.99% sucess rate for a 3us interval between bytes in our SPI transfers at 2MHz.

QuoteReplyEditDelete

 

 

2009-08-20 17:35:59     Re: SPI slave running at 2MHz and interrupt latency

Mike Frysinger (UNITED STATES)

Message: 79096   

 

i'd make sure you have the network packets in L1 data (it's a kconfig option), but interrupt processing with the mac driver is known to be worse than normal.  the irq hardware requires demuxing in software to figure out where the interrupt is coming from.

 

you could try xenomai/adeos and have your smaller processer running in realtime while keeping everything else under linux ...

QuoteReplyEditDelete

 

 

2009-08-21 03:27:26     Re: SPI slave running at 2MHz and interrupt latency

Michael Hennerich (GERMANY)

Message: 79102    > I am starting to think it is next to impossible for us to achieve the

99.99% sucess rate for a 3us interval between bytes in our SPI transfers

at 2MHz.

 

I wonder if you could use the SPORT instead of SPI Slave Mode?

The SPORT features a 8-deep FIFO for word lengths <= 16. So it's

unlikely to loose transfers.

In addition it features dedicated DMA channels for RX/TX.

 

-Michael

QuoteReplyEditDelete

 

 

2009-08-24 16:28:16     Re: SPI slave running at 2MHz and interrupt latency

Ulisses Montenegro (BRAZIL)

Message: 79160   

 

Michael

 

We will be using a SPORT interface for multichannel PCM audio (also working at 2MHz, but we can use DMA there), and the second interface is not available in our EZKIT development board because it shares pins with the UART1 (which we need for debugging purposes). We have reached an apparent limit on 800ns interrupt latency, which is very good, but it's still about 1/3 of the time we have between transfer clocks, leaving us very little to process and answer the SPI master.

QuoteReplyEditDelete

 

 

2009-08-24 22:55:39     Re: SPI slave running at 2MHz and interrupt latency

Sonic Zhang (CHINA)

Message: 79164   

 

If you are looking for a dedicated max interrupt latency, the only choice by now is run your driver and application under the 3rd party hard real time patch ADEOS. You can find it in our releases since 2008R1.

QuoteReplyEditDelete

Attachments

Outcomes