2009-08-19 09:21:44 SPI slave running at 2MHz and interrupt latency
Ulisses Montenegro (BRAZIL)
Message: 79055
We have implemented a SPI slave driver for the Blackfin (we are using the BF527) where the protocol implements a byte-oriented request/response model (i.e., at every cycle I receive a request byte from the master and send a response byte in the next cycle). This means we cannot use DMA, because we cannot predict what will the MISO next byte we will be sending until we have read the current MOSI byte. This needs to work, however over a 2MHz SPI bus, and interrupt latency is extremely erratic in our tests.
We have a window of about 3µs between each byte, and most of the time our code runs between 1.5µs and 2µs after the clock has stopped, which is enough for us to read the RX register and write our payload to the TX, but sometimes our interrupt handler is called while the next byte from the master is already being received. We have no control on when the master starts transmission, so having it wait for us is not an option.
We have also observed that, while most of the times our interrupt handler will run under 1µs, sometimes it can take almost 30x that time, meaning we lose a lot of bytes, for no apparent reason (code flow inside the handler is the same every time, which sould mean it is not our code that is causing the different execution intervals).
Both behaviours have been observed under all kernel preemption models (CONFIG_PREEMPT_NONE, CONFIG_PREEMPT_VOLUNTARY and CONFIG_PREEMPT), and we do disable interrupts inside our handler.
Anyone got any experience on operating the Blackfin SPI at MHz speeds without DMA support? Is it even possible at all?
QuoteReplyEditDelete
2009-08-19 18:48:48 Re: SPI slave running at 2MHz and interrupt latency
Mike Frysinger (UNITED STATES)
Message: 79075
you want the Blackfin SPI peripheral to be a slave on the SPI bus and something else is the master ? that means you've implemented your own slave SPI driver rather than trying to re-use the existing one right ?
what you've described sounds accurate, but i dont think any of the preempt settings will make any difference at all. those really only apply to userspace scheduling, not hardware interrupts.
have you tried raising the SPI priority in the kernel configuration menu ?
how are you talking to the board ? is it via the serial console ? what if you dont read/write to the serial console during the tests ?
QuoteReplyEditDelete
2009-08-20 09:47:23 Re: SPI slave running at 2MHz and interrupt latency
Ulisses Montenegro (BRAZIL)
Message: 79084
Thanks for the answer Mike, here's a rundown of our status:
Yes, we did write our own driver (I have reduced it to the simples, shortest piece of code we could find, it still fails occasionally and inconsistently);
I thought the preemption modes would protect us from context switches inside our interrupt handler, but now that you say it is only meaningful for userspace latency then I can understand why it did not make much of a difference in our results;
I have raise the SPI priority to the highest level, which made things a little better, but it sometimes still takes way longer than it should to execute, losing bytes;
We are monitoring performance through a GPIO pin which is raised when our interrupt handler code starts, and lowered when it finishes. I am aware that using the serial console can add latency because it's much slower than the 2MHz SPI traffic we need to handle -- we are not using the serial during transfer, and no userland code producing output is being run during the tests, meaning the serial output should not be affecting our results.
Once we simplified our code to a simple read RX -> write !RX to TX handler, our latency did improve. Our real code will not be so simple, though (we use workqueues for communication with higher level protocol handlers), but even when our queues were not being processed (being consumed only) latency and interrupt handler duration still suffered.
I believe our timing requirements might be a little too much for a non-realtime OS to handle, but I need to make sure we have exhausted our options with Linux because our userland code does require it.
I have attached our source code, if anyone care to have a quick look and check whether we are doing anything obviously wrong. Once again, thanks again.
slave_spi.c
QuoteReplyEditDelete
2009-08-20 10:52:30 Re: SPI slave running at 2MHz and interrupt latency
Robin Getz (UNITED STATES)
Message: 79086
Ulisses:
What kernel version are you using? It is kind of concerning that you seea 30x variation...
-Robin
QuoteReplyEditDelete
2009-08-20 14:11:55 Re: SPI slave running at 2MHz and interrupt latency
Mike Frysinger (UNITED STATES)
Message: 79089
can you look at /proc/interrupts to see what other interrupts are firing ?
QuoteReplyEditDelete
2009-08-20 14:43:44 Re: SPI slave running at 2MHz and interrupt latency
Ulisses Montenegro (BRAZIL)
Message: 79091
Robin
We are running 2.6.28 (this is actually uClinux-trunk revision 7856, at the time we started our project 2009R1 was not out yet).
Mike, our /proc/interrupts:
root:/> cat /proc/interrupts
6: 1564923 CORE Blackfin Core Timer
13: 1 INTN PPI ERROR
21: 1 INTN rtc-bfin
28: 5469 INTN SPI_IRQ
31: 0 INTN BFIN_UART_RX
32: 147 INTN BFIN_UART_TX
NMI: 0 CORE Non Maskable Interrupt
Err: 0
Even with the minimalistic code I attached to a previous post, we are still seeing some significant differences in interrupt latency and interrupt handler duration, as seen in the attached screenshots, which displays two bytes (first signal is our interrupt handler GPIO pin, second signal is the SCLK).
First byte has a 1.8us interrupt latency (print_08.png), 1.4us interrupt handler duration (print_09.png)
Second byte has a 840ns interrupt latency (print_10.png), 120ns interrupt handler duration ((print_11.png)
Not as obvious as the 30x differences we were experience before, but still rather baffling to us.
print_11.png
print_09.png
print_08.png
print_10.png
QuoteReplyEditDelete
2009-08-20 14:58:46 Re: SPI slave running at 2MHz and interrupt latency
Mike Frysinger (UNITED STATES)
Message: 79092
i guess you need to be clear as to what your "30x" is measuring. if one test case took 100ns and other took 3us, that is a 30x difference, but not something to be concerned with. we look at things in terms of "typical interrupt latency should be below XXX us".
QuoteReplyEditDelete
2009-08-20 15:14:13 Re: SPI slave running at 2MHz and interrupt latency
Ulisses Montenegro (BRAZIL)
Message: 79093
Sorry if I did not make myself clear -- the 30x were not associated with the interrupt latency, but rather with the interrupt handler execution duration. The fact that we cannot guarantee when our code starts running, or how long it takes to run even when there are no alternative code paths in our handler is what concerns us, because when the real code replaces it there will be certainly a lot more points for preemption by the kernel, meaning it will probably take even longer and possibly miss the start of the next byte.
QuoteReplyEditDelete
2009-08-20 15:30:41 Re: SPI slave running at 2MHz and interrupt latency
Mike Frysinger (UNITED STATES)
Message: 79094
presumably you're seeing cache differences
what if you add __attribute__((l1_text)) to your irq handler ? and make sure you have gpiolib disabled in your kernel .config
QuoteReplyEditDelete
2009-08-20 17:18:56 Re: SPI slave running at 2MHz and interrupt latency
Ulisses Montenegro (BRAZIL)
Message: 79095
Disabled gpiolib, added __attribute__((l1_text)) and things have improved quite a bit, though latency still resides solidly in the 1.7us while doing absolutely nothing else on the system. If generate some network traffic the EMAC RX (I know, it was disabled in the results I sent before, but we need the ethernet for this project), it quickly jumps to around 4.5us when the traffic gets heavier, even with the SPI IRQ configured for maximum priority.
I also disabled CONFIG_BFIN_INS_LOWOVERHEAD, but it did not change much of anything. I am aware that these are pretty impressive numbers for a non-realtime OS, but I am starting to think it is next to impossible for us to achieve the 99.99% sucess rate for a 3us interval between bytes in our SPI transfers at 2MHz.
QuoteReplyEditDelete
2009-08-20 17:35:59 Re: SPI slave running at 2MHz and interrupt latency
Mike Frysinger (UNITED STATES)
Message: 79096
i'd make sure you have the network packets in L1 data (it's a kconfig option), but interrupt processing with the mac driver is known to be worse than normal. the irq hardware requires demuxing in software to figure out where the interrupt is coming from.
you could try xenomai/adeos and have your smaller processer running in realtime while keeping everything else under linux ...
QuoteReplyEditDelete
2009-08-21 03:27:26 Re: SPI slave running at 2MHz and interrupt latency
Michael Hennerich (GERMANY)
Message: 79102 > I am starting to think it is next to impossible for us to achieve the
99.99% sucess rate for a 3us interval between bytes in our SPI transfers
at 2MHz.
I wonder if you could use the SPORT instead of SPI Slave Mode?
The SPORT features a 8-deep FIFO for word lengths <= 16. So it's
unlikely to loose transfers.
In addition it features dedicated DMA channels for RX/TX.
-Michael
QuoteReplyEditDelete
2009-08-24 16:28:16 Re: SPI slave running at 2MHz and interrupt latency
Ulisses Montenegro (BRAZIL)
Message: 79160
Michael
We will be using a SPORT interface for multichannel PCM audio (also working at 2MHz, but we can use DMA there), and the second interface is not available in our EZKIT development board because it shares pins with the UART1 (which we need for debugging purposes). We have reached an apparent limit on 800ns interrupt latency, which is very good, but it's still about 1/3 of the time we have between transfer clocks, leaving us very little to process and answer the SPI master.
QuoteReplyEditDelete
2009-08-24 22:55:39 Re: SPI slave running at 2MHz and interrupt latency
Sonic Zhang (CHINA)
Message: 79164
If you are looking for a dedicated max interrupt latency, the only choice by now is run your driver and application under the 3rd party hard real time patch ADEOS. You can find it in our releases since 2008R1.
QuoteReplyEditDelete
