2011-04-08 09:34:18 SPI expert question -- multi master problem

Martin Strubel (SWITZERLAND)

Message: 99698

Hello,

I was wondering if someone would have an opinion about the following situation:

Attached to the Blackfin is a FPGA, communication is happening over SPI. Plus, the FPGA loads its firmware and other data from an SPI flash attached to the same "bus" (ok, SPI is not really a bus...), however, those two masters are separated through a gate, so that they don't interfere with each other while the system is running. In order to update the FPGA firmware, access to the SPI flash from the Blackfin is enabled by a GPIO while the FPGA is hold in reset (so it does not play 'master').

Since this is outside the typical SSELx chip select scheme, I've added another entity in the struct slavedata, i.e. there is now a

struct slavedata *chip; chip->sel_handler(int enable)

that is called withing bfin_spi_cs_active() and .._deactive(), which allows us to pull a GPIO low over the command phase among other things (possibly we might want to check a FPGA_BUSY flag later on). See also attached diff.

Since I couldn't find a better way of doing this, I was wondering if someone had run into a similiar problem. Since we don't want to maintain that as a "hack" over several kernel updates, it might make sense to subject this to discussion and make it a feature for complicate access ways of SPI devices.

Greetings,

- Strubi

spidiff.txt

QuoteReplyEditDelete

2011-04-08 09:52:08 SPI expert question -- multi master problem

Wojtek Skulski (UNITED STATES)

Message: 99699 Martin:

I have run into the same problem with my BlackVME board. I have two SPI

flashes on the same SPI bus. The PF2 flash is booting the BF561, the PF3

flash is booting the FPGA. It all works great except for the boot time,

when both chips are contending for the bus. I somehow missed this when I

was designing the board. My present "solution" is holding one chip (does

not matter, which one) in reset during power-up to allow the other chip

boot. Then I release the reset and the other chip can boot.

In Rev A of my board I am adding a bus switch separating the SPI bus into

two segments. The bus switch will be turned on/off with the PF3. There

will be a pullup on PF3, so the switch will be in OFF state until PF3 is

driven low. In this way the FPGA will boot from its flash without

contention, while BF561 is booting from its flash. After the system is up

and running, BF561 will be able to reprogram the FPGA flash at will. As

soon as it starts writing to it, PF3 will go low and open the switch. It

will happen transparently. Software modifications will not be required.

Ideally there would be two separate SPI busses, but this is not provided

by the BF561. I could also consider bit-banging the FPGA flash rather than

attaching it to the same SPI bus, but I am using all the GPIOs for

something else. Hence, I decided to try the bus switch solution. The worst

what can happen is that I again overlooked something and I will still need

to manually hold down the reset button after power up.

QuoteReplyEditDelete

2011-04-10 23:37:54 Re: SPI expert question -- multi master problem

Mike Frysinger (UNITED STATES)

Message: 99733

if access is only a temporary thing, then you could just have userspace allocate the pin via gpio-sysfs and set it high that way. then you'd never need the kernel to worry about things.

QuoteReplyEditDelete

2011-04-11 09:47:34 Re: SPI expert question -- multi master problem

Martin Strubel (SWITZERLAND)

Message: 99750

Mike: Right, that's what was done before, but turned out to work unstable. We're trying to mimick a bus system (which isn't really designed to be one) and I guess that should be handled within the kernel. We can't guarantee that no access to the MTD device is occuring by the kernel _after_ we switched the GPIO from user space. Fortunately we don't have to use a file system on the double-mastered flash, that would have made things even worse.

Anyhow, I was just wondering if this is a clean solution. It obviously worked so far, fortunately no HW changes required.

Wojtek: Sounds like a similar setup. How do you address this within uClinux? Do you have the FPGA flash (PF3) registered as MTD device via the mt25p80 driver? To make sure the bus doesn't content, I guess you'd have to insert some tristating gate logic (that decouples both masters). In our case, the Blackfin is the (primary) master-master, so if it wants to access the shared slave, it will reset the secondary master (fpga).

Greetings,

- Martin

QuoteReplyEditDelete

2011-04-11 10:29:01 Re: SPI expert question -- multi master problem

Wojtek Skulski (UNITED STATES)

Message: 99752 Martin:

in my case I opted for a HW solution because I am doing both, designing

the HW and later programming it. It should not come to you as a surprise

that once the problem is solved in the HW, it stays solved, while problems

solved in SW will reapper to be solved again. Hence, I prefer HW

solutions. The whole idea is to solve the problem 100% in HW and never

worry about SW.

Concerning the bus switch, I have not manufactured these modified boards

yet. I will manufacture them soon. Initially I will use uBoot to program

the FPGA flash. I already did this in the present revision, so I know it

will work. I simply type "sfprobe" under uBoot and it will switch to

whatever SSEL it is told to switch to. Concerning the uClinux support, it

is already there as far as I understand. But I have not tried it yet.

Concerning the tristate logic, I plan to use the quad SN74CB3T3125PW which

is the SPST switch. It either connects the FPGA SPI bus to the BF bus, or

it keeps it disconnected (the default state). A cleaner solution would be

a mux, which would toggle the flash between two busses. But I alredy have

the SN74CB3T3125PW elsewhere on my board, so I want to use the same part

to keep my inventory list short.

The FPGA is not supposed to be the SPI master on this SPI bus beyond the

boot phase. Therefore, if I decide to reprogram the PF3 flash from the BF

side, I can assume that the bus is tristated by the FPGA. Hence, it is OK

to connect the BF to the SPI bus and drive it. If I have used a mux, then

I would not have to make any assumptions about the FPGA. If you are

designing your board from scratch without any constraints on your part

inventory, then I suggest to use a mux. There are muxes manufactured by

Analog Devices which you can use.

BTW, I have several PFs connected to the FPGA. I can use one of these to

inform the FPGA logic that it should relinquish the SPI bus. So I have the

backup solution like yours. But I do not think it will be needed.

To make the picture complete let me admit that I envision talking to the

FPGA using the same SPI. I devoted PF7 for this. The bus switch is opened

not only by PF3, but by an OR of PF3/PF7. I am using SN74AHC1G08DBV to

create the OR signal. (Technically, it is an AND of low-level logic, but

let's say it is an OR.)

Hope it helps,

QuoteReplyEditDelete

2011-04-11 10:46:31 Re: SPI expert question -- multi master problem

Wojtek Skulski (UNITED STATES)

Message: 99754

Martin:

a picture is worth a thousand words. It is attached.

SPI_chips_schematic.pdf

QuoteReplyEditDelete

2011-04-11 12:18:57 Re: SPI expert question -- multi master problem

Martin Strubel (SWITZERLAND)

Message: 99758

Hi,

just to be clear: I consider the HW implementation solved. But since there is apparently no solution in uClinux yet for a multi SPI master environment hack (let's consider it one), my actual question concerned the implementation (SW) side. Since you're not reconfiguring the FPGA on the fly (from within uClinux), I guess it doesn't concern you much.

BTW, simply using a SSEL pin to switch the bus on the secondary master is not an option, we found it is toggling in a less controllable way, therefore I implemented that handler function.

QuoteReplyEditDelete

2011-04-11 12:39:51 Re: SPI expert question -- multi master problem

Wojtek Skulski (UNITED STATES)

Message: 99759

Martin:

during development I reconfigure the FPGA with JTAG. In the production environment I will reconfigure the FPGA perhaps once every few months. Writing to SPI flash and rebooting the FPGA from flash is OK if done infrequently.

What do you mean by saying that SSEL is toggling in a less controllable way? A hardware bug in BF silicon? What can cause the SSEL to toggle, other than an intentional action?

QuoteReplyEditDelete

2011-04-11 16:02:50 Re: SPI expert question -- multi master problem

Mike Frysinger (UNITED STATES)

Message: 99765

i'm not sure i see how you guarantee things with the FPGA even in your new scenario. if the FPGA is in the middle of doing something, how do you know it is safe for the kernel to start using the SPI flash ?

we could implement a bus-specific solution by moving the bfin_spi_cs_active/bfin_spi_cs_deactive calls into the bfin_spi_slave_data structure. then boards could do whatever they want there.

your current patch takes a more general approach and that needs to be discussed on the common spi mailing list: spi-devel-general@lists.sourceforge.net. we arent going to design/extend that here.

QuoteReplyEditDelete

2011-04-11 17:26:51 Re: SPI expert question -- multi master problem

Wojtek Skulski (UNITED STATES)

Message: 99769

Mike:

are you asking Martin or me? In my case the BF561 is controlling the FPGA. I am in the middle of writing the FPGA firmware. One thing I know for sure is that the FPGA will not be a master of the SPI bus except for a brief period when it is booting. After this has passed, the FPGA is a slave fully under the Blackfin control, as far as the SPI is concerned.

In a more general case you are of course right that the multi-master SPI has to be handled by programming each master such that it can recognize contention and react accordingly.

QuoteReplyEditDelete

2011-04-11 18:10:03 Re: SPI expert question -- multi master problem

Mike Frysinger (UNITED STATES)

Message: 99770

Martin wants a SW solution, so i'm asking about his solution. your design seems to be all set in your eyes, so i'm not going to worry about it.

QuoteReplyEditDelete

2011-04-12 10:46:31 Re: SPI expert question -- multi master problem

Martin Strubel (SWITZERLAND)

Message: 99789

Hi Mike,

> i'm not sure i see how you guarantee things with the FPGA even in your new scenario. if the FPGA is in the middle of doing something, how do you know it is safe for the kernel to start using the SPI flash ?

That's right, for the moment, the blackfin does not check if the FPGA is busy, it immediately overrides the access, but that's ok. If the access should be guarded, the driver should return an EBUSY error somewhere from some kind of check_bus_busy() routine. Not sure how much the current architecture would need to be modified.

> we could implement a bus-specific solution by moving the bfin_spi_cs_active/bfin_spi_cs_deactive calls into the bfin_spi_slave_data structure. then boards could do whatever they want there.

Would work too. Not sure how many people want to replicate that stuff in their board supply packages though (as only a few might use the wicked multi master hack).

> your current patch takes a more general approach and that needs to be discussed on the common spi mailing list: spi-devel-general@lists.sourceforge.net. we arent going to design/extend that here.

Agreed on the more general approach and thanks for the pointer. No extra work from your side expected We keep the hack for the moment, if we see the need to generalize this, we'll discuss it further on this mailing list.

Greetings & thanks,

- Martin

QuoteReplyEditDelete

2011-04-12 10:50:35 Re: SPI expert question -- multi master problem

Martin Strubel (SWITZERLAND)

Message: 99790

Wojtek:

> What do you mean by saying that SSEL is toggling in a less controllable way? A hardware bug in BF silicon? What can cause the SSEL to toggle, other than an intentional action?

No, it's definitely not a silicon bug. We just found that we had to use a GPIO to control the secondary master (and bus switch), because the SSEL pin is being toggled by the SPI framework in between SPI accesses. If I'm right there is an option to change the behaviour, but the GPIO solution turned out to be working first.

- Martin

QuoteReplyEditDelete

2011-04-12 10:56:07 Re: SPI expert question -- multi master problem

Wojtek Skulski (UNITED STATES)

Message: 99791 Martin:

in order to make the multimaster solution completely general, the access

to the common resource (flash chip in this case) has to be guarded with a

semaphore which both master can check and restrict their actions

accordingly. Accessing the common resource needs be made a critical

section for both masters. Otherwise there is a risk that the FPGA will

start using flash after the Blackfin checks the busy status, but before it

claims the ownership of the device. What it means in practice is that the