Problem with SPI tunneling and A2B upstream bus

I'm curious about this as I have a similar network to bring up when hardware comes back next month.  Here's my 20 questions:

• Does it have discovery problems if you don't do any SPI transactions?
• What happens if you discover everything first and then do all of the SPI initialization?
• Does it discover reliably from SS+?  (you can script SS+ to run tests vs. doing things manually)
• Kind of looks like a node has gone dead. After its in the fail state can you still communicate with all nodes (like toggle a GPIO)?  If a node has gone dead is it the same one?
• Maybe probe at the end of the system and see what the A2B traffic looks like?
• What if you use 24 bits upstream slots or 16 bit downstream?  i.e. make them all the same just to rule out that the different sizes are a factor.
• What A2B stack is running on the NXP processor?  Honestly my first thought is that the problem will be in the software.
• Have you captured the I2C traffic between the NXP and main node in the good vs. fail case, and what are the differences? Hate to say it but I've solved many weird issues by slogging through the bus traffic.  I usually copy it into a spreadsheet as that for me can be easier to manipulate.
• How are your nodes powered?  Are any bus powered?  If yes that's another area where I've been burned by a dumb mistake on my part.
  
  

Good luck.

To be more clear as possible, I will first describe the complete system architecture and then answer your questions point by point.

The main board contains two A2B AD2437 devices, implementing two independent A2B branches. An NXP processor manages both branches via SPI and also acts as an audio mixer/matrix between them.

The main board is externally powered, while all subnodes are bus-powered using a 3-wire A2B connection. Unlike the evaluation boards, the nodes are not connected through XLR cables but through a custom interconnection system designed for our application.

Network discovery is performed simultaneously on both branches, following this sequence:

1. Initialization of Main 1
2. Initialization of Main 2
3. Discovery of the subnodes on Branch 1
4. Discovery of the subnodes on Branch 2

Once discovery is complete, the final configuration of both main nodes is performed. At this stage, SPI tunneling is enabled and the codec registers are configured.

All of this is handled by an A2B driver developed in-house, without using the complete A2B software stack provided by Analog Devices.

Brewster said:

Does it have discovery problems if you don't do any SPI transactions?

Before the SPI transaction, we perform a Chip ID read of all subnodes and compare it with the values obtained during the discovery phase. It occurred that while the Chip ID re-read was successful, the reading of the codec registers via SPI tunneling failed. Consequently, the network was functional, but SPI tunneling was not.
Specifically, after writing to a codec register, a read-back is performed, but the value retrieved differs from the expected one. Below are some examples of these mismatches:
Node 3, Codec 1 error: Expected 128, Read 192
Node 3, Codec 2 error: Expected 128, Read 192
Node 3, Codec 1 error: Expected 3, Read 129
Node 3, Codec 2 error: Expected 3, Read 129
Node 3, Codec 1 error: Expected 192, Read 224
Node 3, Codec 2 error: Expected 192, Read 224
A clear bit-shifting pattern is visible in the data.
This indicates a 1-clock cycle misalignment during the SPI tunneling phase.

Brewster said:

What happens if you discover everything first and then do all of the SPI initialization?

This is already how we are currently handling it.

Brewster said:

Does it discover reliably from SS+?  (you can script SS+ to run tests vs. doing things manually)

Unfortunately, it is not possible for us to perform network discovery using SigmaStudio+.

Brewster said:

Kind of looks like a node has gone dead. After its in the fail state can you still communicate with all nodes (like toggle a GPIO)?  If a node has gone dead is it the same one?

I will need to verify this point internally and will get back to you with additional information. I can also add that, on one occasion, the codec register read operation to check the write op failed on one sub node, while the exact same operation succeeded on the next node in the chain.

Brewster said:

Maybe probe at the end of the system and see what the A2B traffic looks like?

We will try moving the probe between different nodes in the network to observe the A2B bus at multiple locations and determine whether the issue originates from a specific node or propagates through the network.

Brewster said:

What if you use 24 bits upstream slots or 16 bit downstream?  i.e. make them all the same just to rule out that the different sizes are a factor.

The issue still occurs even when following this procedure.

Brewster said:

What A2B stack is running on the NXP processor?  Honestly my first thought is that the problem will be in the software.

We are using an in-house developed A2B driver rather than the Analog Devices A2B software stack.

Brewster said:

Have you captured the I2C traffic between the NXP and main node in the good vs. fail case, and what are the differences? Hate to say it but I've solved many weird issues by slogging through the bus traffic.  I usually copy it into a spreadsheet as that for me can be easier to manipulate.

We have not captured the traffic between the NXP processor and the main node. Since we developed the A2B driver ourselves and use only SPI communication, we are reasonably confident about the commands being transmitted by the NXP processor. However, capturing this traffic could still be useful to verify exactly what is being transmitted by the A2B node, particularly when the SPI FIFO register is read.

Brewster said:

How are your nodes powered?  Are any bus powered?  If yes that's another area where I've been burned by a dumb mistake on my part.

All subnodes are bus-powered. The A2B bus power supply voltage is 12 V.

Thanks for the more detailed explanation. This reads like one of those "it should just work but yet it doesn't" and I usually learn some new gotcha about the parts from this type of thing, hence my interest.

I should have said SPI traffic instead of I2C when talking about monitoring what's going on.  I've seen cases where I2C transactions abort in weird ways and the only way to see that is from watching the signals at the main node as the software stack doesn't get that type of visibility.  SPI shouldn't be prone to that but it's still worth capturing, particularly as you seem to be seeing an unexplained bit shift.

The fact you can talk to the next node after the failed one is interesting and I feel like should be a clue, but I can't come up with one.

I would look to simplify the system as much as possible; one chain, reduce sub node count, etc., but that still gives you the failure.

On being bus powered: are you using bus powered 2 wire scheme or separate leads for power? Either way I would still do this check, though if 2 wire you have to use a differential scope probe.  Check the supply voltages at the AD2437 that fails as well as next one down.  I doubt this is a power issue but good to rule that out.

You using LVI mode or supplying 9V to the AD2437s?

How are you generating the main node FSYNC/BCLK?  If coming from your NXP processor worth a check that they're stable.

I wonder if you're getting audio data corruption to but that would be less likely to be noticed.  Actually there's another point; if the error was in A2B transport that would be caught so it's gotta be before/after that.  Wonder if you could just set up an SPI BER test - cram known PR sequence down the pipe and see what you get on the other end? 

Thanks again for taking the time to share your insights, and apologies for the delay—I've been quite busy over the last few days. I also want to emphasize that this error is highly intermittent and completely non-systematic, which makes it quite a headache and very time-consuming to debug.

That being said, I have some new clues regarding the problem. I added a readback of the A2B SPI configuration registers written to the node during the configuration phase: it seems that when the SPI tunneling readback fails, the specific SPI configuration register that controls the chip select of the A2B module actually resets and stops working as intended.

To clarify my previous post: after running several more tests, I noticed that the SPI tunneling readback error doesn't always result in a bit-shift. Actually, the vast majority of the time, the readback just returns all zeros (0x00) or all ones (0xFF). This makes sense with the chip select register resetting itself (since without the CS, the SPI line isn't driven). The bit-shift I mentioned earlier seems to be just one specific symptom of this broader, random failure.

Brewster said:

I would look to simplify the system as much as possible;

I haven't tried simplifying the system yet. The error is already quite rare with the full setup (sometimes it takes over 500 discovery + configuration cycles to reproduce it even once). I am worried that by shortening the chain, the issue might become so rare that it becomes nearly impossible to catch and debug. However, I am keeping this as a fallback step.

Brewster said:

On being bus powered: are you using bus powered 2 wire scheme or separate leads for power?

Regarding the power supply, I measured the voltage on the A2B twisted pair, and the bus is actually running at 12V. All subnodes are bus-powered via a daisy-chain (IN/OUT on each node) using a 3-wire scheme: A2B+, A2B-, and a dedicated Ground wire that connects all sub-node grounds back to the main board ground. Given that the SPI chip select register resets during the codec initialization, I am starting to wonder if the inrush current of the codecs is causing a momentary voltage drop locally on the node, or if my common ground wire is introducing noise that affects the local SPI signals. Unfortunately, I don't have a differential probe at the moment to confirm this, but I will try to get one.

Brewster said:

How are you generating the main node FSYNC/BCLK?  If coming from your NXP processor worth a check that they're stable

As for the audio clocks, they are generated by the NXP processor, and I am fairly confident they are stable. This is mainly because of my dual-branch architecture: when Branch 1 experiences this corruption and SPI failure, Branch 2 (which relies on the exact same NXP FSYNC/BCLK) continues to run without issues.

Brewster said:

Wonder if you could just set up an SPI BER test - cram known PR sequence down the pipe and see what you get on the other end?

Thanks for the suggestion. I am currently digging into the AD2437 Technical Reference Manual to figure out how to implement the PR sequence test and integrate it into my custom driver. I will definitely give this a try if I can.

StevenNob said:

I haven't tried simplifying the system yet. The error is already quite rare with the full setup

I was thinking for starters just run one bus instead of 2. Sounds like they're sort of isolated software wise but seemed like a simple thing to start with.

If the errors happen less often with less 'stuff' then maybe go the other way and hang more nodes off the bus and see if you see the error more often?

StevenNob said:

or if my common ground wire is introducing noise that affects the local SPI signals. Unfortunately, I don't have a differential probe at the moment to confirm this, but I will try to get one.

I honestly don't think power is a problem in your setup but still think it should be formally eliminated.  I have been affected by the collective IR drops lowering the supply side and raising the return side enough that the AD2437 decided it was a brown out.  Those issues were all on a heavily loaded bus and I don't think you have that.  

A-B mode of the scope with the scope ground leads on the supply ground will work well enough to decide if there's a problem there or not.  Might need to AC couple the probe channels - you just want to see if there's a blip of some sort, if you do see a big (couple of volts) then yeah you'll want to make a better measurement.

---------------

I still think the most likely issue is some software related problem and that looking at the hardware SPI line(s) might show something helpful.  Or not...

StevenNob said:

I haven't tried simplifying the system yet. The error is already quite rare with the full setup

I was thinking for starters just run one bus instead of 2. Sounds like they're sort of isolated software wise but seemed like a simple thing to start with.

If the errors happen less often with less 'stuff' then maybe go the other way and hang more nodes off the bus and see if you see the error more often?

StevenNob said:

or if my common ground wire is introducing noise that affects the local SPI signals. Unfortunately, I don't have a differential probe at the moment to confirm this, but I will try to get one.

I honestly don't think power is a problem in your setup but still think it should be formally eliminated.  I have been affected by the collective IR drops lowering the supply side and raising the return side enough that the AD2437 decided it was a brown out.  Those issues were all on a heavily loaded bus and I don't think you have that.  

A-B mode of the scope with the scope ground leads on the supply ground will work well enough to decide if there's a problem there or not.  Might need to AC couple the probe channels - you just want to see if there's a blip of some sort, if you do see a big (couple of volts) then yeah you'll want to make a better measurement.

---------------

I still think the most likely issue is some software related problem and that looking at the hardware SPI line(s) might show something helpful.  Or not...