SPORT DMA error immediately after enabling SPORT, unless a breakpoint is set at DMA enable line

Using ADSP21569.

Using just 1 SPORT, SPORT0B, as RX with DMA1. Interrupt handler is installed, SCP stuff is setup (SPORT0B and SPORT0B DDE).

Things works when running a debug session only when a breakpoint is set at this line:

*pREG_DMA1_CFG |= BITM_DMA_CFG_EN; // Breakpoint here

If a breakpoint is not set, then as soon as SPORT0B is enabled, I get a DMA error, ERRC = 3 (Memory Access or Fabric Error).

Again: as long as there is a breakpoint at that line, as soon as it stops on that breakpoint and immediately hit Resume, everything runs without issue. If I run from the start of the debug session without the breakpoint, the error occurs.

I tried putting some delay before the instruction that needs to stop on the breakpoint, and that didn't make a difference. I tried re-ordering things to see if it made any difference, but nothing helped.

I am really at a loss as to what is going on here. Any ideas? Thanks!

[edited by: InoJosh at 12:51 AM (GMT -5) on 20 Nov 2020]