Yocto Linux build version 5.0.0 on ADSP-SC594 EZKIT - system freezes

Hi hkhauke ,

Is this stock yocto/linux with nothing else running whatsover?

Additionally, could you try the following steps with yocto 5.0.0:

- Load uboot via gdb as usual

- Once in uboot shell, run "run erase_spi"

NOTE: This will take some time, do not worry if there is no output. The process should take about 10-15mins.

- run "env default -a"
- run "run update_spi"

Alternatively, you could also attempt using the OSPI: github.com/.../Getting-Started-with-ADSP‐SC594-(Linux-for-ADSP‐SC5xx-Processors-5.0.0)

Hi,

thank you for your response.
I am using the stock yocto/linux - following in detail what is described on the website for version 5.0.0 Yocto/Linux build on a Ubuntu 24.04 host PC.

I tried what you proposed:

run erase_spi
env default -a
run update_spi

Then switch boot switch to option 1 and reset. The system boots as expected and is then up and running. I then left the room and after 30 minutes, the watchdog had restarted the device. Hence, problem is still present even though the hang seems not to start immediately. A few days ago my feeling was that the uptime is shorter the longer the board is on, therefore, I added a cooling capability on the device - without any real impact.

I also tried to run the OSPI option. However, I am not able to store the overall system on OSPI since the size of the image is too large (seems that OSPI is only 32 MB large).

By the way, the behavior of the linux is also kind of strange: Typing, e.g., "ip a" does not return unless I push return. Could also be that something is broken in the serial output of the Linux with version 5.0.0.

Thank you and best regards

hkhauke

hkhauke said:

I also tried to run the OSPI option. However, I am not able to store the overall system on OSPI since the size of the image is too large (seems that OSPI is only 32 MB large).

You can store the file system on the qspi, if you have another media present like a USB stick, that should work too. You can then run "run update_ospi_uboot_only" and only flash uboot onto the ospi and boot with bootmode 5 that way. The default command that is executed on autoboot or just "boot" is set at "bootcmd". So if you wanted to mix and match boot modes, you could give that a try as well.

hkhauke said:

Hence, problem is still present even though the hang seems not to start immediately.

As for the system after booting with qspi, we will try to reproduce on our side.

hkhauke said:

By the way, the behavior of the linux is also kind of strange: Typing, e.g., "ip a" does not return unless I push return. Could also be that something is broken in the serial output of the Linux with version 5.0.0.

Hmm, about the serial console, it sounds a bit strange. From what I understand, you type the command out - no delay, working as expected, press return - no response and return again - after which you get a response?

Is there any kernel panic or kernel crashes? Any logs to share?

I just got the system booting from USB external drive - memory stick. This works for version 5.0.0 BUT - again system freezes after a short moment and is not useable.

However, I did the same for version 3.1.2 and it works. Hence, for me to work with version 3.1.2 booting from USB memory stick is a good option for now. Hopefully all features for communication between DSPs and ARM core also work with version 3.1.2...

Searching this for a while, I finally found the option to override the "bootcmd" expression in uboot, but now the systems boots straight through which was the desired goal for the next development issues.

Regarding the serial console with version 5.0.0: I type "ip a". Then the information is shown but prompt is not back. Then I push return and see prompt twice. It seems hence that the first "return" after "ip a" was stuck.

can you try to run "run ramboot" from u-boot and if you experience same issue? I have closer look at this issue

I just running Yocto-5 from my current build and I can't reproduce same behaviour, so will need to get clean build and see if I can reproduce 

Yes, I started to "run ramboot" a few hours ago. The system was still stable after 6 hours - looks good. It does not show the weird console behavior reported earlier.

I could provide you with those six files (uboot, fitImage, root filesystem) which are stored in my /tftpboot folder for version 5.0.0 to give it a quick try.

Another update on this issue: I cross combined rootfs and kernels:

1) Rootfs from version 5.0.0, Kernel from version 3.1.2 -> System runs properly and stable for at least 30 minutes
2) Rootfs from version 3.1.2, kernel from version 5.0.0 -> System boots and hangs immediately

No log can be reviewed as the system freezes without any verbose output. I cannot even logon using ssh as the system does not live long enough to view the IP address.

Also, I have run the same tests on a second board today - same results, unfortunately. 

From these tests I conclude that the problem is related to the kernel itself and not a hw issue.

Best regards

Hi, I removed the ethernet from device-tree for Yocto-5 and spi boot seems stable. Will need to spend more time on this and see root cause of issue. 

Hi, I removed the ethernet from device-tree for Yocto-5 and spi boot seems stable. Will need to spend more time on this and see root cause of issue. 

We just found that some of the patches for the Linux kernel Yocto recipe (device tree) are applied for SC598 in case of machine type SC594. In this case, the patches will not be applied for SC594 since SC598 is ARM64 and SC594 is ARM(32) - hence the device trees are not modified. Maybe a hint for what goes wrong..

Heya, have created PR with the fix for SC594 https://github.com/analogdevicesinc/linux/pull/2966 I tried to run overnight with nfsboot and ramboot. Haven't observed any issues with stability anymore. 

Hi ArtursArtamonovs, very good work, thank you very much for that fix.

I wonder: I tried to directly change the yocto files to include your version of the kernel. However, the rev id in the branch of the "meta-adi" layer currently points to version "2d5498652adadf55189c4f1b2d591b8db49e59be" for the linux recipe - which is an older version of the adsp-main-6.12 branch. When I switch that to a current version - e.g., commit id "530b3a667811cc9bbac47f7127996694b38cd1e5" - of branch adsp-main-6.12 in the linux recipe the Yocto build fails since the patches located in the "meta-adi" Yocto layer can not be employed properly. I am using the "main" branch of the "meta-adi" layer, is there another branch I could checkout to obtain the latest Yocto patches and allow to directly involve your patch?

Hi, yes as its PR its now located at branch adsp-main-6.12-sc594-ethernet-dma 
When it will be approved and merged it will be located at adsp-main-6.12

Perfect, thank you very much - PR is done and closed.

However, still there is the problem I reported earlier: 

In order to build the system with your latest patch, I need to modify the file "meta-adi/meta-adi-adsp-sc5xx/recipes-kernel/linux/linux-adi_%.bb" in the meta-adi Yocto layer to checkout the latest SRCREV:


That is the revision which was born today due to the PR.

When doing this, unfortunately, the Yocto build fails:



This makes absolutely sense since between the old revision "2d5498.." and the new version "605ec077..", the *.dtsi file has been changed according to the commit history in Github. In order to fix this, it seems that an update of the meta-adi layer is also required to update the patch files.

Have cloned the repo and applied patch without issue. Build Yocto-5.0.0 with setting for CRR_REV "A" and "D" couldn't reproduce observed issue. 

managed to reproduce, Im using latest dev sources, and its not issue there. Add the proposed changes as patch from PR https://patch-diff.githubusercontent.com/raw/analogdevicesinc/linux/pull/2966.patch and use release commit hash. This fix will go most likely into release 5.0.x

I have checked out version 5.0.0 from scratch and applied a few handwritten patches. Now it seems that I can build kernel 6.12. in the latest version. The problem was related mostly to patches in meta-adi layer that fix issues in the older kernel versions that are integrated in 6.12 anyway and do not need to be fixed anymore. Hence, removing the patches solved most issues.

Running the new kernel in a quick test seemed to be stable now. I will test that more intensively later.

One thing I quickly saw was that the mechanisms to load DSP firmware seems to have gone. In version 3.1.2 with the older kernel, there were entries in the file system, /sys/class/remoteproc/remoteproc0 and /sys/class/remoteproc/remoteproc1. In version 5.0.0, the folder /sys/class/remoteproc is empty. Does this mechanism require another kernel config parameter in version 5.0.0 or does it work somehow in a different way?

Thank you and best regards 

Have verified on my side. Let me look into that. 

Perfect, thank you very much for your efforts.

I ran the kernel 6.12 for a longer time today. It seems principally stable - I could not observe any instable crashes or freezes. That is very good .

However, the console output behavior is still a little annoying - it seems to be stuttering. E.g., when running dmesg, the console output stalls after a few lines. I then need to push return to obtain the next bunch of lines. Also, a "top" only fills a few lines.

I then found that the serial console stuttering behavior seems to have a longer history with the latest kernel and tried many different approaches to fix it. Nothing could improve the console output. The last test was to add option "printk.threaded=0" and "printk.synchronous=1" in the boot args in uboot - with no change of behavior.

This behavior was not observed in the 5.15 kernel. Is there a known workaround I could try to fix the behavior for the kernel 6.12?