Post Go back to editing

ADSP-SC589 console (ttySC0) not working after boot

Category: Software
Software Version: Linux

Hello!

Custom Dunfell yocto build (not targeting devkit or ezkit).

After boot the /dev/ttySC0 is created, but console is not working (the earlycon gives however all kernel output to the point of getty starting)

When looking at /proc/drivers:

/dev/tty             /dev/tty        5       0 system:/dev/tty
/dev/console         /dev/console    5       1 system:console
/dev/ptmx            /dev/ptmx       5       2 system
ttyprintk            /dev/ttyprintk   5       3 console
adi-uart4            /dev/ttySC      4 64-67 serial
pty_slave            /dev/pts      136 0-1048575 pty:slave
pty_master           /dev/ptm      128 0-1048575 pty:master

Devices console and ttySC0 are created (/dev/serial0 however is NOT created):

crw--w----    1 root     tty         4,  64 Jan  1 04:16 /dev/ttySC0
crw-------    1 root     root        5,   1 Jan  1 04:11 /dev/console

Config options, following are set:

CONFIG_SERIAL_ADI_UART4
CONFIG_SERIAL_ADI_UART4_CONSOLE
CONFIG_SERIAL_DEV_BUS
CONFIG_SERIAL_DEV_CTRL_TTYPORT
CONFIG_TTY
CONFIG_UNIX98_PTYS
CONFIG_LDISC_AUTOLOAD
CONFIG_TTY_PRINTK
CONFIG_DEVMEM

We are starting the system in an initramfs file system, mounting emmc and loading up the real filesystem from emmc, doing switch_root to the real file system (as an overlay). All those parts work fine.

We have a similar setup on a sc598 64 bit, where the console works fine.

We can echo data successfully to /dev/printk.

cat /proc/devices gives:

root@sc598-devkit:~# cat /proc/devices

Character devices:
1 mem
2 pty
3 ttyp
4 /dev/vc/0
4 tty
4 ttyS
4 ttySC
5 /dev/tty
5 /dev/console
5 /dev/ptmx
7 vcs
10 misc
13 input
29 fb
89 i2c
90 mtd
128 ptm
136 pts
248 rpmb
249 ttyLP
250 bsg
251 watchdog
252 rtc
253 tpm
254 gpiochip

Block devices:
7 loop
31 mtdblock
179 mmc
259 blkext

cat /proc/tty/driver/adi-uart4 gives:
serinfo:1.0 driver revision:
0: uart:ADI-UART4 mmio:0x00000000 irq:0 tx:0 rx:0 oe:2 RTS|CTS|DTR|DSR|CD

On the sc598 build (where console works) we get:

cat /proc/tty/driver/adi-uart4  
cat /proc/tty/driver/adi-uart4
serinfo:1.0 driver revision:
0: uart:ADI-UART4 mmio:0x00000000 irq:0 tx:2689 rx:65 RTS|CTS|DTR|DSR|CD
(we can tx and rx increments as it should)

The related boot messages for the uart we get are:

ADI serial driver
adi-uart4 31003000.uart: Serial probe
31003000.uart: ttySC0 at MMIO 0x0 (irq = 0, base_baud = 7031250) is a ADI-UART4

Kernel bootargs:
clkin_hz=25000000 earlycon=adi_uart,0x31003000 console=ttySC0,115200 ignore_loglevel mem=224M clkin_hz=25000000 earlyprintk=serial,uart0,115200 ignore_loglevel mem=224M console=ttySC0,115200

In our machine configuration:
SERIAL_CONSOLE ?= "115200 ttySC0"
SERIAL_CONSOLES ?= "115200;ttySC0"

In our inittab the getty is started as:

ttySC0::respawn:/sbin/getty 115200 ttySC0

Have tried lots of varying configuration options but running out of ideas. Can anyone please help or provide clues to what goes wrong?

If needed I can provide devicetree configuration.



correcting about the boot messages
[edited by: Larswad at 8:18 AM (GMT -5) on 17 Jan 2023]
  • Hi Larswad,

    Have you tried with the default baud rate 57600, is that working?

    BTW, Could you clarify the issue a little bit,

    You have mentioned the issue for SC589 Console. But you reported the same it is working for the same SC589 by reaching the userspace. 

    How you can cat the values without reaching the console?

    root@sc598-devkit:~# cat /proc/devices

    Sorry I didn't get your query clearly.

    In addition, you can share the steps followed, complete bootlog, and dts file changes for reference. 

    Thanks,

    Prasanth R

  • Hi Prasanth!

    Thanks your answering, much grateful!

    I do realize I need to give you much more information.

    Honestly I haven't (yet) tried 57600, since we see in the driver that the default is 115200 (and our getty starts with 115200, kernel bootargs are also specifying 115200 for console=ttySC0,115200). And also the 115200 setting works on our sc598 board. But I will try that just to rule it out (and of course changing the relevant parameters while tesitng that).

    Why we can see things in userspace is because we do get network up and we can ssh in to the device, which is fine for troubleshooting. The device seems to be up in all aspects beside the console. It feels right now that we have turned every stone but can't understand the source of the problem.

    We have done some debugging in the driver, by adding some pr_info or dev_info here and there. We do get a lot of frequent calls into "adi_uart4_serial_rx_dma_timeout", but we don't know if that is a normal behavior or not (I mean, whether the timeout is normally "driving" the rx handler). We can also see that if we do like 8 or more key presses in the minicom terminal that we start to get overrun errors in "adi_uart4_serial_dma_rx_chars" (the OE bit is set).

    An interesting observation as a difference between the sc589 and sc598 boot logs is that the sc598 switches out the boot console with the real console, which can not be seen on the sc589.
    Do we somehow perhaps have two drivers conflicting with each other (the earlycon / printk simple driver and the final dma controlled driver)?

    I will here provide you with a boot log and our DTB file (we have not modified the sc58x.dtsi file or any of the other include, which are the original ones from the original release/yocto-2.1.0 branch). Our dts is much like the sc589-devkit dts, but with some removed hardware support that is not relevant for our board. I will also provide a boot log for our sc598 board that has a working console.

    I will also give you our init file for initramfs and our inittab for the real file system. You will see in our init file that we have used the kmsg to debug our way to reach to the final file system.

    I hope attachments work fine here in the forum :-)

    Best regards,

    Larswad

    https://wadefalk.se/toadi/

  • Hi Prasanth!

    Some more info:

    Tried 57600, no go. Modified all relevant parameters, same result.

    I was curious about why the following lines appeared in the 598 log and not for the 589: 

    printk: console [ttySC0] enabled
    printk: console [ttySC0] enabled
    printk: bootconsole [adi_uart0] disabled
    printk: bootconsole [adi_uart0] disabled

    Adding a driver like FSL_LPUART (that is not relevant for the adi uart itself) made those lines appear on the 589 as well and that
    was my suspicion after having followed that flow in the kernel.
    The earlycon is not switched in / bootconsole switched out unless having a driver present that will make these calls. Our 598 config
    happened to have the 8250 uart enabled which is another example of that (enabling 8250 on the 589 crashes kernel somewhere).

    But, it didn't make any difference to the problem itself, the ttySC0 is not working, like I have described, its still the same behavior.

    Can you actually confirm that the 589-devkit uar driver ever have had this working in the release/2.1.0 branch?

    I am starting to doubt everything now. :-(

    /Larswad

  • Hi, I'm working with Larswad on this same issue.

    As a test, I'm trying to disable DMA and use interrupt-based uart tx/rx.

    However, I have problems constructing a device tree that makes sense in this scenario.


    Could you provide an example for how you would write the .dts for that, you could use sc58x.dtsi as a base as we use that for our own board as well.

    It seems you would remove the dma specific nodes like:


    // dmas = <&dma_cluster2 20>, <&dma_cluster2 21>;
    // dma-names = "tx", "rx";

    But in the code there are references like this:

    uart->tx_irq = platform_get_irq_byname(pdev, "tx");
    uart->rx_irq = platform_get_irq_byname(pdev, "rx");
    uart->status_irq = platform_get_irq_byname(pdev, "status");

    How would those appear in the device tree?

  • Hi Erik,

    Kindly share your kernel configuration and the device tree file, If you have any changes in the dtsi file, share that too.

    As you were using a custom board and custom build, I am not sure how exactly the solution can be given.  

    I will check for the changes and let you know if there are any modifications that need to be done to fix this issue.

    Thanks,

    Prasanth R

  • Hi Prasanth!
    Erik and I are troubleshooting the same problem.
    In my earlier post I provided a link to the DTS and kernel config for our device and some other files like bootlogs etc. It's been granted by the forum administrators.

    The .dtsi file has no modifications from the sc58x.dtsi, it's the very same one.

  • Hi Erik & Larswad,

    As you were getting the serial console,  I don't see the issue with the uart-entries.

    Only the user-space was not reachable in th serial port. As a quick check, could you check your inittab entries? I couldn't see the Getty entries which are missing that gets us the console in the serial port.

    You can compare the inittab of SC598 & SC589, is it really missing?

    Thanks,

    Prasanth R

  • Hi again Prasanth!

    Thanks for replying and checking on your side.

    Yes, I understand that you reacted there on the inittab missing getty and I should have clarified (or sent the final file instead of the one from our yocto repo since it wasn't the actual one from the file built rd image). What we do in our recipe is that we during the build process add the getty launch with an echo statement. So actually it is there.
    In fact it is the poky busybox inittab core recipe that does this for us, we don't change that behavior.
    Sorry for giving you a misrepresentative file.

    The other only thing that sticks out in our build is that we have our own bootloader (we do not use u-boot since it is far too bloated for us and doesn't fit our upgrade strategy very well). I cannot however imagine how that would affect things, especially since that works on the sc598-based boards we have.

    Our own bootloader does make use of the uart, but with a very rudimentary initialization and logs out some minor texts before launching the kernel.

    Our own investigations hasn't yet given any results. The whole thing is utterly strange.

    Best Regards,

    Lars