2009-09-03 03:51:54 System hangs while spi operation

Peter Gombos (HUNGARY)

Message: 79512

Hi,

We have a custom board using BF-537. Our board has:

- Hitachi TX-09 display using the Bluetechnix driver.

- AD7877 touchscreen driver

- M25P128 SPI chip

- SD card slot

- Generic ethernet phy

The release is now 2009R1RC6, the toolchain is 2009R1RC10, uboot 2009R1RC3 but observed similar behavior with the SVN trunk too.

The user applications are: a serial communication app, a sport comm. app, an LCD user interface app, BOA server, FTP, telnet, some ethernet communication apps.

All pheriperials work fine if I test only one of them. The board detects the mmcblk0(p1) device. I can use it with ext3.

The spi flash chip as mtd char device and mtd block device works fine, i can use it with jffs2.

But sometimes the system hangs without any exception and the watchdog restarts it.

I wrote simple scripts to overload the file system with create and remove files.

If I don't mount any SPI device and working in the initramfs the system it seems stable even if the user applications are running.

If I mount the MTD device and use its file system sometimes the system hangs.

If I start all of my user applications and run the test scripts the system is very unstable, very easy to crash it.

If I mount the SD card I can not start the user applications because immediatelly hangs.

I tried to debug where it is stopping. As far as I see in the file drivers/spi/spi.c in function spi_sync() the wait_for_completion(&done) sometimes do not returns.

It is a similar problem like in tracker 4443 ?

Any suggestion?

QuoteReplyEditDelete

2009-09-03 05:57:43 Re: System hangs while spi operation

Sonic Zhang (CHINA)

Message: 79522

4443 is about NAND controller on bf548. It has nothing to do with spi MMC and MTD NOR flash on bf537.

QuoteReplyEditDelete

2009-09-03 06:59:42 Re: System hangs while spi operation

Peter Gombos (HUNGARY)

Message: 79523

I know.

But the NAND driver uses the same wait_for_completion() function like spi does.

QuoteReplyEditDelete

2009-09-03 10:55:31 Re: System hangs while spi operation

Robin Getz (UNITED STATES)

Message: 79565

Peter:

Why do you think it is stuck in wait_for_completion?

I have an experimental patch which prints out the trace buffer before a reboot when a watchdog goes off if you think that might help (sometimes it does -- sometimes not).

-Robin

QuoteReplyEditDelete

2009-09-03 15:34:04 Re: System hangs while spi operation

Peter Gombos (HUNGARY)

Message: 79578

I put some printk debug in spi_sync function. The last message was the printk between the spi_async() and the wait_for_competion(). But I'm not a kernel expert. Not at all. I think I won't be able to debug it.

We are seriously thinking about to change the SD card to NAND flash because it seems unuseable now. Moreover today I could hang the system without any spi operation. I'm very disappointed about it.

Any help would be fine.

QuoteReplyEditDelete

2009-09-03 16:31:18 Re: System hangs while spi operation

Robin Getz (UNITED STATES)

Message: 79579

Peter:

I think we can help figure things out -- if we can recreate things. It sounds like it is lots of shared SPI transactions which cause your problems (SPI mtd file system on the M25P128, and SPI based SD card?)

QuoteReplyEditDelete

2009-09-03 22:27:57 Re: System hangs while spi operation

Yi Li (CHINA)

Message: 79592

Peter,

Do you have a JTAG for debugging? Using JTAG would make it easy to locate where kernel hangs.

-Yi

QuoteReplyEditDelete

2009-09-04 02:48:37 Re: System hangs while spi operation

Peter Gombos (HUNGARY)

Message: 79599

Robin,

Yes, SPI concurrency can be. But I removed all other SPI drivers from the kernel (like touchscreen and MMC) except the MTD and I can reproduce it.

Yi,

I have JTAG connector on the board and I have HPUSB-ICE. I used to use it with the Visual DSP. Can I use this device for kernel debugging? Starting dig into the toolchain documentation...

QuoteReplyEditDelete

2009-09-04 03:52:48 Re: System hangs while spi operation

Yi Li (CHINA)

Message: 79601

Peter,

Using HPUSB-ICE, you cannot debug in C source code level. But you can see the asm instructions and its address - so you can use it to find out where kernel hangs.

-Yi

QuoteReplyEditDelete

2009-09-09 10:46:15 Re: System hangs while spi operation

Peter Gombos (HUNGARY)

Message: 79713

I've made some research to catch it:

I use only a serial SPI NOR flash chip divided in 4 partitions. The mdt3 is 8 MB and empty.

I issue the following command: 'flash_eraseall -j /dev/mtd3'

It works fine. I can run it many times without the system hangs.

After I start the video-test application in the background (I have a Hitachi TX-09 display, I use the driver by Bluetechnix with minor modification)

Now trying to erase the flash using the flash_eraseall command almost immediatelly stop responding. If watchdog is enabled the system will be restarted otherwise halts forever.

Now we are working to use the JTAG or KGDB to determine where it stops. But it is not simple for me.

QuoteReplyEditDelete

2009-09-09 23:55:22 Re: System hangs while spi operation

Yi Li (CHINA)

Message: 79720

> Now we are working to use the JTAG or KGDB to determine where it stops. But it is not simple for me.

You can use JTAG (in your case HPUSB-ICE), when kernel hangs, you stop the processor and see what is the "PC". The you can use "next instruction" to trace whether this is a dead loop.

On linux side, you can use "bfin-uclinux-objdump vmlinux > kernel.dump" -- and search the above "PC" in the output file, so you know the function causing the hanging.

BTW, there is spi_lock_bus()/spi_unlock_bus() function which can be used when you want a spi device to use the spi bus exclusively for some time. But I think in this case, to find out where kernel hangs is the first thing to do.

-Yi

QuoteReplyEditDelete

2009-09-16 13:44:52 Re: System hangs while spi operation

Peter Gombos (HUNGARY)

Message: 80027

Finally solved!

I've turned off the CONFIG_EXACT_HWERR option in kernel hacking and everything works fine. The problem is the same as in the tracker 5430.

I can use spi touchscreen driver, MTD driver, PPI LCD driver and SPORT communication driver. Hopefully I won't have any problem with SPI MMC driver.

QuoteReplyEditDelete

2009-09-16 14:35:07 Re: System hangs while spi operation

Robin Getz (UNITED STATES)

Message: 80030

Peter:

Which version of 2009 are you using - the svn should have the fix in it (I put it in svn rev7237 26Aug) - but it looks like it is not in the latest -RC6 (it was found/fixed after the RC6 tag) -- I don't know if Sonic was planning on making an RC7 or not.

-Robin

QuoteReplyEditDelete

2009-09-16 15:00:54 Re: System hangs while spi operation

Mike Frysinger (UNITED STATES)

Message: 80033

i dont think there are any plans for more RC's. at this point we're doing bug fixes for a possible 2009R1.5 down the road.

QuoteReplyEditDelete

2009-09-16 15:32:15 Re: System hangs while spi operation

Peter Gombos (HUNGARY)

Message: 80035

Robin,

I use the latest official release, 2009R1RC6.