2009-09-03 03:51:54 System hangs while spi operation
Peter Gombos (HUNGARY)
Message: 79512
Hi,
We have a custom board using BF-537. Our board has:
- Hitachi TX-09 display using the Bluetechnix driver.
- AD7877 touchscreen driver
- M25P128 SPI chip
- SD card slot
- Generic ethernet phy
The release is now 2009R1RC6, the toolchain is 2009R1RC10, uboot 2009R1RC3 but observed similar behavior with the SVN trunk too.
The user applications are: a serial communication app, a sport comm. app, an LCD user interface app, BOA server, FTP, telnet, some ethernet communication apps.
All pheriperials work fine if I test only one of them. The board detects the mmcblk0(p1) device. I can use it with ext3.
The spi flash chip as mtd char device and mtd block device works fine, i can use it with jffs2.
But sometimes the system hangs without any exception and the watchdog restarts it.
I wrote simple scripts to overload the file system with create and remove files.
If I don't mount any SPI device and working in the initramfs the system it seems stable even if the user applications are running.
If I mount the MTD device and use its file system sometimes the system hangs.
If I start all of my user applications and run the test scripts the system is very unstable, very easy to crash it.
If I mount the SD card I can not start the user applications because immediatelly hangs.
I tried to debug where it is stopping. As far as I see in the file drivers/spi/spi.c in function spi_sync() the wait_for_completion(&done) sometimes do not returns.
It is a similar problem like in tracker 4443 ?
Any suggestion?
QuoteReplyEditDelete
2009-09-03 05:57:43 Re: System hangs while spi operation
Sonic Zhang (CHINA)
Message: 79522
4443 is about NAND controller on bf548. It has nothing to do with spi MMC and MTD NOR flash on bf537.
QuoteReplyEditDelete
2009-09-03 06:59:42 Re: System hangs while spi operation
Peter Gombos (HUNGARY)
Message: 79523
I know.
But the NAND driver uses the same wait_for_completion() function like spi does.
QuoteReplyEditDelete
2009-09-03 10:55:31 Re: System hangs while spi operation
Robin Getz (UNITED STATES)
Message: 79565
Peter:
Why do you think it is stuck in wait_for_completion?
I have an experimental patch which prints out the trace buffer before a reboot when a watchdog goes off if you think that might help (sometimes it does -- sometimes not).
-Robin
QuoteReplyEditDelete
2009-09-03 15:34:04 Re: System hangs while spi operation
Peter Gombos (HUNGARY)
Message: 79578
I put some printk debug in spi_sync function. The last message was the printk between the spi_async() and the wait_for_competion(). But I'm not a kernel expert. Not at all. I think I won't be able to debug it.
We are seriously thinking about to change the SD card to NAND flash because it seems unuseable now. Moreover today I could hang the system without any spi operation. I'm very disappointed about it.
Any help would be fine.
QuoteReplyEditDelete
2009-09-03 16:31:18 Re: System hangs while spi operation
Robin Getz (UNITED STATES)
Message: 79579
Peter:
I think we can help figure things out -- if we can recreate things. It sounds like it is lots of shared SPI transactions which cause your problems (SPI mtd file system on the M25P128, and SPI based SD card?)
?
QuoteReplyEditDelete
2009-09-03 22:27:57 Re: System hangs while spi operation
Yi Li (CHINA)
Message: 79592
Peter,
Do you have a JTAG for debugging? Using JTAG would make it easy to locate where kernel hangs.
-Yi
QuoteReplyEditDelete
2009-09-04 02:48:37 Re: System hangs while spi operation
Peter Gombos (HUNGARY)
Message: 79599
Robin,
Yes, SPI concurrency can be. But I removed all other SPI drivers from the kernel (like touchscreen and MMC) except the MTD and I can reproduce it.
Yi,
I have JTAG connector on the board and I have HPUSB-ICE. I used to use it with the Visual DSP. Can I use this device for kernel debugging? Starting dig into the toolchain documentation...
QuoteReplyEditDelete
2009-09-04 03:52:48 Re: System hangs while spi operation
Yi Li (CHINA)
Message: 79601
Peter,
Using HPUSB-ICE, you cannot debug in C source code level. But you can see the asm instructions and its address - so you can use it to find out where kernel hangs.
-Yi
QuoteReplyEditDelete
2009-09-09 10:46:15 Re: System hangs while spi operation
Peter Gombos (HUNGARY)
Message: 79713
I've made some research to catch it:
I use only a serial SPI NOR flash chip divided in 4 partitions. The mdt3 is 8 MB and empty.
I issue the following command: 'flash_eraseall -j /dev/mtd3'
It works fine. I can run it many times without the system hangs.
After I start the video-test application in the background (I have a Hitachi TX-09 display, I use the driver by Bluetechnix with minor modification)
Now trying to erase the flash using the flash_eraseall command almost immediatelly stop responding. If watchdog is enabled the system will be restarted otherwise halts forever.
Now we are working to use the JTAG or KGDB to determine where it stops. But it is not simple for me.
QuoteReplyEditDelete
2009-09-09 23:55:22 Re: System hangs while spi operation
Yi Li (CHINA)
Message: 79720
> Now we are working to use the JTAG or KGDB to determine where it stops. But it is not simple for me.
You can use JTAG (in your case HPUSB-ICE), when kernel hangs, you stop the processor and see what is the "PC". The you can use "next instruction" to trace whether this is a dead loop.
On linux side, you can use "bfin-uclinux-objdump vmlinux > kernel.dump" -- and search the above "PC" in the output file, so you know the function causing the hanging.
BTW, there is spi_lock_bus()/spi_unlock_bus() function which can be used when you want a spi device to use the spi bus exclusively for some time. But I think in this case, to find out where kernel hangs is the first thing to do.
-Yi
QuoteReplyEditDelete
2009-09-16 13:44:52 Re: System hangs while spi operation
Peter Gombos (HUNGARY)
Message: 80027
Finally solved!
I've turned off the CONFIG_EXACT_HWERR option in kernel hacking and everything works fine. The problem is the same as in the tracker 5430.
I can use spi touchscreen driver, MTD driver, PPI LCD driver and SPORT communication driver. Hopefully I won't have any problem with SPI MMC driver.
QuoteReplyEditDelete
2009-09-16 14:35:07 Re: System hangs while spi operation
Robin Getz (UNITED STATES)
Message: 80030
Peter:
Which version of 2009 are you using - the svn should have the fix in it (I put it in svn rev7237 26Aug) - but it looks like it is not in the latest -RC6 (it was found/fixed after the RC6 tag) -- I don't know if Sonic was planning on making an RC7 or not.
-Robin
QuoteReplyEditDelete
2009-09-16 15:00:54 Re: System hangs while spi operation
Mike Frysinger (UNITED STATES)
Message: 80033
i dont think there are any plans for more RC's. at this point we're doing bug fixes for a possible 2009R1.5 down the road.
QuoteReplyEditDelete
2009-09-16 15:32:15 Re: System hangs while spi operation
Peter Gombos (HUNGARY)
Message: 80035
Robin,
I use the latest official release, 2009R1RC6.
I don't know how complicated is to create a new RC but it was a 3 month long suck for me.
When do you plan to release the 2009R1.5?
QuoteReplyEditDelete
2009-09-16 16:38:52 Re: System hangs while spi operation
Mike Frysinger (UNITED STATES)
Message: 80036
you can simply grab the latest svn branch and replace the linux-2.6.x dir in the release with it. things are completely compatible.