2009-09-25 15:49:22     SD MMC interface stops working after several weeks!

Document created by Aaronwu Employee on Aug 19, 2013
Version 1Show Document
  • View in full screen mode

2009-09-25 15:49:22     SD MMC interface stops working after several weeks!

Tim Shearer (UNITED STATES)

Message: 80458   

 

This is a strange one.

 

I'm using a BF537, and the mainline SPI MMC driver in 2009R1 (built-in).  The SD cards (FAT16) always mount properly at power-up, every single time, every product.

 

The devices they are installed in use the cards for logging operational data, so they're not getting hammered.

 

After two or three weeks in the field, the Blackfins started "losing" their cards.  They reboot automatically, but the SD card is always completely unreconized.  So no MMC block devices are listed in /sys.  This has happened for about 50 out of 100 units in the field over a 6 week period.

 

Interestingly, although a reboot never resolves the issue, power-cycling the processor does.  Even exchanging the SD card with a new one without removing the power, and rebooting, doesn't help.

 

I recovered a card from a failed unit, and it mounts fine with another Blackfin.  The card isn't full or corrupt, and has read/write permissions.

 

Additional information:

dmesg from a failing unit shows the normal

  mmc_spi spi0.2: requested mode not fully supported

  mmc_spi spi0.2: can't change chip-select polarity

 

 

but doesn't show the usual:

  mmc0: new SD card on SPI

  mmcblk0: mmc0:0000 SD02G 1.89 GiB

  mmcblk0: p1

 

 

I should point out that i disabled the interrupt on the SD detect line in the board.c file since we don't need hotplugging and the cards are always present::

 

#define MMC_SPI_CARD_DETECT_INT NULL

 

Since the issue is only resolved by power-cycling the processor, I suspect I may have a corrupt hardware register which isn't configured by either u-boot or the Linux initialization process.  Possibly something IRQ related, since it appears that the kernel isn't even attempting to talk to the card.

 

If anyone has any thoughts then please let me know...  particularly with regard to interrogating hardware register values from within uClinux.  I have plenty of failed units but they're all in the field, and I'd like to be able to recover them without having to visit each one!

 

Many thanks,

 

Tim

QuoteReplyEditDelete

 

 

2009-09-28 12:58:52     Re: SD MMC interface stops working after several weeks!

Tim Shearer (UNITED STATES)

Message: 80572   

 

Here's some more data, with MMC debugging enabled:

 

I haven't dug deep in the driver code yet, but I'm initiallly surprised that it's possible to get an ETIMEDOUT with an SPI interface, since the master can always reads back something.

 

Interestingly, it appears to the driver as if there is no SD card on the bus, even though there is.  I'm going to attempt to blitz all the related hardware registers with their power on reset values, to see if that brings it back to life.

 

Thanks,

Tim

 

mmc_spi spi0.2: ASSUMING 3.2-3.4 V slot power

mmc0: clock 0Hz busmode 0 powermode 0 cs 0 Vdd 0 width 0 timing 0

mmc_spi spi0.2: SD/MMC host mmc0, no DMA, no WP, no poweroff

bfin-spi bfin-spi.0: Blackfin on-chip SPI Controller Driver, Version 1.0, regs_base@ffc00500, dma channel@7

mmc0: clock 0Hz busmode 2 powermode 1 cs 1 Vdd 21 width 0 timing 0

mmc_spi spi0.2: mmc_spi: power up (21)

mmc0: clock 400000Hz busmode 2 powermode 2 cs 1 Vdd 21 width 0 timing 0

mmc_spi spi0.2: mmc_spi: power on (21)

mmc_spi spi0.2: requested mode not fully supported

mmc_spi spi0.2: can't change chip-select polarity

mmc_spi spi0.2: mmc_spi:  clock to 400000 Hz, 0

mmc0: starting CMD0 arg 00000000 flags 000000c0

mmc_spi spi0.2:   mmc_spi: CMD0, resp R1

mmc0: req done (CMD0): -110: 00000000 00000000 00000000 00000000

mmc0: starting CMD8 arg 000001aa flags 000002f5

mmc_spi spi0.2:   mmc_spi: CMD8, resp R3/R4/R7

mmc0: req done (CMD8): -110: 00000000 00000000 00000000 00000000

mmc0: starting CMD5 arg 00000000 flags 000002e1

mmc_spi spi0.2:   mmc_spi: CMD5, resp R3/R4/R7

mmc0: req failed (CMD5): -110, retrying...

mmc_spi spi0.2:   mmc_spi: CMD5, resp R3/R4/R7

mmc0: req failed (CMD5): -110, retrying...

mmc_spi spi0.2:   mmc_spi: CMD5, resp R3/R4/R7

mmc0: req failed (CMD5): -110, retrying...

mmc_spi spi0.2:   mmc_spi: CMD5, resp R3/R4/R7

mmc0: req done (CMD5): -110: 00000000 00000000 00000000 00000000

mmc0: starting CMD55 arg 00000000 flags 000000f5

mmc_spi spi0.2:   mmc_spi: CMD55, resp R1

mmc0: req done (CMD55): -110: 00000000 00000000 00000000 00000000

mmc0: starting CMD55 arg 00000000 flags 000000f5

mmc_spi spi0.2:   mmc_spi: CMD55, resp R1

mmc0: req done (CMD55): -110: 00000000 00000000 00000000 00000000

mmc0: starting CMD55 arg 00000000 flags 000000f5

mmc_spi spi0.2:   mmc_spi: CMD55, resp R1

mmc0: req done (CMD55): -110: 00000000 00000000 00000000 00000000

mmc0: starting CMD55 arg 00000000 flags 000000f5

mmc_spi spi0.2:   mmc_spi: CMD55, resp R1

mmc0: req done (CMD55): -110: 00000000 00000000 00000000 00000000

mmc0: starting CMD1 arg 00000000 flags 000000e1

mmc_spi spi0.2:   mmc_spi: CMD1, resp R1

mmc0: req done (CMD1): -110: 00000000 00000000 00000000 00000000

mmc0: clock 0Hz busmode 2 powermode 0 cs 1 Vdd 0 width 0 timing 0

mmc_spi spi0.2: mmc_spi: power off (0)

QuoteReplyEditDelete

 

 

2009-09-28 13:05:38     Re: SD MMC interface stops working after several weeks!

Robin Getz (UNITED STATES)

Message: 80574   

 

Tim:

 

Does that mean you can re-create the problem without waiting 6 weeks?

QuoteReplyEditDelete

 

 

2009-09-28 13:10:28     Re: SD MMC interface stops working after several weeks!

Tim Shearer (UNITED STATES)

Message: 80575   

 

I can recreate the problem by simply rebooting the kernel, or reinstalling the driver (if compiled as a module), without powering it down.  If I power-cycle the processor, the card successfully mounts.

QuoteReplyEditDelete

 

 

2009-09-28 14:11:44     Re: SD MMC interface stops working after several weeks!

Robin Getz (UNITED STATES)

Message: 80576   

 

Tim:

 

So, it is boot a kernel - install the module - do some activity(?) remove the module, insmod the module (for the 2nd time) - and things now don't work?

 

-Robin

QuoteReplyEditDelete

 

 

2009-09-28 14:51:41     Re: SD MMC interface stops working after several weeks!

Tim Shearer (UNITED STATES)

Message: 80579   

 

Hi Robin,

 

Just to clarify:

 

I have BF537 systems in the field that have suddenly lost the ability to mount their SD cards, though not all of them.  I have remote access to the units over SSH.

 

Rebooting fails to fix the driver.  Replacing a card (without powering down) also doesn't fix it.  Remotely upgrading the filesystem with the SPI and MMC drivers built as modules allows me to install and remove the modules at will.  However, it makes no difference, the driver can't initialize the card.

 

So far, it appears that the only way I can resolve the issue is to power-cycle each unit... this isn't a desirable option, as they are distributed all across the country.

 

My development system works fine.  However it displays exactly the same symptoms when I detach any of the SPI signals.  i.e. it results in -110 ETIMEDOUT errors.

 

I am currently using the MMR debugfs to identify any hardware registers that are configured differently in the failing systems to my working system.

 

Thanks,

Tim

QuoteReplyEditDelete

 

 

2009-09-28 14:58:17     Re: SD MMC interface stops working after several weeks!

Robin Getz (UNITED STATES)

Message: 80581   

 

Tim:

 

Thanks for the clarification  - but the question still remains - is there a easy way to replicate the issue for Yi? (Other than detaching the SPI signals - which of course will stop things from working).

 

-Robin

QuoteReplyEditDelete

 

 

2009-09-28 15:03:32     Re: SD MMC interface stops working after several weeks!

Tim Shearer (UNITED STATES)

Message: 80583   

 

Hi Robin,

 

Unfortunately not at the moment.  If I can find anyway to reproduce whatever is causing this, you will be the first to know.

 

If there is anything you'd like me to try, then I'd obviously be happy to oblige.  I will let you know if I find anything from my MMR investigation.

 

My next step is to visit one of the failed units, armed with a scope, and to actually see which (if any) of the signals are actually missing.

 

Thanks,

 

Tim

QuoteReplyEditDelete

 

 

2009-09-28 22:38:12     Re: SD MMC interface stops working after several weeks!

Yi Li (CHINA)

Message: 80593   

 

Maybe you can also enable debug output in spi_bf5xx.c, and check the spi registers - see if there is differece between reboot and power cycle.

 

-Yi

QuoteReplyEditDelete

 

 

2009-09-29 05:47:00     Re: SD MMC interface stops working after several weeks!

Wolfgang Muees (GERMANY)

Message: 80633   

 

Tim,

 

you should look at the GPIO registers, and at the drive strength registers.

 

You can even enable the GPIO input registers (this is allowed everytime!) and then you can read the state of both input and output pins.

 

I suspect that you have a sort of dangling pointer that change some bits in the CPU registers...Good luck!

 

best regards

 

Wolfgang

TranslateQuoteReplyEditDelete

 

 

2009-09-29 09:36:44     Re: SD MMC interface stops working after several weeks!

Tim Shearer (UNITED STATES)

Message: 80637   

 

Thanks for the suggestions guys, I'll be back onto this tomorrow so I'll keep you posted.

 

Tim

QuoteReplyEditDelete

 

 

2009-10-01 16:13:52     Re: SD MMC interface stops working after several weeks!

Tim Shearer (UNITED STATES)

Message: 80778   

 

Some more information:

 

The previous information  was incorrect.  It turns out that power-cycling the SD card - not the Blackfin, brings it back to life.  From watching the SPI lines on a scope, it was apparent that the SD card (Sandisk Ultra-II 2GB) wasn't replying to any of the intiialization commands - it stays high permenantly.

 

Somehow, the card is locking up.  As long as I can unlock it without power-cycling, then I'm happy.  I'm going to spend some time looking through the Sandisk MMC command set and see what I can do.

 

Thanks,

 

Tim

QuoteReplyEditDelete

 

 

2009-11-06 13:12:48     Re: SD MMC interface stops working after several weeks!

Tim Shearer (UNITED STATES)

Message: 82138   

 

Just a follow-up report in case anyone finds this in a search.

 

A batch of 2GB SanDisk ULTRAII cards seem to be entering a state equivilent to the SD specification's "incactive state."  Sending continuous stream of wake-up clocks and CMD0 commands makes no difference, the card does not respond in any shape or form until it is power-cycled.

 

How it is entering this mode, I don't know.  I am guessing the card sould be reacting to a brown-out.  The SD command that puts the card in inactive state is invalid in SPI mode, so it shouldn't be possible even if we wanted to do it.

 

Either way, we're soak-testing some equivilent spec Kingston cards, so fingers crossed!

 

Thanks,

 

Tim

QuoteReplyEditDelete

 

 

2009-11-06 16:12:33     Re: SD MMC interface stops working after several weeks!

Robin Getz (UNITED STATES)

Message: 82139   

 

Tim:

 

Do you have an extra SanDisk card that we could use to try and figure this out?

 

-Robin

QuoteReplyEditDelete

 

 

2010-03-16 14:36:15     Re: SD MMC interface stops working after several weeks!

Ian Jeffray (UNITED KINGDOM)

Message: 87284   

 

Hi Tim - what was your conclusion to this?  A duff batch of cards, or did you implement some change to stop this happening?

Attachments

    Outcomes