sdhost: command MMC_SEND_EXT_CSD can unmount rootfs

[lategoodbye]

During my investigation about this issue with the upstream version of sdhost, i've found a problem which affects downstream (bcm2835-sdhost.c) and upstream (bcm2835.c).

Preparation:

• Install mmc-utils on Raspbian SD card ( mmc-utils_0+git20180327.b4fe0c8c-1_armhf.deb )

Steps to reproduce on Raspberry Pi 3B:

• run the following command a few times sudo mmc extcsd read /dev/mmcblock0 (since this is timing critical it need some attempts)

Expected results:

• timeout of command MMC_SEND_EXT_CSD since this is a SD card not eMMC

Experienced results:

• at least after 10 attempts the root partition is unmounted which is really bad

Affected kernel versions:

• rpi-4.14.78
• rpi-4.19
• 4.19 (upstream)

Current investigation results:
Debugging this issue was a real pain because most of the commands depends on the rootfs. But i think i narrowed it down to that point that the command MMC_SEND_EXT_CSD confuses the driver / controller. The command MMC_SEND_EXT_CSD fails with an timeout BEFORE the controller FSM has finished. So the next valid command to the SD card could fail with

previous command (8) not complete (EDM 00010004)

after this strange IO errors occur, the ext4 journal is aborted and the rootfs isn't available. The sdhost isn't able to break free from this situation.

As a band-aid for the upstream driver i made the following patch. This avoids the issue but is probably not the best solution. I assume you have (more) documentation about the sdhost controller to identify what's wrong here.

Edit: I apologize but the patch wasn't the right one. Fixed

[lategoodbye]

Additionally here is a screenshot after the issue occured on a Raspberry Pi 3B:

[pelwell]

Thanks for the debug effort on this issue. Using an NFS root partition takes most of the pain out of SD driver errors, and network booting further streamlines the process.

This is not an error I've encountered before since the normal MMC subsystem will avoid sending MMC_SEND_EXT_CSD to an SD card unless forced to. You are correct that we do have at least some documentation for the SDHOST interface, but it wasn't sufficient to stop the writing of the original downstream driver being an exercise in trial and error, and this fix is more of the same.

The active ingredient in your workaround isn't the waiting - without prodding, the controller will remain in the READWAIT state indefinitely - it's setting the FORCE_DATA_MODE bit, which is enough to get the FSM out of the rut.

See #2729 for two possible fixes.

[pelwell]

With those patches applied I can run the offending command in a loop with no apparent ill effect.

[lategoodbye]

Your patch looks nice.

Btw i tried my patched upstream version against the original issue from Ognian, but this one isn't fixed. The system seems to hang (heartbeat LEDs blinks fast).

Here is the source code to provoke the initial issue:
https://github.com/Ognian/sdmon.git

Usage:
sudo sdmon /dev/mmcblk0

[lategoodbye]

Okay, after applying ed6ceb0 and https://gist.github.com/pelwell/265baccf92ea4f2d8e47c4f9e1655714 on top of upstream 4.19, the MMC_SEND_EXT_CSD doesn't cause harm anymore on Raspberry Pi 3.

In case of sdmon i'm getting timeout waiting for hardware interrupt for a SD card which probably doesn't support this command.

Here is the relevant dmesg output in case DMA is enabled:
dmesg-4.19-CMD56-dma.txt
Here is the relevant dmesg output in case DMA is disabled:
dmesg-4.19-CMD56-nodma.txt

[lategoodbye]

I also tested sdmon once with current 4.14 723af8f

[  267.988862] mmc0: timeout waiting for hardware interrupt.
...
[  278.228798] mmc0: timeout waiting for hardware interrupt.
...
[  278.236320] bcm2835-dma 3f007000.dma: DMA transfer could not be terminated
[  278.236522] mmcblk0: timed out sending r/w cmd command, card status 0x400b00
[  278.236526] mmcblk0: command error, retrying timeout

dmesg-4.14.78-CMD56-dma.txt