Yellow not booting from NVMe

[agners]

Describe the issue you are experiencing

rolfl reported on Discord that Yellow won't boot using a SK Hynix NVMe. The green LED quickly flashes then change to solid (green and red). Update to latest Firmware did not seem to help.

The particular NVMe:

• SK hynix BC711 128GB PCIe NVMe M.2 2230 Gen 3 x 4 SSD, 0X3K2X, HFM128GD3GX013N

What operating system image do you use?

yellow (Home Assistant Yellow)

What version of Home Assistant Operating System is installed?

9.0

Did you upgrade the Operating System.

No

Steps to reproduce the issue

1. Use SK hynix BC711 128GB PCIe NVMe
2. Flash HAOS on it
3. Try starting

Anything in the Supervisor logs that might be useful for us?

No

Anything in the Host logs that might be useful for us?

No

System Health information

No response

Additional information

No response

[RobPijpers]

Several other users including myself experience the same issue, more info on the forum for example https://community.home-assistant.io/t/homme-assistant-yellow-installation-issue/474049/15

I use another brand NVMe.

[TimWeyand]

Hi @agners,
I don't have the yellow (yet), but have the same nvme problems with the waveshare baseboard a & b.

As the Problem seems to be u-boot (?). I tried to figure out how this is working and if I could try/debug some stuff. I have no clue how I can build a whole HAOS image.

I might have found one thing in the u-boot configuration - maybe it's nothing - but can you take a look?. They seem to have begun to split up NVME in December 21.

CONFIG_NVME Enable NVMe device support CONFIG_NVME_PCI Enable PCIe NVMe device support CONFIG_CMD_NVME Enable basic NVMe commands

I See a lot of changes in https://github.com/u-boot/u-boot/blob/master/configs/ changing from CONFIG_NVME to CONFIG_NVME_PCI.

Could that be by any chance the problem/solution for cm4?

(Tagging Related Issues #2075 #1887 )

[honkasalo]

I'm having exactly same issue with following hardware:

• CM4 4/16Gb
• SK Hynix BC711 256Gb nvme PCIe gen 3x4
• Waveshare CM4-IO-BASE-A

All the full number updates ending to zero (8.0, 9.0) so far have introduced boot problem, namely absolutely no boot from nvme. Had to revert back to 8.5 from 9.0, didn't even try 9.2 anymore before knowing this nvme gets fixed.

Boot problem is not limited to Yellow.

[agners]

@TimWeyand @honkasalo this issue is about booting problems specifically on Yellow. If you experience problems on other platform, please report in #1887.

To me at least it seams that the Waveboard has with significantly more NVMe's booting problems (e.g. also with Samsung models). So we should keep the list separate.

I will get a SK Hynix BC711 in November, so I plan to look into this issuue more closely next month.

[agners]

jacme31 reported on Discord: Yellow Installer boots from NVMe, but not HAOS (flashed through installer or directly). Synchronizing the Raspberry Pi firmware files (start4.elf/fixup4.dat) did not help. The drive is a Crucial P2 CT250P2SSD8.

[agners]

Riis reported WD_BLACK SN850X does not work at all (confirmed on my end) (helix report problems with WD Black SN750 SE as well, not confirmed if this was the same problem)

[agners]

Riis reported WD_BLACK SN850X does not work at all (confirmed on my end) (helix report problems with WD Black SN750 SE as well, not confirmed if this was the same problem)

I can reproduce the same issue with Raspberry Pi OS 32-bit on the CM4IO. It seems that the BCM2711 can not negotiate a PCIe link with that particular device. I created an issue in the Raspberry Pi Linux kernel repository, see raspberrypi/linux#5217.

[RobPijpers]

Riis reported WD_BLACK SN850X does not work at all (confirmed on my end) (helix report problems with WD Black SN750 SE as well, not confirmed if this was the same problem)

I had the same issue with a Crucial SSD. After replacing it with a Samsung SSD the Yellow boots normally.

[agners]

The Crucial seems to behave a bit different: Apparently Linux is able to detect it (in the installer /dev/nvme0n1 appears), see #2178 (comment).

[p6sfrx725z28knfy]

FYI SK Hynix BC501 seems to be working in the Waveshare IO board now: #1887 (comment)

[agners]

@p6sfrx725z28knfy the SK Hynix seem only to be problematic with CM4 Lite (it seems the Raspberry Pi firmware cannot boot from them). Using it with a CM4 with eMMC (booting from eMMC) and the HAOS data disk feature seems to work fine on Yellow as well (tested with the SK hynix BC711).

Maybe this can be fixed in the CM4 EEPROM firmware, I am not sure. I've opened an issue report with the Raspberry Pi team: raspberrypi/rpi-eeprom#451

[agners]

Some of the NVMe faillure cannot be resolved in HAOS, hence an issue is not appropriate. I've opened a Discussion here: #2235

I consider this issue about SK hynix BC711 (or models with the same failure mode). It seems that this can be fixed with a new firmware.

[agners]

SK hynix BC711 is fixed with the firmware update to 1.20221104. I'll backport the fix to release 9.4.