Raspberry Pi 4B reboot issue, 3.3V pins go to 0V and stay on reboot with HAT attached

[gtrainavicius]

It seems the 3.3V pins of Raspberry Pi 4 (pins 1 and 17) briefly go to 0V during reboot, and they go low until unplugged when the system is shutdown. This seems to be a new behavior of Raspberry Pi 4 - I couldn't reproduce the same on 3B+. 3B+ board has rock solid 3.3V during the same shutdown and reboot scenarios.

With Pisound board attached, during reboot, the 3.3V pins get stuck at about 0.2V. This is also bad for the board itself - it uses the 3.3V for some digital chips, while deriving its own 3.3V level from 5V pins for analog chips. As 5V remains at ~5V during the entire time, 3.3V pins going to 0V even briefly could end up causing damage. If 5V would get switched off too, then I think it'd be OK to power-cycle the entire GPIO header, but switching only 3.3V off seems like a dangerous thing to do.

Once the issue occurs, the 3.3V rail gets unstuck only after leaving RPi4 unplugged for about 15 seconds. (In this case, I was leaving the USB-C connector attached to RPi4, and unplugging it from the outlet)

I have measured the 5V and 3.3V rails of Raspberry Pi 3B+ and 4, here are the Oscilloscope screenshots of the result:

1. Raspberry Pi 3B+ with the official USB power supply, Pisound attached. Ethernet cable was plugged in, and nothing else was connected.

   

   The 3.3V stayed very stable when rebooting and shutting down the system, and remained at 3.3V at all times.

2. Raspberry Pi 4 (4GB version) with the official USB C power supply, with Pisound attached. Ethernet cable was plugged in, and nothing else was connected.

   

   To reboot the device, I ssh'ed to RPi4 and ran sudo reboot. The oscillogram shows the 3.3V went low for a long time and didn't recover. The system does not boot from this state.

3. Same setup, except Pisound detached.

   

   Ssh'ed, ran sudo reboot, and without anything attached, the 3.3V line went low for ~60ms, and then the system booted up again.

To reproduce
Reboot on Raspberry Pi 4, monitor the 3.3V line.

Expected behaviour
3.3V remains stable at all times and does not go to 0V even on shutdown, especially if 5V remains at 5V.

System

Raspinfo.txt

Running kernel 4.19.57-v7l+, on RPi4, I built it on my own locally, it's unmodified from the main branch of this repository, following the steps here. On RPi3B+, running the 4.19.57-v7+, which I got using sudo rpi-update.

VL805 FW version: 000137a8

[arisoft-fi]

I have reported a similar effect on USB VBUS power of RPPi3B+ when rebooted. (issue #3058)
I recorded a video from the console during reboot to find out when the power was actually cut off. In my case it happened when platform devices were reinitialized by the kernel. You may add loglevel=7 to the cmdline.txt before trying this.

[pelwell]

That's a completely separate issue - don't cross the streams.

[timg236]

The halt behaviour can be changed by setting this flag in the EEPROM configuration section

Download the bootloader from https://www.raspberrypi.org/downloads/ then modify the flag as follows and do the normal EEPROM update.

sed -i -e "s/WAKE_ON_GPIO=0/WAKE_ON_GPIO=1/" pieeprom.bin

I think it's very likely that this will be the default behaviour on new boards and a bootloader update will be created.

The reboot case is more difficult because it looks as though it's a choice between disabling 1V8 on sd-cards or coping with a brief loss of 3V3.

[gtrainavicius]

The reboot case is more difficult because it looks as though it's a choice between disabling 1V8 on sd-cards or coping with a brief loss of 3V3.

Just before you replied, I found that if I 'disable' the 1V8 state in the device tree, the reboot issue with Pisound attached no longer occurs, and 3V3 stays at the same voltage, by applying the following changes to bcm2711-rpi-4-b.dts:

diff --git a/arch/arm/boot/dts/bcm2711-rpi-4-b.dts b/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
index 9b5c4f8a4..ecd6279cb 100644
--- a/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
+++ b/arch/arm/boot/dts/bcm2711-rpi-4-b.dts
@@ -107,14 +107,14 @@
                compatible = "regulator-gpio";
                vin-supply = <&vdd_5v0_reg>;
                regulator-name = "vdd-sd-io";
-               regulator-min-microvolt = <1800000>;
+               regulator-min-microvolt = <3300000>;
                regulator-max-microvolt = <3300000>;
                regulator-boot-on;
                regulator-always-on;
                regulator-settling-time-us = <5000>;

                gpios = <&expgpio 4 GPIO_ACTIVE_HIGH>;
-               states = <1800000 0x1
+               states = <3300000 0x0
                          3300000 0x0>;
        };
 };

coping with a brief loss of 3V3.

The problem is that on reboot, 3V3 does not recover to the 3V3 state with Pisound attached. Also, as the analog ICs have separate analog and digital supply pins, they go to a bad state when digital supply is lost and heat up. Other HATs that use both 5V and 3V3 supply nets may fail in various ways, if 3V3 gets cut off, 60ms gives a long time for magic smoke to escape. :)

I think it's very likely that this will be the default behaviour on new boards and a bootloader update will be created.

Do you mean the WAKE_ON_GPIO=1 will be the new default at some point? It would definitely make sense, the 3V3 should really not go lower to avoid damage to existing HATs, and on shutdown it permanently would remain low.

I tried with WAKE_ON_GPIO=1 EEPROM, 3.3V stays on at shutdown. 👍

When should 1V8 be used on SD cards? I do not know about this area.

[timg236]

Yes, WAKE_ON_GPIO will be set to 1 by default, if the user wants to force halt=power_off then they need to change the EEPROM config.

The 1.8V signalling for SD-CARDs is described here https://www.sdcard.org/developers/overview/low_voltage_signaling/index.html

The problem is that the sd-card power is not toggled during a reboot and the only way to do that is to reset the PMIC. AFAIK the card will believe that it's already in 1V8 and fail to declare this preventing UHS mode from working after reboot if the PMIC is not reset. Hopefully, others can give more information on the sd-card behaviour!

[gtrainavicius]

Hmm, actually having 3V3 at 3.3V at shutdown does not solve the issue completely - once the system shuts down, unplugging the power supply, waiting for ~3 seconds and plugging the power supply back in, the 3.3V line may fail to reach 3.3V, instead, it gets stuck at 0.5V, and oscillates at 370Hz, 232mV peak-to-peak.

So this splits this issue into two - first one is the voltage changes on 3.3V are not expected by HATs and could lead to damage if using both 3.3V and 5V pins for supply.

The other issue is 3.3V supply not powering up after unplugging and plugging Raspberry Pi 4 back in.

I measured that it takes 60ms after 5V power is connected for 3V3 supply to be switched on. This could give enough time for the electronic parts which depend on both rails to latch into a bad state that could prevent 3V3 from powering up to the right level. On Raspberry Pi 3B+, the 3V3 supply takes less than 6ms to power up, this includes the ramp-up time, it is instant, compared to 3V3 on RPi4. Normally, one would have to take into account the long time delay for 3V3 to come online when designing electronics.

Is it the firmware that is controlling 3V3 power supply on Raspberry Pi 4, or is it done somewhere in the kernel? Could it switch it on as soon as possible instead of a 60ms delay?

Also, shouldn't 1.8V logic for SD card be completely isolated from the GPIO header's 3.3V?

Raspberry Pi 4 power up:

Raspberry Pi 3 power up:

[gtrainavicius]

Hey guys, any more thoughts on this issue? Is there something that we could add to Pisound HAT EEPROM to make Raspberry Pi 4 turn on 3.3V rail as early as possible and keep it on? Even if that would mean disabling some new features related to SD cards while Pisound is attached?

[timg236]

3V3 being removed at halt will be addressed via a bootloader update which makes the WAKE_ON_GPIO=1 the default.

The loss of 1V8 during a reboot can only be resolved by never switching the sd-card for 3V3 i.e. never use UHS mode. This requires adding the following kernel command line option via cmdline.txt
sdhci.debug_quirks2=4

Not sure if that could be added automatically e.g. if the HAT EEPROM is recognised or maybe if the dtoverlay for particular HATS are spotted before the kernel is loaded

There's no software mechanism for initialising the 3V3 earlier because the power sequencing is controlled via the PMIC / RC network and this happens before the SOC runs.

[gtrainavicius]

Looks like an overlay like this can be added to HAT EEPROM to force 'no 1.8V' mode:

       fragment@8 {
               target = <&emmc2>;
               __overlay__ {
                       no-1-8-v;
               };
       };

However, if mounted on older Raspberry Pi's, the overlay won't get applied, because they don't have emmc2, so probably instead target-path would have to be used, or maybe there's more ways to make it compatible...

Anyway, I don't think this would be the right solution, but would work as an interim workaround to avoid rebooting issues.

I see the voltage level switch seems to be implemented using a GPIO pin, but MxL7704 PMIC seems to be controlled using I²C, what exactly does the GPIO pin do? Trigger some other IC to signal the PMIC via I²C to change the mode, or is it hooked to 'GLOBAL EN' pin or some other MxL7704 pin?

[pelwell]

Another way to disable high-speed mode is with an sdhci quirk, specified on the kernel command line:

sdhci.debug_quirks2=4

A lesser known fact is that an overlay can append to the command line:

	fragment@0 {
		target-path = "/chosen";
		__overlay__ {
			bootargs = "sdhci.debug_quirks2=4";
		};
	};

[JamesH65]

Did the updated bootloader https://www.raspberrypi.org/forums/viewtopic.php?f=117&t=246027 fix this issue?

[gtrainavicius]

The updated bootloader fixes the 3.3V supply issue on shutdown, but 3.3V getting stuck at 0.2V~0.5V on reboot issue still remains.

[timg236]

The reboot behaviour can't be changed without a board change. The workaround for soft-reboot is the SDHCI quirks but I don't believe that can be applied automatically.

[JamesH65]

Given the only remaining issue is not a SW problem, I believe this can be closed?

[pelwell]

The firmware can easily append to the kernel command line, if that counts as automatic.

[timg236]

By automatic I was referring to the firmware detecting whether fixing SD at 3.3V was necessary. I don't think it can in the general case so this would need to be added to the HAT specific overlay.

[rafaello7]

I have noticed that systemctl reboot 6 command (6 is the number of my /boot partition) doesn't boot to the 6th partition when the kernel command line does not have the sdhci.debug_quirks2=4 parameter. Rainbow on screen appears instead. Is it the same problem cause or is it something different?

[pelwell]

Because the SD card power on the 4B is driven by the common 3V3 rail and the SD standard only allows switching from 1V8 to 3V3 mode when the card is power-cycled, the only way to get out of 1V8 mode is to power-cycle the SoC. This has the unfortunate side effect of losing the state where the partition selector is stored, therefore systems that rely on selecting an alternate partition (e.g. NOOBS) must remain in 3V3 mode (which is what the SDHCI quirk does).

[rafaello7]

Nice to know. I have spent a few days to figure out the problem reason. A bug in systemd? Maybe missing kernel module? Maybe specific kernel config parameter? Maybe interference of another module with the Broadcom watchdog module rebooting system? The quirk for SD card looked innocent...

[rafaello7]

BTW Is the switch to low speed mode during soft reboot really necessary? I understand that machine must start at low speed because SD card may not support high speed. But nobody changes SD card on the fly at soft reboot?

[pelwell]

Yes, it's required, because... reasons. We spent a few days trying to come up with ways to avoid the current compromise, but failed.

[rafaello7]

OK, I understand 🙂 Thanks.

[subatomicglue]

Watching this thread because I can't sudo reboot with pisound attached to my rpi4b (my own report here)

Interesting stuff!
Also, did you see the pisound workaround here (Giedrius, Jul 18)?

[adamreisnz]

So.. if a watchdog is connected to the 3.3V, and triggers a reboot, the whole setup currently stays dead because the watchdog loses its power during the reboot and the reset pin remains grounded.

Is this a design flaw in the Pi4 boards? Anyone have a workaround for that which doesn't involve powering the watchdog via a separate external 3.3v power supply?

[pelwell]

So.. if a watchdog is connected to the 3.3V, and triggers a reboot, the whole setup currently stays dead because the watchdog loses its power during the reboot and the reset pin remains grounded.

To me a design flaw is a watchdog which when unpowered keeps the Pi held in reset.

[adamreisnz]

Perhaps, I don't know enough about electronics or watchdogs to either confirm or deny that. We were using this one, which we thought to be reputable and well thought out: https://www.switchdoc.com/dual-watchdog-timer/

As it seemed to work fine with Pi 2 and 3, we assumed the problem lies with the Pi 4 dropping the 3.3v during reset.

But we'll probably switch to simply using the built-in watchdog instead to avoid all these problems.