Suspend to ram support: suspend is fine (after some tweaking), but can't resume from it

[piso77]

Hi there,

i was trying to make the 'suspend to ram' work on the raspberrypi boards (both on rpi2 and rpi3, branch rpi-4.13.y, but the same applies to 4.9 apparently), and i got to the point where i can succesfully suspend, but i can't find a way to resume from it.

By default, suspend support is off in the raspberry kernel but after tweaking the .config and enabling the necessary option (CONFIG_SUSPEND&c), i was finally able to test it:

$ sudo sh -c 'echo mem > /sys/power/state'
sh: echo: I/O error

...
[ 317.843806] PM: Syncing filesystems ... done.
[ 317.852920] PM: Preparing system for sleep (freeze)
[ 317.853143] PM: Adding info for No Bus:vcs63
[ 317.853395] PM: Adding info for No Bus:vcsa63
[ 317.857163] Freezing user space processes ... (elapsed 0.001 seconds) done.
[ 317.858883] OOM killer disabled.
[ 317.858885] Freezing remaining freezable tasks ... (elapsed 0.001
seconds) done.
[ 317.860187] PM: Suspending system (freeze)
[ 317.860191] Suspending console(s) (use no_console_suspend to debug)
[ 317.880754] platform regulatory.0: bus suspend
[ 317.880848] sdio mmc1:0001:2: bus suspend
[ 317.880867] sdio mmc1:0001:1: bus suspend
[ 317.880891] mmcblk mmc0:aaaa: bus suspend
[ 317.880915] mmc mmc1:0001: bus suspend
[ 317.882143] sdhost-bcm2835 3f202000.mmc: bus suspend
[ 317.882193] mmc-bcm2835 3f300000.mmc: bus suspend
[ 317.882207] bcm2835_thermal 3f212000.thermal: bus suspend
[ 317.882230] i2c-bcm2835 3f804000.i2c: bus suspend
[ 317.882260] spi-bcm2835 3f204000.spi: bus suspend
[ 317.882354] bcm2835-aux-uart 3f215040.serial: bus suspend
[ 317.882367] bcm2835-aux-clk 3f215000.aux: bus suspend
[ 317.882373] usb 1-1.1: type suspend
[ 317.882385] uart-pl011 3f201000.serial: bus suspend
[ 317.882469] bcm2835-clk 3f101000.cprman: bus suspend
[ 317.882719] leds led1: class suspend
[ 317.882893] leds led0: class suspend
[ 317.883007] platform Fixed MDIO bus.0: bus suspend
[ 317.883701] smsc95xx 1-1.1:1.0 eth0: entering SUSPEND2 mode
[ 317.884184] usb 1-1: type suspend
[ 317.884233] usb usb1: type suspend
[ 317.884407] dpm_run_callback(): usb_dev_suspend+0x0/0x20 returns -2
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ 317.884416] PM: Device usb1 failed to suspend async: error -2
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[ 317.884449] PM: Some devices failed to suspend, or early wake event detected
[ 317.884528] usb 1-1: type resume
[ 317.884779] platform Fixed MDIO bus.0: bus resume
[ 317.884803] leds led0: class resume
[ 317.884827] leds led1: class resume
[ 317.884848] bcm2835-clk 3f101000.cprman: bus resume
[ 317.884861] uart-pl011 3f201000.serial: bus resume
[ 317.884883] bcm2835-aux-clk 3f215000.aux: bus resume
[ 317.884894] bcm2835-aux-uart 3f215040.serial: bus resume
[ 317.884915] spi-bcm2835 3f204000.spi: bus resume
[ 317.884933] i2c-bcm2835 3f804000.i2c: bus resume
[ 317.884947] bcm2835_thermal 3f212000.thermal: bus resume
[ 317.885009] mmc-bcm2835 3f300000.mmc: bus resume
[ 317.885021] sdhost-bcm2835 3f202000.mmc: bus resume
[ 317.885062] platform regulatory.0: bus resume
[ 317.885247] usb 1-1.1: type resume
[ 317.885257] mmcblk mmc0:aaaa: bus resume
[ 317.885267] mmc mmc1:0001: bus resume
[ 317.939038] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
[ 317.940621] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[ 317.942203] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[ 317.945048] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
[ 318.033873] sdio mmc1:0001:1: bus resume
[ 318.033878] sdio mmc1:0001:2: bus resume
[ 318.033896] PM: resume of devices complete after 149.435 msecs
[ 318.034055] usb 1-1.1: completing type resume
[ 318.034092] usb 1-1: completing type resume
[ 318.034139] usb usb1: completing type resume
[ 318.035489] dwc_otg 3f980000.usb: completing power domain resume
[ 318.055572] PM: Finishing wakeup.
[ 318.055578] OOM killer enabled.
[ 318.055584] Restarting tasks ... done.
...

$ lsusb
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp. SMSC9512/9514 Fast Ethernet Adapter
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp. SMC9514 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

it seems the usb hub didn't suspend properly, and after studying a bit the source code, i noticed a couple of things:

-the raspberry kernel ships an external usb driver (dwc_otg commits), and it appears to be, pretty much, the same driver that was shipped in the original raspbian kernel (3.18?), forward ported to recent kernels - my guess is that, back then, the Linux kernel didn't have an upstream, open source driver, thus the choice to go with the vendor driver
-but nowadays, the upstream kernel ships an open source driver for the same chip (CONFIG_USB_DWC2&c), and it appears to be working fine - upstream uses it, and for as much as i could test, everything i attached to the usb hub worked fine

so i decided to give is a shot - i reverted the dwc_otg commits first:

4026c73 Revert "Add dwc_otg driver"
d7b820c Revert "ARM64/DWC_OTG: Port dwc_otg driver to ARM64"
58c323f Revert "dwc_otg: make periodic scheduling behave properly for FS buses"
f90659e Revert "dwc_otg: fiq_fsm: Make isochronous compatibility checks work properly"
4f41876 Revert "dwc_otg: add module parameter int_ep_interval_min"
2e94393 Revert "dwc_otg: fiq_fsm: Add non-periodic TT exclusivity constraints"

and then i removed the usb dt override:

$ git diff
diff --git a/arch/arm/boot/dts/bcm270x.dtsi b/arch/arm/boot/dts/bcm270x.dtsi
index 81914a6..e58e7e7 100644
--- a/arch/arm/boot/dts/bcm270x.dtsi
+++ b/arch/arm/boot/dts/bcm270x.dtsi
@@ -107,14 +107,6 @@
status = "disabled";
};

•           usb@7e980000 { /* usb */
  

•                   compatible = "brcm,bcm2708-usb";
  

•                   reg = <0x7e980000 0x10000>,
  

•                         <0x7e006000 0x1000>;
  

•                   interrupts = <2 0>,
  

•                                <1 9>;
  

•           };
  

•           v3d@7ec00000 { /* vd3 */
                    compatible = "brcm,vc4-v3d";
                    power-domains = <&power RPI_POWER_DOMAIN_V3D>;
  

since the root arch/arm/boot/dts/bcm283x.dtsi already refer to the upstream dwc2 driver:

...
usb: usb@7e980000 {
compatible = "brcm,bcm2835-usb";
reg = <0x7e980000 0x10000>;
interrupts = <1 9>;
#address-cells = <1>;
#size-cells = <0>;
clocks = <&clk_usb>;
clock-names = "otg";
phys = <&usbphy>;
phy-names = "usb2-phy";
};
...

rebuilt the kernel, and i was finally able to suspend the board with this setup:

$ sudo sh -c 'echo mem > /sys/power/state'

all leds but power turned off, screen went black, nic went down, etc but i couldn't find a way to resume from this state.

According to the kernel, these are the wakeup sources available:

$ find /sys -name wakeup
/sys/devices/platform/soc/3f980000.usb/usb1/power/wakeup
/sys/devices/platform/soc/3f980000.usb/usb1/1-1/power/wakeup
/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.1/power/wakeup
/sys/devices/platform/soc/3f804000.i2c/i2c-1/1-0068/power/wakeup
/sys/devices/platform/soc/3f215040.serial/tty/ttyS0/power/wakeup
/sys/devices/platform/soc/3f201000.serial/tty/ttyAMA0/power/wakeup

but, even after enabling all of these before suspending:

$ echo "enabled" | sudo tee $abovefiles

i couldn't wake up the board in any way: it didn't react to the serial console, nor to a usb keyboard attached directly to the usb ports, couldn't wake it from lan (not sure that would have worked at all though, etc)

I tried switching serial (dtoverlay=pi3-disable-bt), still no dice.

I attached an external rtc chip on i2c (ds3231 - dtoverlay=i2c-rtc,ds3231,wakeup-source), checked that it was working:

$ dmesg | grep rtc
[ 3.860452] hctosys: unable to open rtc device (rtc0)
[ 16.737232] rtc-ds1307 1-0068: 'wakeup-source' is set, request for an IRQ is disabled!
[ 16.756005] rtc-ds1307 1-0068: registered as rtc0

$ sudo i2cdetect -y 1
0 1 2 3 4 5 6 7 8 9 a b c d e f
00: -- -- -- -- -- -- -- -- -- -- -- -- --
10: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
20: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
30: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
40: -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
50: -- -- -- -- -- -- -- 57 -- -- -- -- -- -- -- --
60: -- -- -- -- -- -- -- -- UU -- -- -- -- -- -- --
70: -- -- -- -- -- -- -- --

$ sudo hwclock -r
Thu Sep 14 11:22:50 2017 .922813 seconds
$ sudo hwclock -s

and set the wakealarm in the future:

$ sudo sh -c "echo 0 > /sys/class/rtc/rtc0/wakealarm"
$ sudo sh -c "echo date '+%s' -d '+ 3 minutes' > /sys/class/rtc/rtc0/wakealarm"

suspended the board as usual, but it didn't wake up after the timeout.

So, i'm running out of ideas at the moment, and i was wondering if you happen to know if there's a way to resume the board after suspending it or shall i just mark the feature as 'not available'?
And tangentially from that, is there a reason why the raspberry kernel still uses the dwc_otg driver instead of the upstream dwc2 one?

[popcornmix]

or shall i just mark the feature as 'not available'?

That would be my suggestion. In general with software if something has never been tested it won't work. I'm pretty sure dwc_otg won't handle this.

More significantly the GPU firmware won't handle this. You may suspend when the GPU is driving HDMI in a different resolution from the default boot resolution, perhaps outputting audio through a currently running DMA. There may be 3D or ISP running with a huge amount of context (both RAM and in peripheral registers). A huge amount of work would need to be done to support this.

You are free to use the dwc2 driver, but you'll find its performance is far lower than dwc_otg (partly due to not supporting the FIQ as well as multiple years of bug fixes and performance improvements).

[lategoodbye]

@piso77 Nice to see that someone working on this. How about implementing suspend to idle, which should be easier?

[JamesH65]

Any progress to report on this? Should this particular issue be closed as a wont fix? (Since the GPU cannot really function in this mode without considerable work)

[piso77]

Yeah, wont' fix makes sense.