Crust Wakeup issue #204
Comments
|
PinePhone Pro have different SoC which is not supported by Crust, so the issue on forums is not related to your problem. |
|
Well I will report that back to the guy who posted on the forums. Am I correct to assume my issue is related to crust as far as the standard pinephone? |
|
Yes, it is possible that your issue is related to Crust. What distro are you using? Some distros build Crust with a few debugging options already enabled. Otherwise, you will need to rebuild Crust with those options enabled. Then, after a failure to wake you will be able see what Crust was last doing and if it finished its part of the resume process with Please also provide a full With that information, I can suggest further troubleshooting steps. |
|
I am using manjaro phosh is that built in or do i need to rebuild crust? |
|
It does not have those options enabled. You can enable them by changing that block of lines and rebuilding the package: echo -e "\nBuilding CRUST for Pine64 PinePhone...\n"
echo CONFIG_DEBUG_RECORD_STEPS=y >> configs/pinephone_defconfig
echo CONFIG_DEBUG_VERIFY_DRAM=y >> configs/pinephone_defconfig
make CROSS_COMPILE=or1k-elf- pinephone_defconfig
make CROSS_COMPILE=or1k-elf- build/scp/scp.bin
cp build/scp/scp.bin ../u-boot-${pkgver/rc/-rc}
You can verify that it's working if you see nonzero values in the |
|
Ok just so I am clear because I have only built AUR package I would do this; test with then report back? |
|
Yes, that should work. It looks like installing the updated package will try to flash the new U-Boot binary. |
Is that an issue? or should it be fine to flash the new U-Boot binary? Also I am reluctant to do this now as I am out of town and if something gets trashed I wont have a phone so I will wait until monday to do this and report back. |
|
Crust is inside the U-Boot binary, so you have to flash U-Boot to get the updated Crust. There's no rush; install the debug firmware and try to collect the logs whenever is convenient for you. |
|
I believe I have successfully installed U-Boot with the debug options. Running At this point I should wait for a freeze then reboot the phone and run |
|
Yes, that's the plan. The NVMEM output you provided looks good (but see below). Examining the nonzero word, the first two digits confirm this is output from Crust v0.5. The last two digits come from this list of steps. 0x0503 means So you have two options:
"crust-${_scpver}.tar.gz::https://github.com/crust-firmware/crust/archive/refs/heads/${_scpver}.tar.gz"
|
|
I don't have UART adapter as an option so number 2 is it. I am a but fuzzy on turning Interesting piece of information. Since rebuilding crust (yesterday at midnight) I have yet to get hung up in suspend. This could very well be a coincidence however its very rare that I have gone a full day without a hang up in suspend. I am going to try to stay off the charger as much as I can today to see if this is maybe more then coincidence and there is simply something wrong with the crust build that comes on the latest Manjaro betas ( seems unlikely as arch did the same thing and I have tried flashing OS fresh a few times). |
|
As of today I have not had a single hang up in suspend, could this problem have resolved itself with the recompile of crust? I tried fresh OS installs in the passed with no luck? Does debug need to be on or was it simply manually rebuilding the package that fixed my issue? |
From the Manjaro packaging, it looks like you would need to add the option to the
That's very interesting. Either one is possible. It's also possible (though unlikely at this point) that the issue still exists, but it has happened not to be triggered. You could try reinstalling the official package, or building a new package without the debug options, to see if the crashes come back. If they do, then likely the tiny additional delay introduced by the debugging code is allowing suspend/resume to complete. |
|
I just reinstalled using the git clone method without changing the debug options which in theory should be the same as the stock uboot setup when the OS is installed. I will report back in a day or two with if I have any suspend resume problems. |
|
With debug off I had it stall twice in the last few days I am going to recompile with debug on and see what happens. |
|
OK so it seems it was just dumb luck. With debug enabled I just had my first suspend hang so lets go back to plan A. As my pinephone is my daily driver I want you to look at this to make sure I am doing it correctly before I do it and botch my phone and loose my setup as I like the way the phone is setup right now. First I will go nano into /boot/boot.txt and adjust the file to look like so; Then I wll run; Next I will follow the same process I did above for turning on debug mode as well as changing the _scpver=master and the url link leaving me with a PKGBUILD that looks like so: Finally I will run and reboot. Then on the next freeze I reboot and run To get some use-able information to try and trouble shoot this? Sorry for the long winded post I just dont want to botch the install as everything is just the way I want it. Thanks so much for the help |
|
Ok so I decided to take a dd image of my eemc as a safety net and followed the above instructions I laid out for myself. Initially crust-master had an integrity check issue, so I ran makepkg with the --skipinteg switch to get it build (i know that's bad form but I really want to fix this issue). here is the output of: This was run after completing everything, rebooting, and then allowing the phone to suspend then wake again. Am I ready to debug now? |
|
Yes, all of that looks good. The integrity check failing is expected, since you did change which version of the crust source code got downloaded. The purpose of adding the |
|
Here is the output after a reboot but before a suspend, I assume because before it was rebooted the suspend worked correctly so it is the same |
|
Explain the cpuidle.off=1 setting a bit more. It seems I am not having the suspend hang at least not yet but I am noticing phosh restarting from time to time under heavy use-age which must be related to the cpuidle setting? If this ends up being a fix can I just leave it that way? |
|
OK, I finally had it get stuck in suspend, the blue indicator light was flashing which is new but I could not wake it from suspend so I had to hold power until off then power it back on. The output once booted back up was the same as above; Maybe I don't have something done correctly? |
|
Can you paste the full output from I'll also try to answer your questions.
The purpose of The cpuidle subsystem (documentation) opportunistically powers off CPU cores when the kernel thinks they will be unused for some period of time. The purpose is to save power; for the A64 SoC in the PinePhone, it reduces power consumption by up to 30 mW. That's not a lot, but it does mean a few extra minutes of battery. Every time Linux decides to power off a CPU, after doing its internal bookkeeping, it calls TF-A, which then calls Crust, to actually flip the power switch. Then, when a hardware interrupt occurs, Crust turns the core back on, so Linux can use it again. This happens hundreds of times per second. When Crust flips the power switch, it records that
Enabling/disabling cpuidle should have no noticeable effect on any application running on the PinePhone. cpuidle doesn't change any behavior visible from userspace; it only affects performance/power consumption.
Yes, absolutely. However, if the Crust debug settings or
A flashing LED suggests a kernel panic, not an issue with Crust. During the part of system suspend where Crust gets involved, the main CPU is completely powered off. So there is no software running that could flash the LED.
This new issue seems unrelated to your original suspend issue. Manjaro would be the best place to ask about debugging the kernel panic. As for the |
|
Thanks so much for the detailed response! Here is the output of dmesg; Here is the contents of /proc/cmdline; So just to add to the information. I just took a rather large update to manjaro which overwrote crust and a bunch of other programs (turning debug off but did not seem to turn cpuidle off). I decided to let it ride for a bit and within an hour suspend was jammed up but with debug off so obviously no use-able crash data. Once I recmompiled crust like described above, no more suspend issue except the one i described that sounds like it was not a crust issue. The other interseting piece of information is I actually forgot a step when I recompiled so I did the master branch changes but forgot the debug changes and that sent me back into suspend lock out intially. It seems debug is the key to this, So your original estimation of an extra bit of time to write to the log preventing this issue from happeneing is the most likely answer. Let me know what you think. Thanks again for all the help. |
The typo here explains why you were still seeing
I will try to figure out where adding delay makes a difference. I don't know how long this will take. |
|
Well that is embarrassing. After re-running the /boot/boot.txt bit without the typo and restarting, then letting the phone suspend and wake I get this; Seems my typo slowed down this progress. Sorry about that. |
|
Finally had a suspend lockup here was the output after reboot; Same as above? Maybe I still have something wrong? |
|
You probably have everything correct. That code means Crust completed all of its resume steps and turned the ARM CPU back on. So Crust itself is not hanging. But if there is a bug in how Crust restores the hardware state, that could prevent Linux from running correctly after the resume. I think you've given me all of the data that you can at this point, so thank you. Like I said, I will have to do some more digging in to what is going wrong. |
|
Thanks for all the help, it is certainly better with debug on as it only happens once or twice a week now as opposed to upwards of a few times a day. Let me know if you need any other testing done. |
|
As mentioned here, I am having the same issue as described above, also a few times a day. The phone crashes when entering suspend and thus is unable to wake up, even if the power button is pressed. Thanks to this script, typically the phone would sleep for a specific amount of time and then wake up, as shown below: However, sometimes it cannot wake up and thus must be force-restarted. For example, as shown below, the phone crashed at |
|
Have you tried recompiling crust using the debug flag as I posted above? That has cut down the amount this happens to nearly nothing. |
|
I have tried to clone the repository and run I can see a |
|
@XaviDCR92
That gives you those tools that are required to run makepkg |
|
So this problem has come back up. I just took a large update and uboot was updated. I followed these instructions and rebuilt the package as with the update suspend is hanging frequently again. Is this because the github package has updated and these instructions need adjustment? How can I get back to how it was before the update? I tried downgrading to this version in my cache; but I am still seeing this as the hexdump output: Maybe some other package that was updated is causing this? It is freezing every third suspend. Making the phone nearly unusable with suspend at this point. Thanks |
As I stated above, it looks like there are some missing packages. Whereas the packages below are in fact available: All of the commands below return nothing: It looks like an |
|
@XaviDCR92 my apologies I thought maybe you did not have the devel packages installed, I was mistaken. Those packages you are missing are not installed on my system and my u-boot works so something else must be going on. I just resolved my new suspend issues from updating by restoring to a old dd image of my system from just before i built u-boot from the PKGBUILD described in this thread. uboot-pinephone 2022.04-1 which is what you are trying to install gave me massive issues. I can not tell if it was because of uboot-pinephone 2022.04-1 or because of other packages that updated around it. my current version that was built but not installed just prior to taking my dd image when this thread started is uboot-pinephone 2022.04rc3-2, Installing that on the old dd iimage without updating my system has returned my phone to how it was which is nearly working perfect. Maybe try building that package and do not update the system if you have not already done sp. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Type of issue
Question:
I am not entirely sure if this is a crust issue but I can not seem to trouble shoot it so I figured asking here may point me in the correct direction;
I am using an original Pinephone and generally crust works fine to wake my phone for calls and text messages. I use my Pinephone as a daily driver so it suspends and wakes a lot throughout the day as any phone does. At least once but sometimes upwards of four or five times a day when I hit the power button while the phone is in suspend my phone will not wake. I know its still on because holding the power button does not turn it on. I have to hold the power button to turn it off then hold the power button again to turn it on again.
Ssh does not allow me in obviously because it is in the deep sleep state so I can not get into the OS to trouble shoot what the issue is. This has never happened on the charger as my phone is set to never sleep on the charger.
I posted in the Pine64 forums and found another person having the same issue with a PInePhone Pro which I assume has the same crust implementation, he has tested on multiple OS's as well so its not specific to a distribution.
Any help trouble shooting this would be greatly appreciated as this is the only thing that really makes daily driving my phone a pain, otherwise it works 100% as a regular phone.
The text was updated successfully, but these errors were encountered: