Kernel oops or hard freeze when streaming video on Zero W (and Pi 3B+) #2555
Comments
|
Could you please try http://downloads.raspberrypi.org/raspbian_lite/images/raspbian_lite-2018-03-14/ with the Zero W? |
|
When leaving a dmesg: journald: But it seems I'm unable to provoke a kernel oops! UPDATEIt seemed harder to trigger but after making and disconnecting ssh connections with a few seconds in between while running raspivid I got a crash on raspbian_lite-2018-03-14 as well: And another (before any hard freeze or reboot): And another (this time ssh stops working but ping replies) |
|
I'm having this issue when streaming video for long periods of time, even if the CPU average usage is below 30%. The 3B+ hard locks some time between 30 minutes to a few hours, consistently. Using the exact same SD card image, it works perfectly stable for over a week on a 3B. It will usually not let the hardware or software watchdog kick it after the hard lock occurs. I've been seeing this behavior since I got the 3B+ on day one. |
|
@amovitz What version of kernel are you running? Does rpi-update fix the issue. |
|
It should be noted, the Pi camera module is not being used on my system. It has been tested with every version of the kernel available - rpi-update has not fixed the problem. This particular test lasted approximately 45 minutes before hard locking this morning.
|
|
If you drop the max CPU frequency to 1200, does that help? |
|
I have set arm_freq=1200 and will let it run. I'll report back.
|
|
@amovitz I have that same problem on 3B+ as well without any camera attached, from start it has frozen within a day or maximum 2 days. The recent crash logs are from my Zero W though. I never managed to get a dump from the 3B+, so maybe or maybe not two different issues. And these two rPi's are my first experience, not great :P |
|
Almost certainly different problems, the SoC on the Zero and 3B+, along with the different wireless chips means they are sufficiently dissimilar. Unless (!) there is something in your environment that is triggering the fault. We've sold a lot of Pi's which run for years without issue, so this is unusual. First check is power supply, is it sufficient? |
|
The power supply I'm using is the branded one. we've tried many different power supplies and it will even hard lock when no applications are running. We don't use any Zeros, so I couldn't tell you if the same happened for some reason. We have extensively tested with the 3B, running for months at a time, and never had an issue until we put the SD card into a 3B+ (and upgraded the boot for it). Same image works perfectly well on a 3B, but the 3B+ will constantly lock up. So far, it's been a little over 6 hours, and it hasn't locked with the lowered CPU Freq, but we'll see within 24 hours if it's more stable. From our testing, it feels like it is an issue with either the upgraded SoC or another chip on the board which has changed that the Kernel is not handling properly or that the Broadcom firmware is not interfacing properly with the new chip. |
|
@JamesH65 yeah I would have expected more people to react if it was common, I must have hit the jackpot with 2 out of 2. My power is the raspberry branded ones which I'm assuming has the correct rating. The thing both have in common is that the SDCard brand is Sandisk class 10 and that it is written to by Zero WThe installed image is not customized in any way except for configuring WiFI, camera module via raspi-config and SSH access. 3B+This one runs Kubernetes master which averages on 20% cpu. |
|
About 1 hour and 15 minutes after my last post (Total Uptime: 07:35:46), it hard locked. |
|
Hmm, if it had kept running at 1200 I would have thought it was one of the test escapes (i.e. an SoC that passed test but shouldn't have). But I believe they work at 1200. So confused, and not sure what to suggest. Might be worth RMA'ing for a replacement. |
|
@amovitz and when it freeze it become burning hot right. sounds very much the same as my 3B+ |
|
Burning hot is bad. That would indicate a broken SoC I suspect. |
|
That's my issue, too. It may be a factory defect since we got the first batch on launch day and all of ours exhibit the same behavior (3 boards). It still freezes even when it's clocked at 1200MHz. I would definitely agree that it is high above ambient temperature. We even have a custom heatsink on it with a fan and it's still very hot after it hard locks. If it will help, I can try to get external thermal readings and thermal images, but I'm pretty certain it's the SoC that's hot. |
|
@popcornmix @pelwell Any thoughts?
…On 25 May 2018 at 14:22, Alex Movitz ***@***.***> wrote:
That's my issue, too. It may be a factory defect since we got the first
batch on launch day. It still freezes even when it's clocked at 1200MHz
I would definitely agree that it is high above ambient temperature. We
even have a custom heatsink on it with a fan and it's still very hot after
it hard locks. If it will help, I can try to get external thermal readings
and thermal images, but I'm pretty certain it's the SoC that's hot.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#2555 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADqrHeoMOmt466TZiJpA1CjBm9nJS2V6ks5t2AV4gaJpZM4T8-N0>
.
--
James Hughes
Principal Software Engineer,
Raspberry Pi (Trading) Ltd
|
|
@amovitz i had the same thing with some PIs. |
|
Overvoltage did not help, it still hard locks. |
|
setup: pi 3B+, Linux raspberrypi 4.14.44-v7+ #1117. Completely stuck after ~1 hour of playing video. gpu_mem=256, everything other is by default. |
|
So I've gotten a 2nd Zero W and it seems that at least for that model, it's a hardware issue. I noticed one difference on the hardware layout, the "H" marking on the chip closest to the camera module connector. Not sure if it helps with anything but here are photos of the two: Broken one
Working one
|
|
@balboah Are you still having issues after the latest kernel update? |
|
I'm seeing this (or something similar) on the latest kernel (and whatever kernel I was running before I upgraded - still 4.14). It's a Pi Zero W.
The reported process in the Oops and the backtrace appears to be random, though this one does show vc.ril.video_en. Maybe the video codec is trashing random bits of memory somehow? I don't have a serial port hooked up, and it normally crashes before the Oops makes it onto the SSH session, so gathering useful crash dumps is a bit hard. |
|
@amovitz sorry for the late reply. After replacing the hardware I am no longer experiencing the problem (still the same sdcard) and I've stopped using the broken one. Maybe there are faulty hardware or some small differences which cause the bug to trigger |
|
I did some experiments. On a different Pi Zero, my card + camera + power supply worked fine. That seems to point to an issue with my actual hardware, supporting @balboah's theory. Another example of an Oops when running the camera below: Pretty annoyed that this seems to be a HW issue, as this zero is now deeply embedded in the project. |
|
With further fiddling, and following @zoff99 and this forum thread: https://www.raspberrypi.org/forums/viewtopic.php?f=43&t=212777, I resolved my issue by over-volting. Seems like Broadcom/RPi shipped a batch or two of marginal chips. |
|
I can confirm the very same problem with my Raspberry Pi Zero W (Revision with H*) an dsolved ith with It would be interesting to get to know what the differences between the two HW revisions are! |
|
I was to early: 950MHz reduces the freezes, but they still occur. @eldad-a Had all six Raspi Zeros the H*? |
@avanc NB: I have it set at |
@avanc Hi again, Hope this is of any help |
|
@eldad-a Thanks for the update. I also got freezes with 600MHz. So it seems the device is really broken. As last try, I will install Rasbian and do some tests. |
|
Hi all, sorry about the delay in replying. Can anyone seeing this issue who hasn't tried it, try a core voltage increase to see if this changes anything? In config.txt, add the line over_voltage=1 If no help, slightly larger numbers might help. |
|
@JamesH65 No luck. I tried |
|
Did you try larger numbers for over_voltage? |
|
After reading a little more about over_voltage, I'm confused: Shall I set it to 7? In that case I have to set |
|
over_voltage is added to the default voltage (1.35V for turbo mode on Pi0). |
|
the same behavior here on stretch RPIZW when streaming via gst-launch-1.0 - both via wlan0 and ppp0 (modem) UPDATE: before i set over voltage to 2 RPI was crashing consistently after 15 mins, thanks! |
|
also set my Ri0 W to over_voltage=1 after crashing after 15 mins and now it's been running over an hour no issue. Using streameye |
|
Also having the h* sign near my camera port and also getting kernel panic. I am using a 1.3 Raspberry Pi Camera module. I get crashes after few minutes and lost wifi connection. |
|
I have the RPi Zero W with the H*. I got it working well (so far) with below added to config file. |
|
Ok, I had a PI Zero W ("model H") that, after fresh install of Raspbian buster, consistently froze when trying to Setting in config.txt fixed the problem for me, too. |
|
got a similar problem. I just setup my pi zero w with the latest raspian image. Now it gets stuck once streaming between minutes and hours. Any ideas?
|
|
after half a day with it seems to be more stable. So far no freeze. |
|
I've also just had these issues when trying to overclock the raspberrypi zero. So far the best experience is keeping everything default. One can check the defaults on this page: https://www.raspberrypi.org/documentation/configuration/config-txt/overclocking.md |
|
i think the issue is more likely the turbo mode or frequency switching (maybe even idle) than the overvolt itself. My freezes are completely gone with which should be a light undervolt as per the defaults, but compared to stock its 100% stable if i do longer streaming sessions and my PI zero runs 24/7. |
|
i am also having this problem when using mavlink-router with a pi zero. With a CM4 based board the issue doesnt exist. for me works to prevent it. kernel panic: |
|
Problem is that most including ne had it without OC at all. Might be a issue with the Turbo |
This solution solved my problem with crashing Pi Zero 2 W |
…with the raspberry pi zero, which yield to crashes of the system
and others
This is a summary from troubleshooting on https://www.raspberrypi.org/forums/viewtopic.php?f=28&t=213423&p=1315106
When trying to stream video on Zero W with the noir v2 camera module, I get many variants of kernel oopses and freezes. For example:
It can easily be reproduced by these steps:
The issue still exists when testing after updating with
rpi-update.I have also had freezes on Pi 3B+ with the same camera, but have not been able to create a proper reproduce or catched any kernel messages. It might be more related to #2387
The text was updated successfully, but these errors were encountered: