rpi 3 freezes / rpi 2 camera module not found kernel: Bump to 4.9.78 23a007716a0c6a8677097be859cf7063ae093d27

[lesvaches]

rpi 3 freezes.
rpi 2 camera module not found after update.

[ngineerken]

I too have a rpi 3 frozen after the 4.9.78 update

[pelwell]

When does it freeze?

[ngineerken]

Right after the SSH warning screen, I can get to the console screen and Nano a file, but then frozen.
I first tried booting from a piDrive and it froze after updating.
Then I went to the SD card updated apt-get update etc. rpi-update reboot frozen. in all if you leave it alone it will lockup around a minute on the desktop on either piDrive or SD card.

[lesvaches]

i have the official 7" screen connected. cold boot the access via ssh is ok. after waking the screen it seems to freeze via ssh and screen.

[pelwell]

Can you try taking the start*.elf and fixup*.dat files from here (the previous release)? There should be no compatibility issues with the kernel.

[ngineerken]

With the suggested fix in place both the SD & piDrive now boot and running fine

Thanks for the turnaround

[lesvaches]

confirming it works for me too. tank you very much.

[xavierdono]

Hi,

Got the same issue, blank screen.

I copy the boot.bak to boot and it's work but now I don't have bluetooth don't know if it's related.

[pelwell]

Your Bluetooth problem shouldn't be related.

We've been trying to reproduce this problem today, and although we have seen Pi 3s occasionally fail to boot (or hang shortly afterwards), we don't yet have a real handle on what is going wrong.

[xavierdono]

I can give you some logfiles from my rpi to help you, if you want.

[pelwell]

Although I'm not optimistic they'll tell us much, it's more than I have at the moment so yes please.

[xavierdono]

Which one?

[pelwell]

The only ones that might be useful are the kernel (dmesg) logs if you were able to capture them via a serial port - otherwise the chances of them making it to SD card are small.

[xavierdono]

Sorry just on my SD.

Good luck!

[pelwell]

I'm no closer to understanding the failures, but I have made a test firmware that removes some of the changes from the previous firmware. If you have the time and can test without risking your data, the replacement elfs and dats can be downloaded here: https://drive.google.com/file/d/1eCctauevqC6XCbRLItf196Etd7j9Fu43/view?usp=sharing

[xavierdono]

I replace it in the boot folder. No blank screen, boot normally.

[xavierdono]

...but now it's freeze.

[pelwell]

Thanks for trying - we've had similar results here, although the cause is still unknown. The next step is likely to be a partial rollback of the changes in that release.

[xavierdono]

FYI I downgrade to 9f5eea7 and I got bluetooth back.

[ngineerken]

With the suggested fix in place both the SD & piDrive now boot and running fine

Thanks for the turnaround

[ngineerken]

I just updated to 4.9.79 and the Raspberry Pi 3 is frozen again

[ngineerken]

I reverted to the Fix for the lockup, I'm back up & running.

Does anyone test the firmware before releasing it?

[ngineerken]

I opened this ticket to get more eyes looking at the problem
raspberrypi/firmware#936

[pelwell]

Not more eyes, just the same eyes in two different places.

Does anyone test the firmware before releasing it?

Yes of course, but this appears to be a rare problem, and I'm not yet completely convinced we've reproduced it here (there is some doubt about the Pi we thought was showing the same symptoms - it might be a bad power cable). I'm not going to be happy until we've completely understood and fixed whatever is causing you the boot failure, but without a (un)reliable test-bed that is going to be difficult.

I have another set of patches that may perhaps help - some oddities I spotted when examining the differences - I'll try to get those into the next update. If any of you feel strongly enough about this problem (given that you have a workaround) and can spare a Pi for a week or so, you could ship it to us at Pi Towers. If the failure is as consistent as you say then it shouldn't take long to bisect the changes and identify the culprit. Email me - phil@raspberrypi.org - if that appeals and I'll send you the shipping address. We normally throw in a decent SD card as a thank you under such circumstances.

[pelwell]

The patches mentioned above are in today's firmware release.

[pelwell]

Has anybody had time to try the test firmware I posted above?

[k-toffel]

I think I have the same Problem. I did set up a clean installation of archlinuxarm and updated to the 4.9.79 kernel. After that my RPi3 was either not booting at all or freezing within the first five minutes or so.
I have a non official charger but I have been using this one for quiet some time. It's also capable to provide 3A instead of the recommended 2.5A so I'm fairly sure that this is not the problem.
The firmware you posted above is working fine for me. There have been zero freezes and boot problems since then. I did not change anything in the config.txt.

[pelwell]

Thank you for the feedback - it's a big help. If I could get confirmation that it works for one more person then I'll merge the partial reversion and continue the bisection.

[Sharktear]

Hi,
I've not yet tried the firmware.

@k-toffel

I have a non official charger but I have been using this one for quiet some time. It's also capable to provide 3A instead of the recommended 2.5A so I'm fairly sure that this is not the problem.

The point is not original power supply vs not original. The strange thing I've noticed is that If I use a poor (for example 1 A) power supply, the raspberry boot fine and not freeze.

[popcornmix]

Latest rpi-update firmware has a potential fix for this issue from @pelwell

[apepper]

I just tried out the latest rpi-update firmware f6f7e04. Sadly my raspberry pi 3 still freezes. I'll try some of the suggested steps and will report again.

[apepper]

@pelwell

I did some tests. I upgraded to the latest version f6f7e04, which still freezes my raspberry pi 3 (as written before).

I tried out the following tweaks in /boot/config.txt as asked in #168 (comment):

• force_turbo=1 - does not boot ❌
• arm_freq=600 - reliably boots and does not freeze ✅
• arm_freq=900 - reliably boots and does not freeze ✅
• arm_freq=1050 - reliably boots and does not freeze ✅
• arm_freq=1125 - reliably boots and does not freeze ✅
• arm_freq=1200 - does not boot or freezes ❌

I'm using a WD PiDrive 1TB together with an older version of the WD PiDrive 2A cable kit.

I hope this helps to solve the issue.

[pelwell]

@apepper Once again, thanks for trying and reporting back.

At your convenience, can you post the output of these two commands:

$ vcgencmd otp_dump
$ vcgencmd get_config int

The fact that recent updates have been causing problems where things were OK before is clearly a regression, but I do note that the page describing the WD PiDrive 2A cable kit recommends the 3A cable kit for a Pi 3. The official RPi adapter supplies 2.5A, and we wouldn't recommend anything less.

[apepper]

With arm_freq=1125 in /boot/config.txt:

$ sudo vcgencmd otp_dump
08:00000000
09:00000000
10:00000000
11:00000000
12:00000000
13:00000000
14:00000000
15:00000000
16:00280000
17:1020000a
18:1020000a
19:ffffffff
20:ffffffff
21:ffffffff
22:ffffffff
23:ffffffff
24:ffffffff
25:ffffffff
26:ffffffff
27:00002727
28:fb70e06b
29:048f1f94
30:00a02082
31:00000000
32:00000000
33:00000000
34:00000000
35:00000000
36:00000000
37:00000000
38:00000000
39:00000000
40:00000000
41:00000000
42:00000000
43:00000000
44:00000000
45:00000000
46:00000000
47:00000000
48:00000000
49:00000000
50:00000000
51:00000000
52:00000000
53:00000000
54:00000000
55:00000000
56:00000000
57:00000000
58:00000000
59:00000000
60:00000000
61:00000000
62:00000000
63:00000000
64:00000000
65:00000000
66:00000000

$ sudo vcgencmd get_config int
arm_freq=1125
audio_pwm_mode=514
config_hdmi_boost=5
core_freq=400
desired_osc_freq=0x36ee80
disable_commandline_tags=2
disable_l2cache=1
display_hdmi_rotate=-1
display_lcd_rotate=-1
force_eeprom_read=1
force_pwm_open=1
framebuffer_ignore_alpha=1
framebuffer_swap=1
gpu_freq=300
hdmi_force_cec_address=65535
init_uart_clock=0x2dc6c00
lcd_framerate=60
over_voltage_avs=0x19f0a
over_voltage_avs_boost=0x19f0a
overscan_bottom=32
overscan_left=32
overscan_right=32
overscan_top=32
pause_burst_frames=1
program_serial_random=1
ramfsaddr=-1
sdram_freq=400
temp_limit=85

The fact that recent updates have been causing problems where things were OK before is clearly a regression, but I do note that the page describing the WD PiDrive 2A cable kit recommends the 3A cable kit for a Pi 3. The official RPi adapter supplies 2.5A, and we wouldn't recommend anything less.

"Back in the days" when I bought the WD PiDrive Kit it only included a 2A cable kit. It has been running stable with that cable for at least a year or longer. I'll see, if I get a 3A charger.

[ngineerken]

@pelwell
I have the same configuration as apepper, and I just ran the update and frozen again.
I took your "test_firmware_180205" files on a good boot-able SD and copied them to the PiDrive.
Rebooted and It back up and running.
I did not make any changes to the config.txt

[k-toffel]

I just updated to the latest firmware and it managed to boot straight away. It runs for around 15 minutes, but I'll let it run for some more time and see if it freezes.
I should have mentioned in my last post that I don't use a PiDrive but rather a SD card..

$ vcgencmd otp_dump
08:00000000
09:00000000
10:00000000
11:00000000
12:00000000
13:00000000
14:00000000
15:00000000
16:00280000
17:1020000a
18:1020000a
19:ffffffff
20:ffffffff
21:ffffffff
22:ffffffff
23:ffffffff
24:ffffffff
25:ffffffff
26:ffffffff
27:00002727
28:070379b2
29:f8fc864d
30:00a02082
31:00000000
32:00000000
33:00000000
34:00000000
35:00000000
36:00000000
37:00000000
38:00000000
39:00000000
40:00000000
41:00000000
42:00000000
43:00000000
44:00000000
45:00000000
46:00000000
47:00000000
48:00000000
49:00000000
50:00000000
51:00000000
52:00000000
53:00000000
54:00000000
55:00000000
56:00000000
57:00000000
58:00000000
59:00000000
60:00000000
61:00000000
62:00000000
63:00000000
64:00000000
65:00000000
66:00000000

$ vcgencmd get_config int
arm_freq=1200
audio_pwm_mode=514
config_hdmi_boost=5
core_freq=400
desired_osc_freq=0x36ee80
disable_commandline_tags=2
disable_l2cache=1
display_hdmi_rotate=-1
display_lcd_rotate=-1
force_eeprom_read=1
force_pwm_open=1
framebuffer_ignore_alpha=1
framebuffer_swap=1
gpu_freq=300
hdmi_force_cec_address=65535
init_uart_clock=0x2dc6c00
lcd_framerate=60
over_voltage_avs=0x186a0
over_voltage_avs_boost=0x16e36
overscan_bottom=48
overscan_left=48
overscan_right=48
overscan_top=48
pause_burst_frames=1
program_serial_random=1
ramfsaddr=-1
sdram_freq=400
temp_limit=85

[pelwell]

From the small sample of users who had problems before and have tried the new firmware, the results seem to be that:

• 2 users with PiDrives (@apepper, @ngineerken) found that the latest firmware still fails to boot but test_firmware_1802025 works.
• 1 user without a PiDrive (@k-toffel) found that the latest firmware works.

This is very useful feedback - are there any more datapoints? @Sharktear? @xavierdono? If you do still see a failure, can you also run vcgencmd otp_dump and vcgencmd get_config int on a working system? Thanks in advance.

[popcornmix]

Latest rpi-update firmware has another fix from @pelwell

[pelwell]

Yes, by underpowering a Pi3 to the point where the voltage limiting was kicking in and out I was able to reproduce a crash and pinpoint the bug.

@apepper and @ngineerken, although your systems should boot as before with today's firmware, your Pi3s have marginal power supplies which are causing reduced performance from time to time. The firmware is detecting the undervoltage and throttling back, but I could imagine that there might exist a workload that would cause the voltage to drop too rapidly to be corrected. It's up to you whether you consider that important enough to warrant a new supply.

[apepper]

I can confirm, that the latest sudo rpi-update (5c80565) fixed the issue 🎉 . Thanks a lot for all the work!

It's up to you whether you consider that important enough to warrant a new supply.

A new 3A power supply is already ordered and on its way to me.

[Sharktear]

Thank you! I'll try to update this weekend... the fix also correct the problem I reported with the correct power supply (with poor power supply the boot was fine) ?

[pelwell]

I think that was just a different aspect of the same issue. With a permanently poor supply there would have been no switch to turbo speed, avoiding the problems around switching voltages.

[Sharktear]

This also should explain why tryng many times to reboot (with the correct power supply), sometimes the raspy boot correctly

[popcornmix]

@lesvaches is latest firmware good for you? If so you can close the issue.

[lesvaches]

@popcornmix yes, looks like I'm good.

[pelwell]

Thanks for your patience, everyone.

[xaviervilla]

I think I might be having the same issue. Every 4 or so days, SSH to my pi stops working and I have to unplug it and plug it back in. At first I thought it was disconnecting from wifi, so I wrote a script to ping google every 5 minutes, and report the results.. Turns out it wasnt failing to ping.. It would just suddenly stop pinging for a day or so until I unplugged it. I checked my syslog at the time of the last ping, and It sows it undergoing updates at that time. It's unclear to me, but it seems that the pi is rebooting at night like 1am to do automatic updates and not powering off correctly.. I have no pi drive.. I only use my pi as a WOL server for my desktop. I know the issue is rare, so Its worth mentioning my name is Xavier, like one of the previous commenters. lol just a stretch. I also have the official pi3 power supply.

[xaviervilla]

I will do rpi-update. I will be honest I didnt know that was a thing. I just do apt-get update/upgrade and I will get back after 2 weeks.

[xaviervilla]

rpi-update did fix the issue. Glad to see commit done.