Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RPi4: Long delay before booting linux after power on #1375

Open
acidtech opened this issue Apr 20, 2020 · 79 comments
Open

RPi4: Long delay before booting linux after power on #1375

acidtech opened this issue Apr 20, 2020 · 79 comments

Comments

@acidtech
Copy link

On an RPi4
I am using Raspbian
I am using bootloader:
Mar 4 2020 14:24:08
version a445ea4fa94708c89875dbbb7a0b19e72987cbb2 (release)
timestamp 1583331848
Linux retropie 4.19.66-v7l+ #1253 SMP Thu Aug 15 12:02:08 BST 2019 armv7l GNU/Linux

I'm seeing a fairly long delay before Linux starts booting after powering on my RPi4. I have a custom kernel module that starts ~2.5 seconds after "Booting Linux on physical CPU 0x0" according to dmesg. However it takes more than 9 seconds before I see my module running(it loads a splash screen via dispmanx). There is almost no delay from when my module loads and when the splash screen is displayed(tested via manually loading the kernel module).

I am using a DPI LCD. This uses the ID_SC and ID_SD pins for display purposes. I noticed early on that if I did NOT add external pullups on those 2 pins the RPi4 would get stuck there. The impedence of the LCD pins was such that the signal was less than 1v when the DPI LCD was attached. I could see the I2C sequence run by the PI on my scope. It just repeated indefinitely. After I added the pullups(10k at first and now I've tried 4.7k) booting worked correctly. But I see this long delay. I dont remember having this problem before. I expect a firmware update may have introduced it.

I have the boot splash(rainbow and raspberries) disabled and text disabled. It takes ~7 seconds from power on till I see the LCD dispaly a caret(eg when the RPi starts driving the LCD). Before this there is no signal sent to the DPI LCD(checked with a scope).

I tried adding "bcm2708.vc_i2c_override=1" to cmdline.txt but this doesnt seem to make a difference.

Any thoughts on what the delay is? Is there something I can do to reduce this delay. I'd like my splash to come up as fast as possible. 9 to 10 seconds may not seem that long but for a handheld PDA style device it is an eternity to have no user feedback after switching on power.
@6by9
Copy link

6by9 commented Apr 21, 2020

If memory serves correctly, you can disable the HAT eeprom scan with force_eeprom_read=0 in config.txt.
The HAT eeprom is read from the firmware, so nothing you add to the kernel command line will make any difference.

@pelwell
Copy link
Contributor

pelwell commented Apr 21, 2020

There are limits to how quickly the Pi 4 can boot - the SDRAM PHY has quite a lengthy training sequence, several seconds from what I remember.

Specifying start_cd=1 selects the cut-down firmware, which is smaller and loads quicker as a result. Otherwise keep config.txt as short as possible. You could also try experimenting with the initial_turbo setting which forces the system into turbo mode for a specified number of seconds at boot time.

@timg236
Copy link

timg236 commented Apr 21, 2020

A reasonable amount of time is taken up walking through the root directory checking for the existence of files because there is minimal filesystem caching. Removing unnecessary files from the boot partition and defragmenting might help a bit.

@pelwell
Copy link
Contributor

pelwell commented Apr 21, 2020

An explicit kernel=... and device_tree=... will also bypass any form of searching.

@acidtech
Copy link
Author

Here is my current config.txt

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

[pi4]
# Enable DRM VC4 V3D driver on top of the dispmanx display stack
dtoverlay=vc4-fkms-v3d
max_framebuffers=2

[all]
disable_splash=1
boot_delay=0
intial_turbo=30

start_x=0
enable_uart=0

audio_pwm_mode=2
dtoverlay=audremap,pins_18_19

dtoverlay=dpi18_2
dpi_group=2
dpi_mode=87
dpi_output_format=0x070016
dpi_timings=640 1 44 2 42 480 1 16 2 14 0 0 0 75 0 28000000 1
enable_dpi_lcd=1

#[edid=*]
#enable_dpi_lcd=0

If memory serves correctly, you can disable the HAT eeprom scan with force_eeprom_read=0 in config.txt.
The HAT eeprom is read from the firmware, so nothing you add to the kernel command line will make any difference.

Whether I have force_eeprom_read=0 in config.txt or not the Pi still locks up if I dont have the external pullups on ID_SC and ID_SD.

There are limits to how quickly the Pi 4 can boot - the SDRAM PHY has quite a lengthy training sequence, several seconds from what I remember.

Specifying start_cd=1 selects the cut-down firmware, which is smaller and loads quicker as a result. Otherwise keep config.txt as short as possible. You could also try experimenting with the initial_turbo setting which forces the system into turbo mode for a specified number of seconds at boot time.

start_cd=1 in config.txt doesnt appear to have any effect on boot limes. SHould that be in cmdline.txt instead?

A reasonable amount of time is taken up walking through the root directory checking for the existence of files because there is minimal filesystem caching. Removing unnecessary files from the boot partition and defragmenting might help a bit.

You can see my config.txt is about as small as I can get it so not much parsing going on.

An explicit kernel=... and device_tree=... will also bypass any form of searching.

Can you provide an example of using these?

I'm not sure I was clear in my first post. This is the time from when power is applied till I see the DPI output start. About 7.5 seconds as I just tested before I saw the dot clock activate after power was applied. I've used Pi3s with DPI lcds and that time was at most 2 seconds from power till DPI was active. I could have sworn the Pi4 booted faster in the past. Specifically I had LibreElec running on it and could swear it was a couple seconds at most before the Libre Elec icon would show.

@pelwell
Copy link
Contributor

pelwell commented Apr 21, 2020 

intial_turbo=30

There's a typo in this line. It might not have a big effect, but correctly spelt it might have some.

@acidtech
Copy link
Author

acidtech commented Apr 21, 2020 via email

@6by9
Copy link

6by9 commented Apr 22, 2020

If you just want the DPI output active faster then set up the DPI pinmuxing directly from config.txt instead of from an overlay that the kernel applies.
gpio=0-21=a2

@pelwell
Copy link
Contributor

pelwell commented Apr 22, 2020

an overlay that the kernel applies.

To be clear, the firmware would apply the overlay by loading the .dtbo file and manipulating the in-memory DTB. Each of those tasks takes longer than executing the gpio= setting. When the kernel starts it then has to make changes to the pin mapping based on the content of the DTB, which is going to take at least as long as the config setting - that's another saving. However, each of those tasks probably only takes a few milliseconds, so it won't be a big win.

@acidtech
Copy link
Author

acidtech commented Apr 22, 2020 via email

@leragequit
Copy link

@acidtech any progress?

I ran into the same issue, it takes a solid 6-7 seconds for the display to initialize, making the process of minimizing boot time quite unrewarding.

@acidtech
Copy link
Author

acidtech commented Jun 15, 2020 via email

@popcornmix
Copy link
Contributor

popcornmix commented Jun 15, 2020 •
edited
Loading

@acidtech initial_turbo=10 should give you the benefit of faster booting without the downside of force_turbo. It will just force turbo for the first 10 seconds.

@acidtech
Copy link
Author

acidtech commented Jun 15, 2020 via email

@popcornmix
Copy link
Contributor

force_turbo=1 only sets warranty bit if used with over_voltage.
And only on some Pi models (not Pi4).

There should be no increase in chance of corrupting sdcard with initial_turbo (or force_turbo), as long as your overclock settings are stable.

@acidtech
Copy link
Author

acidtech commented Jun 15, 2020 via email

@leragequit
Copy link

While the additional 9 seconds might be ok for applications that are just started once and run for eternity, it's a real barrier for portable batterie powered applications with a screen which are meant to be started and shut down.

@popcornmix is the some data what the pi is doing during this time? any way to speed it up?

@popcornmix
Copy link
Contributor

Setting initial_turbo or using a faster sdcard are the obvious ways of speeding it up.
Connecting a uart and enabling bootloader and/or firmware logging may provide some information about what is happening.

@pelwell
Copy link
Contributor

pelwell commented Jun 16, 2020

Note that enabling UART logging will slow things down even more, but it might give an idea of where the time is going.

@timg236
Copy link

timg236 commented Jun 16, 2020

The bootloader is about 3.5 seconds. About 2 seconds are SDRAM init and the reset depends on which start.elf variant is used.

A USB 3.0 pen drive seems to be the best balance between throughput and initialisation time and should be faster than the SD card driver when loading the kernel.

@acidtech
Copy link
Author

acidtech commented Jun 16, 2020 via email

@acidtech
Copy link
Author

acidtech commented Jun 16, 2020 via email

@Yoshqu
Copy link

Yoshqu commented Jun 19, 2020

Yes

The SD card is (almost) completely irrelevant. My config is:

initial_turbo=10
start_cd=1
arm_64bit=1
gpu_mem=16
disable_splash=1
force_eeprom_read=0
boot_delay=0
#initramfs initram 0x00200000

[pi4]
armstub=armstub8-rpi4.bin
kernel=kernel8-rpi4.img

kernel8-rpi4.img is a bare metal 4kb boot loader which blinks on very start. It takes 5s from powering up the rpi4 to this blink.

If I uncomment initramfs it will load additional 200kb of data in marginally longer time.

Boot partion uses very small clusters (one sector) it helps if it is cleared (backup first) by:
mkfs.fat /dev/sdb1 -n boot -F 32
and files are copied in one go (won't be fragmented that way)

@pelwell I suppose training sequence can't be cached :) ? oh well... I bought the device for computing power anyway (but my use case consist fast boot up too - so any improvement is most welcome)
Thanks for wonderful device!

@cleverca22
Copy link

@Yoshqu there is a much simpler way to measure boot time to the usec, without having to use that custom bootloader
https://gist.github.com/cleverca22/58784f67690bfb97492f3f439ff00ed7

basically, ST_CLO begins counting at 1mhz, the instant the SoC comes out of reset
and /proc/uptime begins counting seconds when linux starts up its internal clocks

the code in the gist, will then print both of them at once, and if you find the difference, youll see how long it took for linux to gain control of the core

@acidtech
Copy link
Author

acidtech commented Jul 29, 2020 via email

@s00500
Copy link

s00500 commented Jul 30, 2020

I am experiencing the same issue on the PI4 currently.... I have noticed that when not enabeling 64 bit mode I can reduce the time by almost 1 second, 4 seconds seems to be the sonic barrier here... I would really like to understand why exactly that is...

@pelwell
Copy link
Contributor

pelwell commented Jul 30, 2020

It's not an issue, it's a fact of life. The memory interface needs to calibrated at boot, and that takes time.

@popcornmix
Copy link
Contributor

The 32-bit kernel can decompress itself, so is stored compressed on sdcard. The 64-bit kernel doesn't support that so it is a larger file and takes longer to load from sdcard.

@acidtech
Copy link
Author

acidtech commented Jul 30, 2020 via email

@pelwell
Copy link
Contributor

pelwell commented Jul 30, 2020

Once the bootloader (i.e. the EEPROM, not the main start*.elf loaders) has loaded the appropriate start*.elf and launched it, it has to load config.txt, the kernel, the dtb, cmdline.txt, apply overlays and other DTB changes, and load any external armstub or dt-blob file. There's also a fair amount of moving things around because it isn't always possible to know up front where to load something to. Most of the hardware devices need to be reinitialised, a HAT EEPROM and POE fan probed for. Boot is essentially sequential and it all adds up.

@timg236
Copy link

timg236 commented Oct 7, 2020

boot_delay happens very early on during the execution of start.elf an is before display initialization. Recent improvements to boot time have reduced the time between the splash screen being displayed and kernel load. Consequently, it's likely to have removed the splash screen before some monitors have switched resolution.

@franklesniak
Copy link

franklesniak commented Dec 31, 2020 •
edited
Loading

Here is my current config.txt

# Enable audio (loads snd_bcm2835)
dtparam=audio=on

[pi4]
# Enable DRM VC4 V3D driver on top of the dispmanx display stack
dtoverlay=vc4-fkms-v3d
max_framebuffers=2

[all]
disable_splash=1
boot_delay=0
intial_turbo=30

start_x=0
enable_uart=0

audio_pwm_mode=2
dtoverlay=audremap,pins_18_19

dtoverlay=dpi18_2
dpi_group=2
dpi_mode=87
dpi_output_format=0x070016
dpi_timings=640 1 44 2 42 480 1 16 2 14 0 0 0 75 0 28000000 1
enable_dpi_lcd=1

#[edid=*]
#enable_dpi_lcd=0

Hey @acidtech, slightly off-topic, but is the effect of the [edid=*] section that the Parallel Display Interface would be disabled if any HDMI display is connected? And if no HDMI display is connected, the section is ignored - so the Parallel Display Interface is left configured?

@acidtech
Copy link
Author

acidtech commented Dec 31, 2020 via email

@leragequit
Copy link

Hello again,

I have been looking at the boot times again and it seems not much has changed. I am still sitting at 9 second before I see the rainbow screen while building something that is meant to be a portable device at some point.
Are there any developments, like new options for the bootloader and config.txt to stop the pi from unecessary processing when the enviroment is very defined? @acidtech did you manage to trim down the time a bit?

Best regards,

Sascha

@lurch
Copy link
Contributor

lurch commented Apr 22, 2021

Have you tried enabling the bootloader UART output? That might give you some idea of what the bootloader is actually doing?
https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2711_bootloader_config.md

@pelwell
Copy link
Contributor

pelwell commented Apr 22, 2021

"sudo vcdbg log msg" is a quicker way to the same information.

@acidtech
Copy link
Author

acidtech commented Apr 22, 2021 via email

@lurch
Copy link
Contributor

lurch commented Apr 22, 2021

I think the EEPROM changes you refer to are just to make it less likely that the diagnostics screen gets shown before the rainbow screen appears? (i.e. only show the diagnostics screen if there's actually a problem preventing booting.) People sometimes panic if they see "unexpected text" on screen, even if that text is harmless 😉
Shouldn't have any affect on bootup times AFAIK.

@leragequit
Copy link

@acidtech what did you get the boot times down to? do you mind sharing settings :D?

@lurch
Copy link
Contributor

lurch commented Apr 23, 2021

@pelwell Just out of curiosity, is it possible that the SDRAM initialisation on a 2GB Pi4B would be fractionally quicker than on an 8GB Pi4B ? 🤷 (or would any difference be imperceptible?)

@pelwell
Copy link
Contributor

pelwell commented Apr 23, 2021 •
edited
Loading

There are minor timing between different SDRAM parts, but it's more about the number of dies on the chip than the capacity.

The latest stable EEPROM image is quicker than previous version, but in general there is no work happening (or likely to happen) specifically to reduce boot times - either in the EEPROM or the firmware - so unless an optimisation becomes apparent during other work, the current releases are probably as good as it gets.

@acidtech
Copy link
Author

acidtech commented Apr 23, 2021 via email

@lurch
Copy link
Contributor

lurch commented Apr 23, 2021

I guess if you wanted a more "instant on" behaviour you could use the Raspberry Pi Pico microcontroller? But I've obviously got no idea what your application does so I dunno if that's a workable option for you.

@acidtech
Copy link
Author

acidtech commented Apr 24, 2021 via email

@tam1111574
Copy link

Hello all,
Anybody have the way to reduce this time ?..

@hippyau
Copy link

hippyau commented Jul 31, 2021

I'm trying to get times down too, can anyone confirm if APPEND_DTB to kernel and removing from boot works / helps? Not had any luck booting with an appended 5.10 kernel.

@hippyau
Copy link

hippyau commented Jul 31, 2021 •
edited
Loading

I have a config.txt as follows. Trying to use a CM4 with eMMC in an industrial project. It needs to come up as soon as possible.

It takes 7 seconds from power-on for the kernel to start, and 3 seconds for the kernel to boot.

Surely there has to be a faster way? Will appending the DTB to the kernel and removing from boot help? Is that possible?

This guy boots a RPI3 in less than 2 seconds?!?! https://www.furkantokac.com/rpi3-fast-boot-less-than-2-seconds/

start_file=start.elf
fixup_file=fixup.dat
kernel=Image
arm_64bit=1
#active cooling required
initial_turbo=15
over_voltage=6
arm_freq=2000

# gpu_mem_256=100
# gpu_mem_512=100
# gpu_mem_1024=100

max_framebuffer_height=1920
hdmi_ignore_edid=0xa5000080
hdmi_timings=480 1 48 32 80 1920 0 3 10 56 0 0 0 60 0 75840000 3
hdmi_group=2
hdmi_mode=87
hdmi_force_mode=1
hdmi_drive=1
config_hdmi_boost=4
display_hdmi_rotate=2

dtparam=i2c_arm=on
#dtparam=audio=on

dtoverlay=vc4-fkms-v3d
max_framebuffers=2

dtoverlay=dwc2,dr_mode=host

dtoverlay=disable-bt

enable_uart=1

init_uart_clock=16000000
dtoverlay=uart0
dtoverlay=uart2
dtoverlay=uart3
dtoverlay=uart4
dtoverlay=uart5

disable_splash=1
boot_delay=0

@Yoshqu
Copy link

Yoshqu commented Jul 31, 2021

As it was said in messages above - SDRAM PHY is lengthy (seems around 4s?). It seems as least time for boot loader. With full blown config as yours it will be hard to achieve.

So, comparing RPi3 to RPi4 is comparing apples to oranges.

@hippyau
Copy link

hippyau commented Aug 2, 2021

Specifying start_cd=1 selects the cut-down firmware, which is smaller and loads quicker as a result. Otherwise keep config.txt as short as possible. You could also try experimenting with the initial_turbo setting which forces the system into turbo mode for a specified number of seconds at boot time.

@pelwell I understand the start_cd only allows for a small CPU/GPU memory division, is there a way to change that?

@popcornmix
Copy link
Contributor

@pelwell I understand the start_cd only allows for a small CPU/GPU memory division, is there a way to change that?

Why do you want a large gpu memory with the cutdown firmware?
The cutdown firmware has basically no firmware services available (so no firmware 3d, no codecs, no camera, no mall/openmax, no submitting firmware vpu/qpu jobs). For what reason do you need the extra gpu memory?

@hippyau
Copy link

hippyau commented Aug 2, 2021

Ah, thank you. This makes much sense, as it was not clear exactly what was cut down. I need the GPU, and little else, I presumed as it controls the boot process from my understanding, it would be available. I suppose I am poking in the dark for solutions.

@pelwell
Copy link
Contributor

pelwell commented Aug 2, 2021 •
edited
Loading

Note that the most recent firmware builds (7208c3d and later) should boot fractionally quicker (~0.8s less).

@hippyau
Copy link

hippyau commented Aug 2, 2021

Thanks @pelwell, I will give it a go. So surprised by this issue 💯

@lurch
Copy link
Contributor

lurch commented Aug 2, 2021

I dunno if it'll make any noticeable difference, but I might be worth tweaking BOOT_ORDER to only include the boot method that you're actually using? https://www.raspberrypi.org/documentation/hardware/raspberrypi/bcm2711_bootloader_config.md

@tianxun-ng
Copy link

tianxun-ng commented Apr 27, 2022 •
edited
Loading

Hi, any updates so far? I'm also facing the same issue. The rainbow screen comes out (kernel starts) only after 6 seconds. Previously on CM3Plus the rainbow screen appears within 2 seconds then continue booting the rest of OS.

@JamesH65
Copy link
Contributor

The latest Pi4 does have a longer boot time than earlier models, due to the new network install (1s), plus AVS calibration (0.5s),plus grabbing the edid for second displays. You can get rid of the network boot delay using NET_INSTALL_ENABLED=0 (see https://github.com/raspberrypi/rpi-eeprom/blob/master/firmware/release-notes.md), but there's not a lot that can be done about everything else.

@tianxun-ng
Copy link

tianxun-ng commented Apr 27, 2022 •
edited
Loading

Thanks. How do I get rid of the network boot delay NET_INSTALL_ENABLED=0? I'm currently using 64-bit Debian 11 (not Raspbian) so there's no tool to update the EEPROM. What else can I disable to make the boot faster other than this option? thanks

@JamesH65
Copy link
Contributor

https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#updating-the-eeprom-configuration

If necessary, get a new SD card, install PiOS and do the work in that. Then go back to Debian.

@faaizrafi
Copy link

I'm using Raspberry Pi 4B for about an year now, last week i upgraded my sd card from 8gb to 32gb to get more space on the system. Now the raspi is taking too long to boot. I ran the command [b]systemd-analyze blame[/b] to know which process is taking too long. I have attached the snap below, what should i do?

systemanalyze-raspi

@advancel advancel mentioned this issue Sep 21, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests