Test T440p with Coreboot NO_GFX_INIT

[rbreslow]

From Coreboot's src/device/Kconfig:

Select this to not perform any graphics initialization in coreboot. This is useful if the payload (e.g. SeaBIOS) can initialize graphics or if pre-boot graphics are not required.

@tlaurion says that the Linux kernel should be able to perform this initialization: #1282 (comment). It's possible that allowing Coreboot to initialize graphics as well is redundant/increases boot times.

The pull request should provide before and after cbmem timings.

Waiting on #1282

[srgrint]

I can confirm I have tried with NO_GFX_INIT set with no adverse effects. For some reason, it still dosen't redraw the screen after the kexec until the deian kernel starts - not quite sure why.

Here it the cbmem log (have built using coreboot 4.19, although 4.17 has the same behaviour)

coreboot_t440p_no_gfx-init.log

[srgrint]

Also as a side issue regarding fine tuning the t440p kernel config, it is possible to safely remove CONFIG_DRM_AST=y

(I think this was a leftover for when the kernel config was cloned from the common librem kernel - in order to make it work with their server boards - clearly not needed on the t440p)

[rbreslow]

I think this was a leftover for when the kernel config was cloned from the common librem kernel [...]

Definitely. I had evaluated all of the Coreboot defaults, but there were too many kernel config options to try and understand each one.

Maybe we open another issue to try and come up with a process for auditing kernel config options?

[tlaurion]

I can confirm I have tried with NO_GFX_INIT set with no adverse effects. For some reason, it still dosen't redraw the screen after the kexec until the deian kernel starts - not quite sure why.

The kexec'ed OS will take over the screen only when and if i915/another fb driver initializes the display. That can be before/after plymouth.

This is where igfx init from coreboot (or no ifgx_init) should create a difference.

[tlaurion]

For example, it is known that installing Archlinux will require end users to add i915 into its initrd.

Never tested archlinux with igfx on on coreboot. That is actually a good test case for this and will try.

[srgrint]

The kexec'ed OS will take over the screen only when and if i915/another fb driver initializes the display. That can be before/after plymouth.

This is where igfx init from coreboot (or no ifgx_init) should create a difference.

Thanks - I wonder whether it is that my version of debian on the HDD is fairly old (version 10 rather than 11). Perhaps older versions didn't add enough to the initramfs. I'll try and upgrade and see what happens.

[srgrint]

Just providing some metrics about adding NO_GFX_INIT to the t440p build. The size of the ramstage is reduced from 120925 to 99316 - so a saving in the CBFS of 21609 bytes.

Clearly as the ramstage is LZMA compressed, the saving when loaded into RAM would be over twice this value. But we care mostly care about CBFS space which is the more precious commodity.

I'll do some timing tests tomorrow with and without NO_GFX_INIT

[srgrint]

So have compared timings.

Without NO_GFX_INIT (so both libgfxinit and linux kernel initialising graphics) - takes 9037ms
With NO_GFX_INIT (so only linux kernel initialising graphics) - takes 8788ms

So by setting NO_GFX_INIT, saves 248ms

I have attached the cbmemc -t logs for both the builds

cbmemc_withoutgraphics.log
cbmemc_withgraphics.log

[srgrint]

The kexec'ed OS will take over the screen only when and if i915/another fb driver initializes the display. That can be before/after plymouth.
This is where igfx init from coreboot (or no ifgx_init) should create a difference.

Thanks - I wonder whether it is that my version of debian on the HDD is fairly old (version 10 rather than 11). Perhaps older versions didn't add enough to the initramfs. I'll try and upgrade and see what happens.

I've upgraded my debian install. All works fine now - presumably my very old debian install didn't have the i915 driver in the initramfs or something.

The behaviour of my T440p is the same as my X230 now from a graphics point of view when NO_GFX_INIT is set

[tlaurion]

@srgrint thanks for #1326

My main question in link to having NO_GFX_INIT set is that OSes known to offer i915+drm drivers into initramfs will boot and have graphics as early as the kexec'ed kernel taking over graphics.

But for OSes not packing i915+drm, does coreboot GFX init (text/framebuffer support) permit to have a terminal on kexec'ed kernel prior of i915+drm and whn none are part of the initrd?

Basically: is coreboot gfx init useful for OSes not supporting specific graphic cards directly in their initramfs?
The question is not targeted specifically for t440p, but related to #1293

[akunterkontrolle]

@srgrint thanks for #1326

My main question in link to having NO_GFX_INIT set is that OSes known to offer i915+drm drivers into initramfs will boot and have graphics as early as the kexec'ed kernel taking over graphics.

But for OSes not packing i915+drm, does coreboot GFX init (text/framebuffer support) permit to have a terminal on kexec'ed kernel prior of i915+drm and whn none are part of the initrd?

Basically: is coreboot gfx init useful for OSes not supporting specific graphic cards directly in their initramfs? The question is not targeted specifically for t440p, but related to #1293

For me, NO_GFX_INIT not set (as it was/is currently) doesn't help at all with kernels that don't have i915-drivers. I have the same problem with Debian showing no graphics at early boot stage. This creates three problems for me:

• The normal Debian installers don't work. Only way to install is via live cd, however the installer than offered lacks severely in features and options.
• No chance of seeing early kernel output to debug problems.
• Memtest doesnt work. That's the most pressing problem for me currently since I upgraded RAM and have the suspicion that something isn't right with the RAM (more segfaulting processes and kernel panics than it's healthy). I tried Proper way to run memtest from heads? #535 (comment) however that leaves also only an unchanged screen after kexec.

[tlaurion]

@akunterkontrolle are you saying that the behavior changes with NO_IGFX or not for the 3 problems you specified above?

Memtest doesnt work. That's the most pressing problem for me currently since I upgraded RAM and have the suspicion that something isn't right with the RAM (more segfaulting processes and kernel panics than it's healthy). I tried #535 (comment) however that leaves also only an unchanged screen after kexec.

Last time I tested this on x230 (NO_IGFX) with manual kexec call from command line as specified there, it worked, and was confirmed by another user in need. It doesn't on latest memtest+ version? Note that it was 64 grub version. Please replicate and report under that issue, reopening it if needed.

[srgrint]

* The normal Debian installers don't work. Only way to install is via live cd, however the installer than offered lacks severely in features and options.

I would also be grateful if you could provide more information about your setup - ie what type thinkpad, what heads version and whether you have customised the configuration of heads when building. Also what version of the debian installer you have used would be helpful.

I fairly regularly run the debian installer under Heads and haven't had any problems myself. I usually use installer in text mode, but the graphics mode seems to boot fine as well.

I just tried on both my x220 (which has NO_IGFX set) and t430 (which by default does /not/ have NO_IGFX set). Both seem to work fine.

I am currently using the installer image at - https://cdimage.debian.org/cdimage/bookworm_di_alpha2/amd64/iso-cd/debian-bookworm-DI-alpha2-amd64-netinst.iso

It would be helpful if you try the exact same version of the installer as I have used, so we can try and work out what is different between our setups.

[tlaurion]

It seems that the simple answer on what Debian bookworm works while earlier didn't is because i915 is included in initramfs as tested here #1285 (comment)

[srgrint]

It seems that the simple answer on what Debian bookworm works while earlier didn't is because i915 is included in initramfs as tested here #1285 (comment)

Indeed - I think this is true. Although the reason I mentioned bookworm is that is the installer I have lying around. I have previously used bullseye (current stable) which also worked on systems both with and without NO_IGFX.

Basically: is coreboot gfx init useful for OSes not supporting specific graphic cards directly in their initramfs

As we have eluded early versions of debian probably didn't have i915 in the initramfs - as you know, I initially struggled with my t440p as I had not realised I was trying to boot from a very old hard drive with circa 2018 version of Debian. Both with or without NO_IGFX set, this produced no graphics after the kexec until after the hard drive password had been entered and LUKS unlock had happened.

So as far as I have seen not having NO_IGFX dosen't help in these cases.

Presumably most (?all) the mainstream linux distros now package i915 in initramfs

[akunterkontrolle]

First of all sorry for not reporting back sooner.

I would also be grateful if you could provide more information about your setup - ie what type thinkpad, what heads version and whether you have customised the configuration of heads when building. Also what version of the debian installer you have used would be helpful.

Setup: Thinkpad T440p without dGPU (so Intel graphics only), i7-4810MQ, 16GB RAM, 128GB SSD in normal 2,5'' hard disk bay (I don't think that the last parameters have really any influence, but listed for completeness.)
I did update the BIOS and therefore the EC firmware to the latest available version before flashing heads.

I did initially build heads based on that commit ed8c74e, since back then the pull request was not merged yet. I did not change any config options, so NO_GFX_INITwas not set.
I have since then rebuild heads from commit 96a20a7 without any changes whatsoever and flashed internally. However with both heads builds I do have stability issues with the kexec-d kernel liking to segfault/kernel panic, especially early on in the boot process. I currently don't think that this is a problem with coreboot in general or my system in particular, since skulls works flawlessly. I will open an issue about that in a few weeks when I have more spare time at hand to actually debug that - although I am currently clueless on how to debug this more or less random problem.

I fairly regularly run the debian installer under Heads and haven't had any problems myself. I usually use installer in text mode, but the graphics mode seems to boot fine as well.

I just tried on both my x220 (which has NO_IGFX set) and t430 (which by default does /not/ have NO_IGFX set). Both seem to work fine.

I am currently using the installer image at - https://cdimage.debian.org/cdimage/bookworm_di_alpha2/amd64/iso-cd/debian-bookworm-DI-alpha2-amd64-netinst.iso

I did try it with that bookworm-installer and my latest heads build. Still no change, the screen doesn't change after the kexec-command whether I try the normal or the graphical installer. And I do think the installer should have booted a few times despite my stability issues, since I was still able to reboot the system with Ctrl-Alt-Del - when it hangs, this normally not possible anymore.

[srgrint]

Definitely all very strange as we are both using identical installler/EC version, etc

Can I clarify whether your machine is just intel grapgics or dual intel/nvidia? Reason I ask is that both I and rbreslow have Intel only boards I think, so not sure if a t440p with nvidia has been tried.

[akunterkontrolle]

Definitely all very strange as we are both using identical installler/EC version, etc

Yes, indeed …

Can I clarify whether your machine is just intel grapgics or dual intel/nvidia? Reason I ask is that both I and rbreslow have Intel only boards I think, so not sure if a t440p with nvidia has been tried.

My machine is also Intel graphics only. I have updated my comment to clarify that.

[srgrint]

It turns out I think I was mistaken in that I don't think I have ever managed to boot the debian installer on my t440p. Have only succeeded on my x220, x230 and t430.

The installed debian system does work on all the machines though.

I still am struggling to work out why it dosen't work. What I know is:

• The debian installer boots successfully on the x220, x230 and t430
• A version of debian once installed works on all my machines including the t440p
• The debian installer image lacks i915 in the initramfs, whereas debian once actually installed has this in the initramfs

On the t440p, I have tried the following permutations individually to my heads build to try and boot the debian installer (none seem to make any difference)

• Upgrading the coreboot version from 4.17 to 4.19
• Trying with NO_GFX_INIT set and NO_GFX_INIT not set
• Trying with GENERIC_LINEAR_FRAMEBUFFER set and GENERIC_LINEAR_FRAMEBUFFER not set

Whilst it sort of makes sense that the debian installer dosen't initialise graphics properly (as it lacks i915 in the initramfs), I am struggling to work out why the intel graphics t430 behaves differently (ie the installer works).

The only other difference is the heads kernel version (5.10 versus 4.14). Trying a t440p build with a 4.14 kernel might be the next experiment ...

All very confusing ...

[srgrint]

I have now tried building heads for the t440p with a 4.14 kernel (ie using the same linux config as is used on the t430). This builds fine and the debian installer works.

So it must be something related to the 4.14 -> 5.10 kernel bump.

From the sounds of it, there have been previous similar issues in the past with librem 13 machines - so presumably it isn't a t440p specific problem

https://forums.puri.sm/t/how-to-boot-debian-installer-from-pureboot/9584

[JonathonHall-Purism]

Huh, good thinking, I will have to try the 4.14 kernel. I tried kexec --reset-vga the other day thinking that the Debian kernel might just be unable to switch back to text mode due to lacking the i915 driver, but that didn't improve anything.

I had thought that maybe the kernel was not able to display anything because we weren't providing either a VESA or GOP interface, but maybe this is wrong, I have not had a chance to dig in that direction. If it does work for me on the 4.14 kernel, it'll be interesting to see what driver the Debian kernel ends up using, so we can figure out why it is not working when Heads uses 5.10.

[tlaurion]

@JonathonHall-Purism got a little time to figure out what librem_common 5.x kernel config is missing as compared to 4.x?

Other similar problems encountered by NixOS users on i915 side.

[srgrint]

I intend to grab the dmesgs from the heads kernels (4.14 and 5.10) on my t440p to see if any obvious differences.

Whilst I am trying both versions, is there any other information you think may be relevant for me to grab from within heads (eg are there any particular files within /sys) which might be relevant to compare?

[tlaurion]

Comparison of oldconfig of both 4.14 and 5.10 would inform of what is missing inside of 5.10. Some insights would be about some fb related default having been dropped in the latter.

[tlaurion]

Comparison of oldconfig of both 4.14 and 5.10 would inform of what is missing inside of 5.10. Some insights would be about some fb related default having been dropped in the latter.

Doesn't seem to be the case. So I created a branch which is not PR material as of now under https://github.com/tlaurion/heads/tree/add_tools_to_ease_kernel_version_bumping to add make BOARD=XYZ linux.blah additional statements to help us create proper linux kernel version bumps in the future.

I produced with that versioned config files for linux-x230-maximized.config (shared accross xx20/xx30 but not the t440p which was based on librem_common.config without nvme support):

• defconfig files for 4.14 and 5.5.10
• olconfig (full configs) for 4.14 and 5.5.10

I then compared defconfig produced for 5.5 vs librem_common.config (defconfig for 5.5) and found nothing popping to my eyes related to framebuffers or anything that shows clear link with graphic framebuffers.

[tlaurion]

One thing I see though as a difference that could matter when comparing librem14/t440p/x230:

• librem14 taken as example adds in KERNEL_REMOVE=intel_iommu=on intel_gfx=off (directed at qubes) and then KERNEL_ADD=intel_iommu=igfx_off while having CONFIG_LINUX_COMMAND_LINE="intel_iommu=igfx_off passed to heads from coreboot config.
• t440p board config removed the whole block on KERNEL_ADD/KERNEL_REMOVE
• xx20/xx30: have coreboot linux command line argument (for Heads kernel) disable intel graphics IOMMU (intel_iommu=igfx_off, iommu not specified) while having board config pass the following to OS through KERNEL_ADD statement (iommu=on, intel_igfx=off) (which QubesOS adds by default through grub.cfg command line if not removed like the librem14 does above through KERNEL_REMOVE statement).

[tlaurion]

@srgrint @rbreslow :
could you please add and test into t440p board config:

export CONFIG_BOOT_KERNEL_ADD="intel_iommu=on intel_iommu=igfx_off"
export CONFIG_BOOT_KERNEL_REMOVE="quiet"

And report back here?

[tlaurion]

I have further investigated linux config taking x230 linux config 4.14 as base, expended it into oldconfig format and then compared to librem under 6f7f209 (branch comparison under master...tlaurion:heads:add_tools_to_ease_kernel_version_bumping)

Comments are under latest commits. Will test 5.10 converted and verified on x230-hotp-maximized to test hypothesis of similar (nothing related to FB outside of AST changes there) linux config but different iommu settings when comparing x230 with 4.14 linux kernel (now 5.x per change) to boot debian installer.

@srgrint if you see this, please point to iso used (there is so many). Will download https://cdimage.debian.org/cdimage/bookworm_di_alpha2/amd64/iso-cd/debian-bookworm-DI-alpha2-amd64-netinst.iso as you referred from comment #1323 (comment) and just dd it over usb thumb drive to bypass anything related to booting iso + detached signature.

[tlaurion]

x230-maximized master (linux 4.14 based config):

• graphical_installer boot option works, with early kernel output showing on console prior of jumping in graphical installer through i915 drm+fb drivers
• installer option (text based installer) boot option works, same early kernel ouput showing on screen soon after kexec call.

x230-maximized based on 5.10.5 kernel:

• stuck at kexec call for debian iso tested. Can reproduce.
• Boots qubes 4.1 installer and also tails
  • qubes 4.1 pass i915.alpha which is neglected by i915 driver. other then that, intel_iommu=on intel_iommu=igfx_off are passed, vgaarb is first set but erly replaced by i915....
  • Tails requires to select external hard drive boot menu option to boot.

So what's up with debian netinstaller in https://cdimage.debian.org/cdimage/bookworm_di_alpha2/amd64/iso-cd/debian-bookworm-DI-alpha2-amd64-netinst.iso would need some unpacking of its initrd and looking up what is happening there.

[srgrint]

@srgrint if you see this, please point to iso used (there is so many). Will download https://cdimage.debian.org/cdimage/bookworm_di_alpha2/amd64/iso-cd/debian-bookworm-DI-alpha2-amd64-netinst.iso as you referred from comment #1323 (comment) and just dd it over usb thumb drive to bypass anything related to booting iso + detached signature.

I usually use the debian netinst images. Usually for the stable version of debian (which at present is: https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.6.0-amd64-netinst.iso ). Given that Debian bookworm is so close to release though, I am currently doing some testing of the bookworm installers - hence also used the alpha2 image which you referred to above.

[tlaurion]

Some additional progress. Reusing https://cdimage.debian.org/cdimage/bookworm_di_alpha2/amd64/iso-cd/debian-bookworm-DI-alpha2-amd64-netinst.iso with qemu boards setups.

Some important output after kexec'ing into iso:

[    0.115754] Kernel command line: vga=788 theme=dark --- console=ttyS0 console=tty systemd.zram=0 
[    0.117317] Unknown kernel command line parameters "--- vga=788 theme=dark", will be passed to user space.

Basically, kexec-parse-boot doesn't seem to be happy with --- added in grub.cfg, which means that under Heads just appending kernel options at the end of the grub.cfg parsed multiboot line, KERNEL_ADD options are passed while some grub options aren't).
Edit: digging this up.

In our case for Librem/T440P 5.10 defconfig (added after 4.14 by default but not under 4.14):
CONFIG_INTEL_IOMMU_DEFAULT_ON=y

Might be a culprit. Testing things.

[tlaurion]

Past comment is irrelevant. Tests and improvements in PoC just removed duplicate entries where the kernel does the right thing with what is passed from command line: if kernel option is not recognised, it is passed to user space to be interpreted there in init.

[tlaurion]

One interesting thing, now being investigated was the absence of CONFIG_X86_SYSFB under linux configs for 5.x kernel, which was present under 4.14 but might have changed a lot since then:

Firmwares often provide initial graphics framebuffers so the BIOS,
bootloader or kernel can show basic video-output during boot for
user-guidance and debugging. Historically, x86 used the VESA BIOS
Extensions and EFI-framebuffers for this, which are mostly limited
to x86.
This option, if enabled, marks VGA/VBE/EFI framebuffers as generic
framebuffers so the new generic system-framebuffer drivers can be
used on x86. If the framebuffer is not compatible with the generic
modes, it is advertised as fallback platform framebuffer so legacy
drivers like efifb, vesafb and uvesafb can pick it up.
If this option is not selected, all system framebuffers are always
marked as fallback platform framebuffers as usual.

Note: Legacy fbdev drivers, including vesafb, efifb, uvesafb, will
not be able to pick up generic system framebuffers if this option
is selected. You are highly encouraged to enable simplefb as
replacement if you select this option. simplefb can correctly deal
with generic system framebuffers. But you should still keep vesafb
and others enabled as fallback if a system framebuffer is
incompatible with simplefb.

src: https://www.kernelconfig.io/config_x86_sysfb

[tlaurion]

@rbreslow @srgrint @JonathonHall-Purism @akunterkontrolle:

None of the above tests were really useful, while still continuing to investigate.
Came the momentum to go to linuxboot community and see what knowledge they have on the matter and try to rebridge some gap between our communities.

To make a short version of it: things constantly change in kernel. Right.

Seems to Heads being thought almighty in its kexec-parse-boot (syslinux+grub config parser) is still doing "the right thing" in most cases, while non-xen/multiboot2 config fall into "elf" generic "else" case for iso+normal boot path when not using QubesOS (xen/multiboot), and that no hacks there whatsoever are passed to instruct kexec to do anything different then passing path to bzimage, initrd and command line parameters. All good, but not enough.

Talking with linuxboot people made clearer a couple of things:

• kexec'ing into second kernel needs to use 32 kernel entry point from kexec. This is not the case in master.
• There are known problem as of today kexecing between LTS kernels (5.4->5.15, 5.15 -> 5.4) which depends on how those kernels were compiled (they produced a doc, put it public and then changed rights to it to be private again. Not same timezone, didn't make a local copy of it, will get back to that later)
• Their "BIOS" is EFI in their most tested case, where they use qemu directly pointing to kernel+initrd.
  • We use it through coreboot bios image through CONFIG_BOARD_EMULATION_QEMU_X86_Q3 platform under coreboot config.
  • Since we use coreboot, we also rely on libgfxinit, which is different then simply relying on linux provided drivers to setup vga
• They witnessed discrepencies through testing on real hardware vs qemu.... What was working on qemu was not working on real hardware, and vice versa.

---

Anyway those are relevant traces of really important knowledge and discussion should continue over there to resolve the situation on both qemu+real hardware #1351 (comment)

[rbreslow]

kexec'ing into second kernel needs to use 32 kernel entry point. This is not the case in master.

In our master branch or are you saying the current Linux kernel doesn't have a 32-bit entrypoint?

[tlaurion]

@rbreslow : I edited previous post.
kexec calls, from linuxboot experience, need to kexec into second kernel through second kernel's 32-bit entry point.

Rest of this discussion should happen under #1351 (comment)

[tlaurion]

@akunterkontrolle @srgrint @rbreslow It would be awesome if you guys could confirm that adding NO_GFX_INIT added on top of #1378 fixes all your issues (even memtest elf launch) AND permit to remove libgfxinit altogether under coreboot as well.

If it does, I think it might be a good time to remove libgfxinit on all boards enabling i915 kernel config.

[akunterkontrolle]

@akunterkontrolle @srgrint @rbreslow It would be awesome if you guys could confirm that adding NO_GFX_INIT added on top of #1378 fixes all your issues (even memtest elf launch) AND permit to remove libgfxinit altogether under coreboot as well.

If it does, I think it might be a good time to remove libgfxinit on all boards enabling i915 kernel config.

Sorry for not reporting back earlier. I think others have already confirmed it, but for good measurement: That pull request including NO_GFX_INIT=y does fix the issue on my t440p as well. The Debian (net)installer as well as memtest are now able to give a graphical output.

Oddities / remarks:

• The "classical" / console based ncurses-like Debian installer shows up only in black and white instead of the normal grey on blue background.
• The graphical Debian installer does boot into a very stretched out resolution.
• Memtest has no working keyboard (both with and without the option ```keyboard=legacy`` as detailed in Proper way to run memtest from heads? #535 (comment)) but since memtest doesn't really need any user inputs anyways I don't think that's really a big deal.
• Memtest shows up in a really small "window" centered on the screen. But afaik that's not heads specific but true on all coreboot based firmware.

[srgrint]

* The "classical" / console based ncurses-like Debian installer shows up only in black and white instead of the normal grey on blue background.

Is this because you have selected the "high contrast" version (ie "dark theme") of the installer?

In the boot options menu from Heads, there is a list of 37 (In bookworm RC2 netinst) boot options - including 2 "Install" options - 1 being the usual text installer and the other being "high contrast".

[akunterkontrolle]

* The "classical" / console based ncurses-like Debian installer shows up only in black and white instead of the normal grey on blue background.

Is this because you have selected the "high contrast" version (ie "dark theme") of the installer?

In the boot options menu from Heads, there is a list of 37 (In bookworm RC2 netinst) boot options - including 2 "Install" options - 1 being the usual text installer and the other being "high contrast".

For documentation purposes, in case some comes across this in the future: Yes, you are absolutely right, I choose the wrong "install" option and ended with the "high contrast" installer. If one "guesses" the right of the two install options, the normal installer will show up