"Failed to check for processor microcode upgrades" on Arch Linux

[Wuestengecko]

I'm using needrestart on an Arch machine, packaged using this PKGBUILD (bash, essentially . ./PKGBUILD && prepare && build && package). Especially with how frequent updates roll out on Arch, this is a really handy tool to have so I do not need to reboot the whole machine twice a day. ;)
Since the update to v3.0 however I'm getting the error message

Failed to check for processor microcode upgrades.

every time I run the command from the shell (among the usual expected messages), and an "UNKNOWN" status for the Nagios plugin mode with the following line:

UNKN - Kernel: 4.15.6-1-ARCH, Microcode: unknown, Services: none, Containers: none, Sessions: none

Full log file of sudo LC_ALL=C needrestart -rl -v

Line 17 says [Kernel/Linux] Got garbage from linux image header (/boot/intel-ucode.img) - I suspect that this is what it's ultimately choking on.

On Arch, Intel microcode updates reside inside /boot/intel-ucode.img, which actually is an initrd image which must be prepended to the "regular" one by e.g.

cat /boot/intel-ucode.img /boot/initramfs-linux.img > /boot/initramfs-merged.img

Some bootloaders allow to simply list multiple initrd images, like systemd-boot (which I use) - my bootloader entry file is as follows:

title Arch Linux
linux /vmlinuz-linux
initrd /intel-ucode.img
initrd /initramfs-linux.img
options root=PARTUUID=b054d669-ad3e-4479-bfff-1f85d23561ff rw rootflags=subvol=@ quiet elevator=deadline intel_iommu=on iommu=pt nohz_full=1-3,5-7 isolcpus=1-3,5-7

Paths in lines 2-4 are relative to the EFI system partition, which is mounted during runtime at /boot.

The official method to check for new microcode, according to the Arch wiki, is comparing /proc/cpuinfo to the output of

bsdtar -Oxf /boot/intel-ucode.img | iucode_tool -tb -lS -

example output from my machine:

iucode_tool: system has processor(s) with signature 0x000306c3
microcode bundle 1: (stdin)
selected microcodes:
  001/147: sig 0x000306c3, pf_mask 0x32, 2017-11-20, rev 0x0023, size 23552

grep microcode /proc/cpuinfo gives me 8x the line

microcode       : 0x23

[liske]

Line 17 says [Kernel/Linux] Got garbage from linux image header (/boot/intel-ucode.img) - I suspect that this is what it's ultimately choking on.

This is only for the kernel image check (which seems to work for you), so this is not a problem.

needrestart relies on the output of iucode_tool -Sl -tb /lib/firmware/intel-ucode to detect the current and available microcode revisions. It might be enough to change the path to /boot to make the detection work on Arch - could you give it a try?

Since not all microcode patches are applicable for all processor types or hw revisions the microcode field in /proc/cpuinfo is completely ignored - only iucode_tool has the knowledge to detect which of the microcode updates should be applied.

[liske]

Update: you might need to add --ignore-broken to make iucode_tool work:

iucode_tool --ignore-broken -Sl -tb /lib/firmware/intel-ucode /boot

[Wuestengecko]

Plainly running iucode_tool --ignore-broken -Sl -tb /lib/firmware/intel-ucode /boot/intel-ucode.img just results in an unknown microcode format error. However, unpacking it with the bsdtar command from above does the trick:

% bsdtar -Oxf /boot/intel-ucode.img | iucode_tool --ignore-broken -Sl -tb /lib/firmware/intel-ucode -
iucode_tool: system has processor(s) with signature 0x000306c3
iucode_tool: /lib/firmware/intel-ucode: cannot open: No such file or directory
microcode bundle 1: (stdin)
selected microcodes:
  001/147: sig 0x000306c3, pf_mask 0x32, 2017-11-20, rev 0x0023, size 23552

Which seems to be the same output as the command from the wiki.

(This .img file is not built on my machine as part of an initrd generation process, but just extracted from the repo package - you might want to download it for code testing.)

[liske]

I've changed the lib/iucode-scan-versions wrapper to handle Arch's early boot initrd file. Could you please give it a try?

[Wuestengecko]

I modified the PKGBUILD to build the current git revision, which right now is 3281733.
The error message no longer appears, but it doesn't detect the actual update. I installed an older microcode package (version 20170707), rebooted to load it, then upgraded it again on disk. I then ran these commands:

# journalctl --dmesg -b 0 | grep -i microcode
Mar 29 01:24:59 zoey kernel: microcode: microcode updated early to revision 0x22, date = 2017-01-27
Mar 29 01:24:59 zoey kernel: microcode: sig=0x306c3, pf=0x2, revision=0x22
Mar 29 01:24:59 zoey kernel: microcode: Microcode Update Driver: v2.2.
# grep microcode /proc/cpuinfo
microcode       : 0x22
microcode       : 0x22
microcode       : 0x22
microcode       : 0x22
microcode       : 0x22
microcode       : 0x22
microcode       : 0x22
microcode       : 0x22
# bsdtar -Oxf /boot/intel-ucode.img | iucode_tool -tb -lS -
iucode_tool: system has processor(s) with signature 0x000306c3
microcode bundle 1: (stdin)
selected microcodes:
  001/148: sig 0x000306c3, pf_mask 0x32, 2018-01-21, rev 0x0024, size 23552
# needrestart -rl -v
[main] eval /etc/needrestart/needrestart.conf
[main] needrestart v3.1
[main] running in root mode
[Core] Using UI 'NeedRestart::UI::stdio'...
[main] systemd detected
[Core] #517 is a NeedRestart::Interp::Python
[Python] #517: source=/usr/bin/carbon-cache.py
+ type bsdtar
+ imgfiles=
+ for img in /boot/intel-ucode.img /boot/early_ucode.cpio
+ '[' -r /boot/intel-ucode.img ']'
+ imgfiles=' /boot/intel-ucode.img'
+ for img in /boot/intel-ucode.img /boot/early_ucode.cpio
+ '[' -r /boot/early_ucode.cpio ']'
+ '[' -n ' /boot/intel-ucode.img' ']'
+ cat /boot/intel-ucode.img
+ bsdtar -Oxf /dev/stdin
+ iucode_tool -Sl -tb -
[uCode/Intel] current signature: 0x000306c3
[uCode/Intel] available signature: 0x000306c3
+ exit 0
[Kernel] Linux: kernel release 4.15.13-1-ARCH, kernel version #1 SMP PREEMPT Sun Mar 25 11:27:57 UTC 2018
[Kernel/Linux] /boot/vmlinuz-linux => 4.15.13-1-ARCH (builduser@heftig-3811) #1 SMP PREEMPT Sun Mar 25 11:27:57 UTC 2018 [4.15.13-1-ARCH]*
[Kernel/Linux] Got garbage from linux image header (/boot/intel-ucode.img): 'ÎX£á
�º-T�8ðwîÖÎÓÝÎÒ%Âe÷£cl B3/xj²%©úì¾n6ê«É¼Nú³Ç=�ñô,p�Q·X4>¡é���½±Q�y�qí�}�©®Û�©½ï~Ùí�zÎ�{ß��´[����,¸�ÁMÚ�¬xýÅ�ãÁWµ�@LC&�ãX¿'
++ mktemp
+ tmp=/tmp/tmp.hXcKSngG27
+ trap 'rm -f /tmp/tmp.hXcKSngG27' 0
+ get_version /boot/intel-ucode.img
+ strings /boot/intel-ucode.img
+ grep -m 1 '^Linux version '
+ which gunzip
+ try_decompress '\037\213\010' xy gunzip
++ tr '\037\213\010\nxy' '\nxy='
++ grep -abo '^xy'
+ which unxz
+ try_decompress '\3757zXZ\000' abcde unxz
++ tr '\3757zXZ\000\nabcde' '\nabcde='
++ grep -abo '^abcde'
+ which bunzip2
+ try_decompress BZh xy bunzip2
++ tr 'BZh\nxy' '\nxy='
++ grep -abo '^xy'
+ which unlzma
+ try_decompress '\135\0\0\0' xxx unlzma
++ tr '\135\0\0\0\nxxx' '\nxxx='
++ grep -abo '^xxx'
+ for pos in `tr "$1\n$2" "\n$2=" < "$img" | grep -abo "^$2"`
+ pos=135341
+ tail -c+135341 /boot/intel-ucode.img
+ unlzma
+ get_version /tmp/tmp.hXcKSngG27
+ strings /tmp/tmp.hXcKSngG27
+ grep -m 1 '^Linux version '
+ for pos in `tr "$1\n$2" "\n$2=" < "$img" | grep -abo "^$2"`
+ pos=305305
+ tail -c+305305 /boot/intel-ucode.img
+ unlzma
+ get_version /tmp/tmp.hXcKSngG27
+ strings /tmp/tmp.hXcKSngG27
+ grep -m 1 '^Linux version '
+ which lzop
+ try_decompress '\211\114\132' xy 'lzop -d'
++ tr '\211\114\132\nxy' '\nxy='
++ grep -abo '^xy'
+ rm -f /tmp/tmp.hXcKSngG27
[Kernel/Linux] Could not get version string from /boot/intel-ucode.img.
[Kernel/Linux] Expected linux version: 4.15.13-1-ARCH

Running kernel seems to be up-to-date.

The processor microcode seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

After one more reboot, the journal and /proc/cpuinfo confirm that revision 0x24 actually applies:

# journalctl --dmesg -b 0 |grep -i microcode
Mar 29 01:38:14 zoey kernel: microcode: microcode updated early to revision 0x24, date = 2018-01-21
Mar 29 01:38:14 zoey kernel: microcode: sig=0x306c3, pf=0x2, revision=0x24
Mar 29 01:38:14 zoey kernel: microcode: Microcode Update Driver: v2.2.
# grep microcode /proc/cpuinfo
microcode       : 0x24
microcode       : 0x24
microcode       : 0x24
microcode       : 0x24
microcode       : 0x24
microcode       : 0x24
microcode       : 0x24
microcode       : 0x24

[liske]

The microcode detection using iucode_tool now seems to work the same way as it does on Debian.

But theres is another (severe) issue:

Needrestart currently uses the signature value to detect a difference between running and available microcode revisions which is wrong. I've opened #108 for this new issue.

[liske]

Could you give git HEAD another try? Thanks!

[Wuestengecko]

ee6042a seems to work now. I downgraded the microcode on disk without rebooting yet, and it detects the change:

Pending processor microcode upgrade!

Diagnostics:
  The currently running processor microcode revision is 0x24 which is not the expected microcode revision 0x22.

Restarting the system to load the new processor microcode will not be handled automatically, so you should consider
rebooting. [Return]