Please support AMD microcode checks

[tomreyn]

While, so far, AMD prefers to make microcode updates available via firmware upgrades primarily (and checking for those is not yet implemented in needrestart - but maybe could be via fwupd), this may change in the future. Also, some systems supported by needrestart may already provide microcode updates for AMD systems (e.g. to ensure continued CPU hardware security support on systems where mainboard manufacturers no longer provide firmware upgrades, or chose against including the latest microcode upgrades in firmware upgrades they continue to provide).

So supporting tests for pending reboots to apply microcode upgrades on AMD systems may be nice to have. (Admittedly, I consider this to be of much lesser urgency than in Intel land, though.)

[liske]

I'm not aware of a generic way to check for microcode updates on AMD. Intel has iucode-tool which does most of the magic on most Linux distributions.

[yochananmarqos]

How about:

# dmesg | grep microcode
[    2.852047] microcode: CPU0: patch_level=0x0810100b
[    2.852067] microcode: CPU1: patch_level=0x0810100b
[    2.852093] microcode: CPU2: patch_level=0x0810100b
[    2.852102] microcode: CPU3: patch_level=0x0810100b
[    2.852125] microcode: CPU4: patch_level=0x0810100b
[    2.852134] microcode: CPU5: patch_level=0x0810100b
[    2.852155] microcode: CPU6: patch_level=0x0810100b
[    2.852163] microcode: CPU7: patch_level=0x0810100b
[    2.852213] microcode: Microcode Update Driver: v2.2.

See Verifying that microcode got updated on boot.

[liske]

dmesg is not usable since it prints the kernel ring buffer, only. The messages will be dropped after some time (i.e. due tof iptables rules are using the LOG target). Could you please check if the patch_level is found in /proc/cpuinfo or somewhere in /sys/devices/system/cpu/microcode?

The amd64-microcode userspace stuff seems to put all microcode files into the initramfs (/usr/share/initramfs-tools/hooks/amd64_microcode on Debian) - but needrestarts need to know which microcode update should be loaded and the expected patch_level of the file.

The analysis in Opteron Exposed: Reverse Engineering AMD K8 Microcode Updates looks promising giving us an idea howto parse the microcode files to get the available revisions and check for CPU compatibility.

[yochananmarqos]

Yes, it is:

$ cat /proc/cpuinfo | grep microcode
microcode	: 0x810100b
microcode	: 0x810100b
microcode	: 0x810100b
microcode	: 0x810100b
microcode	: 0x810100b
microcode	: 0x810100b
microcode	: 0x810100b
microcode	: 0x810100b

[liske]

I've created a small PoC script. Could you please give test-amd-ucode.pl a try? It would be nice if you could try it booting with and w/o ucode updates applied.

[yochananmarqos]

@liske

-=======[ SYSTEM ]=======-
Failed to read CPUID (Missed `modprobe cpuid`?): No such file or directory

I don't have the cpuid module.

$ ls -l /dev/cpu/0
total 0
crw------- 1 root root 202, 0 Dec 19 08:40 msr

[tomreyn]

Thanks for the PoC. This is Ryzen 7 1800X running Ubuntu 18.04.3 with a 5.3.0-24-generic kernel:

$ sudo ./test-amd-ucode.pl 
-=======[ SYSTEM ]=======-
CPUID=00800f11
UCODE=08001138


=======[ microcode_amd ]=======-
Table size = 288

CPUID=00100f80 UCODE=10800000
CPUID=00100f81 UCODE=10810000
CPUID=00100f62 UCODE=10620000
CPUID=00100f23 UCODE=10220000
CPUID=00100f43 UCODE=10430000
CPUID=00100f91 UCODE=10810000
CPUID=00100f2a UCODE=10200000
CPUID=00100f63 UCODE=10430000
CPUID=00100f42 UCODE=10410000
CPUID=00300f10 UCODE=30100000
CPUID=00200f31 UCODE=20310000
CPUID=00100f52 UCODE=10410000
CPUID=00100fa0 UCODE=10a00000
CPUID=00100f53 UCODE=10430000
CPUID=00100f22 UCODE=10220000
CPUID=00500f10 UCODE=50100000
CPUID=00500f20 UCODE=50200000

=======[ microcode_amd_fam15h ]=======-
Table size = 64

CPUID=00600f20 UCODE=60200000
CPUID=00610f01 UCODE=61010000
CPUID=00600f12 UCODE=60120000

=======[ microcode_amd_fam16h ]=======-
Table size = 32

CPUID=00700f01 UCODE=70010000

=======[ microcode_amd_fam17h ]=======-
Table size = 48

CPUID=00800f12 UCODE=80120000
CPUID=00800f82 UCODE=80820000

[tomreyn]

And this is an EPYC 7351P running Debian 10 with a 4.19.0-6-amd64 kernel:

$ sudo ./test-amd-ucode.pl
-=======[ SYSTEM ]=======-
CPUID=00800f12
UCODE=08001230


=======[ microcode_amd ]=======-
Table size = 288

CPUID=00100f80 UCODE=10800000
CPUID=00100f81 UCODE=10810000
CPUID=00100f62 UCODE=10620000
CPUID=00100f23 UCODE=10220000
CPUID=00100f43 UCODE=10430000
CPUID=00100f91 UCODE=10810000
CPUID=00100f2a UCODE=10200000
CPUID=00100f63 UCODE=10430000
CPUID=00100f42 UCODE=10410000
CPUID=00300f10 UCODE=30100000
CPUID=00200f31 UCODE=20310000
CPUID=00100f52 UCODE=10410000
CPUID=00100fa0 UCODE=10a00000
CPUID=00100f53 UCODE=10430000
CPUID=00100f22 UCODE=10220000
CPUID=00500f10 UCODE=50100000
CPUID=00500f20 UCODE=50200000

=======[ microcode_amd_fam15h ]=======-
Table size = 64

CPUID=00600f20 UCODE=60200000
CPUID=00610f01 UCODE=61010000
CPUID=00600f12 UCODE=60120000

=======[ microcode_amd_fam16h ]=======-
Table size = 32

CPUID=00700f01 UCODE=70010000

=======[ microcode_amd_fam17h ]=======-
Table size = 48

CPUID=00800f12 UCODE=80120000  UPDATE
CPUID=00800f82 UCODE=80820000

https://github.com/speed47/spectre-meltdown-checker currently reports this for it:

CPU microcode is the latest known available version:  
NO  (latest version is 0x8001250 dated 2019/04/16 according to builtin firmwares DB v130.20191104+i20191027)

[liske]

@liske

-=======[ SYSTEM ]=======-
Failed to read CPUID (Missed `modprobe cpuid`?): No such file or directory

I don't have the cpuid module.

I did expect it should be available on most systems. What kernel and linux distri are you using? Maybe the CPUID could be build from /proc/cpuinfo, although this mit be fragile. I would not like to add additional dependencies or add binary code to needrestart just to get the CPUID.

[liske]

@tomreyn Thanks for trying the PoC. It looks like for the Ryzen there are no microcode updates available. For the Epic the PoC has found an update but getting the ucode version from the update file seems to be incomplete. We still need to find a way to get the valid version number from the update file.

[liske]

I've updated the PoC test-amd-ucode.pl to revision 6:

@yochananmarqos
It is now able to get the CPUID from /proc/cpuinfo based on a script from Todd Allen's cpuid tool, please give it another try.

@tomreyn
I've extended the patch decoding and it looks like to find the (correct?) microcode revision value. It looks like you Epic CPU already has a newer microcode revision (BIOS?) running than provided by the package.

[yochananmarqos]

What kernel and linux distri are you using?

I'm using 5.4.5 on Manjaro GNOME. I forgot I hadn't mentioned that. I've got a Ryzen 5 2500U.

The new revision works:

-=======[ SYSTEM ]=======-
Failed to read CPUID (Missed `modprobe cpuid`?): No such file or directory
CPUID=00110710  (/proc/cpuinfo)
UCODE=0810100b


=======[ microcode_amd ]=======-
File size = 12684
Table size = 288

CPUID=00100f80 PRID=1080
CPUID=00100f81 PRID=1081
CPUID=00100f62 PRID=1062
CPUID=00100f23 PRID=1022
CPUID=00100f43 PRID=1043
CPUID=00100f91 PRID=1081
CPUID=00100f2a PRID=1020
CPUID=00100f63 PRID=1043
CPUID=00100f42 PRID=1041
CPUID=00300f10 PRID=3010
CPUID=00200f31 PRID=2031
CPUID=00100f52 PRID=1041
CPUID=00100fa0 PRID=10a0
CPUID=00100f53 PRID=1043
CPUID=00100f22 PRID=1022
CPUID=00500f10 PRID=5010
CPUID=00500f20 PRID=5020

PRID=1022 PID=01000083
PRID=1020 PID=01000084
PRID=1062 PID=010000c7
PRID=1043 PID=010000c8
PRID=1081 PID=010000d9
PRID=1080 PID=010000da
PRID=1041 PID=010000db
PRID=10a0 PID=010000dc
PRID=2031 PID=02000032
PRID=3010 PID=03000027
PRID=5010 PID=05000029
PRID=5020 PID=05000119

=======[ microcode_amd_fam15h ]=======-
File size = 7876
Table size = 64

CPUID=00600f20 PRID=6020
CPUID=00610f01 PRID=6101
CPUID=00600f12 PRID=6012

PRID=6012 PID=0600063e
PRID=6020 PID=06000852
PRID=6101 PID=06001119

=======[ microcode_amd_fam16h ]=======-
File size = 3510
Table size = 32

CPUID=00700f01 PRID=7001

PRID=7001 PID=0700010f

=======[ microcode_amd_fam17h ]=======-
File size = 9700
Table size = 64

CPUID=00830f10 PRID=8310
CPUID=00800f12 PRID=8012
CPUID=00800f82 PRID=8082

PRID=8012 PID=08001250
PRID=8082 PID=0800820d
PRID=8310 PID=08301025

[tomreyn]

@liske You're right, the Epic system has a 'BIOS update' (an UEFI firmware upgrade released for this Supermicro Board in summer 2019) provided microcode which is newer than what's available through Debians amd64-microcode package (Debian 10 provides package version 3.20181128.1, meaning AMD released this microcode on 2018-11-28), so the OS did not perform an early microcode update during boot.

So far, AMD primarily shipped microcode updates for Family 17h through mainboard firmware upgrades, but they have also released separate microcode upgrades to the public in a few occasions now. Unfortunately, the latest release in October caused stability problems on some systems so neither Ubuntu nor Debian ship it for now). I am, however, hopeful that AMD will continue to make microcode updates available directly to the general public so that end users will not depend on mainboard vendors to (continue to) provide those (in a timely fashion).

Output of rev6 on the same Epyc system:

$ sudo ./test-amd-ucode.pl 
-=======[ SYSTEM ]=======-
CPUID=00800f12  (/dev/cpu/0/cpuid)
CPUID mismatch: 8392466 != 1050386
UCODE=08001230


=======[ microcode_amd ]=======-
File size = 12684
Table size = 288

CPUID=00100f80 PRID=1080
CPUID=00100f81 PRID=1081
CPUID=00100f62 PRID=1062
CPUID=00100f23 PRID=1022
CPUID=00100f43 PRID=1043
CPUID=00100f91 PRID=1081
CPUID=00100f2a PRID=1020
CPUID=00100f63 PRID=1043
CPUID=00100f42 PRID=1041
CPUID=00300f10 PRID=3010
CPUID=00200f31 PRID=2031
CPUID=00100f52 PRID=1041
CPUID=00100fa0 PRID=10a0
CPUID=00100f53 PRID=1043
CPUID=00100f22 PRID=1022
CPUID=00500f10 PRID=5010
CPUID=00500f20 PRID=5020

PRID=1022 PID=01000083
PRID=1020 PID=01000084
PRID=1062 PID=010000c7
PRID=1043 PID=010000c8
PRID=1081 PID=010000d9
PRID=1080 PID=010000da
PRID=1041 PID=010000db
PRID=10a0 PID=010000dc
PRID=2031 PID=02000032
PRID=3010 PID=03000027
PRID=5010 PID=05000029
PRID=5020 PID=05000119

=======[ microcode_amd_fam15h ]=======-
File size = 7876
Table size = 64

CPUID=00600f20 PRID=6020
CPUID=00610f01 PRID=6101
CPUID=00600f12 PRID=6012

PRID=6012 PID=0600063e
PRID=6020 PID=06000852
PRID=6101 PID=06001119

=======[ microcode_amd_fam16h ]=======-
File size = 3510
Table size = 32

CPUID=00700f01 PRID=7001

PRID=7001 PID=0700010f

=======[ microcode_amd_fam17h ]=======-
File size = 6476
Table size = 48

CPUID=00800f12 PRID=8012 <<
CPUID=00800f82 PRID=8082

PRID=8012 PID=08001227 OUTDATED
PRID=8082 PID=0800820b

[liske]

The CPUID calculation from /proc/cpuinfo seems to be broken on AMD and I'm only able to test it on Intel systems. Can you @yochananmarqos @tomreyn please provide the first section of your /proc/cpuinfo?

@tomreyn There is a more recent amd64-microcode package in Debian testing. I had a look at it's ChangeLog and it should provide the microcode revision 08001250 for your Epyc CPU. Is it possible to give it a try and run the PoC after installing the update once before and once after reboot?

[yochananmarqos]

I'm using the amd-ucode 20191215.eefb5f7-1 package.

processor	: 0
vendor_id	: AuthenticAMD
cpu family	: 23
model		: 17
model name	: AMD Ryzen 5 2500U with Radeon Vega Mobile Gfx
stepping	: 0
microcode	: 0x810100b
cpu MHz		: 2019.252
cache size	: 512 KB
physical id	: 0
siblings	: 8
core id		: 0
cpu cores	: 4
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme ssbd sev ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
bugs		: sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips	: 3994.11
TLB size	: 2560 4K pages
clflush size	: 64
cache_alignment	: 64
address sizes	: 43 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate eff_freq_ro [13] [14]

[tomreyn]

/proc/cpuiinfo from the Epyc system:

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 23
model           : 1
model name      : AMD EPYC 7351P 16-Core Processor
stepping        : 2
microcode       : 0x8001230
cpu MHz         : 1199.140
cache size      : 512 KB
physical id     : 0
siblings        : 32
core id         : 0
cpu cores       : 16
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
bugs            : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 4799.18
TLB size        : 2560 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate cpb eff_freq_ro [13] [14]

/proc/cpuiinfo from the Ryzen system:

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 23
model           : 1
model name      : AMD Ryzen 7 1800X Eight-Core Processor
stepping        : 1
microcode       : 0x8001138
cpu MHz         : 2199.442
cache size      : 512 KB
physical id     : 0
siblings        : 16
core id         : 0
cpu cores       : 8
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate sme ssbd sev ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
bugs            : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 7199.80
TLB size        : 2560 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 43 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate eff_freq_ro [13] [14]

The microcode package in Debian bullseye is the one Ubuntu reverted the upgrade for (see the link to the bug - and its duplicate report - posted above). While it may work without problem on this particular Epyc system, this is a production system and I am afraid I have no means to reboot it with a different microcode (from a different Debian release) just for testing purposes. I can do further tests on the Ryzen system if this would help (please specify tests to run / information to provide)?

Maybe there is code in ME Extractor (and its microcode repository) or in spectre-meltdown-checker which you could make use of? I am not a developer myself, so can't really tell.

On a side note, a new microcode package was just released by AMD which apparently drops the problematic ones from October.

[liske]

Just another (final?) PoC update test-amd-ucode.pl revision 7.

I've updated the CPUID calculation from /proc/cpuinfo. Could you give it another try? I did some tests with cpuinfo dumps found online and it looks promising.

[yochananmarqos]

The output is the same for me with rev. 7.
EDIT: See below

[liske]

@yochananmarqos The CPUID= output in the first section (SYSTEM) should have changed to report the correct value, although there might be no microcode update avail for your cpu.

[yochananmarqos]

You're right, I missed that eyeballing it. Using diff would be too efficient. 😜️

EDIT: Updated diff after amd-ucode updated to 20191220.6871bff-1.

2c2
< CPUID=00110710  (/proc/cpuinfo)
---
> CPUID=00810f10  (/proc/cpuinfo)
62,63c62,63
< File size = 9700
< Table size = 64
---
> File size = 6476
> Table size = 48
65d64
< CPUID=00830f10 PRID=8310
71d69
< PRID=8310 PID=08301025

[liske]

I've added experimental AMD microcode support to git HEAD.