Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node not ready after cluster upgrade to 4.7.0-0.okd-2021-08-22-163618 #878

Closed
gialloguitar opened this issue Sep 17, 2021 · 16 comments
Closed

Node not ready after cluster upgrade to 4.7.0-0.okd-2021-08-22-163618 #878

gialloguitar opened this issue Sep 17, 2021 · 16 comments
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.

Comments

@gialloguitar
Copy link

gialloguitar commented Sep 17, 2021 •
edited
Loading

Describe the bug
After latest cluster upgrade unexpectedly some worker nodes become NotReady, and just only hard reboot restore it.
Node is unavalible with a network, all pods stucked in terminating state from this node.

Version
OKD 4.7.0-0.okd-2021-08-22-163618
IPI installation with Openstack provider
Fedora CoreOS image 34.20210808.3.0

How reproducible
It happening randomly with one or two node per time. All nodes working on dedicated hypervisors without compute resource overcommitting

Log bundle
ClusterID: 47c3fb43-737e-4d4f-a992-e304ba338430
ClusterVersion: Stable at "4.7.0-0.okd-2021-08-22-163618"
ClusterOperators:
All healthy and stable
@JaimeMagiera
Copy link
Contributor

By "log bundle", we're asking you to use the must-gather tool and post the resulting archive file for download. Details here...

https://docs.okd.io/latest/support/gathering-cluster-data.html

@yaroslavkasatikov
Copy link

Hi team, I got the same issue with the same version of OKD in AWS.

Here is the console output (sorry can't find a way of full log extraction from aws console):

systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
ager Script Dispatcher Service.
[   84.735580] audit: type=1130 audit(1631952592.997:167): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher co
mm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[   84.674803] configure-ovs.sh[1234]: + '[' 11 -lt 12 ']'
[   84.676304] configure-ovs.sh[1341]: ++ ip route show default
[   84.677549] configure-ovs.sh[1342]: ++ awk '{ if ($4 == "dev") { print $5; exit } }'
[   84.681646] configure-ovs.sh[1234]: + iface=
[   84.704551] configure-ovs.sh[1234]: + [[ -n '' ]]
[   84.705595] configure-ovs.sh[1345]: ++ awk '{ if ($4 == "dev") { print $5; exit } }'
[   84.707164] configure-ovs.sh[1344]: ++ ip -6 route show default
[   84.709173] configure-ovs.sh[1234]: + iface=
[   84.804762] configure-ovs.sh[1234]: + [[ -n '' ]]
[   84.805635] configure-ovs.sh[1234]: + counter=12
[   84.806395] configure-ovs.sh[1234]: + echo 'No default route found on attempt: 12'
[   84.807937] configure-ovs.sh[1234]: No default route found on attempt: 12
[   84.905080] configure-ovs.sh[1234]: + sleep 5
[   89.684418] configure-ovs.sh[1234]: + '[' 12 -lt 12 ']'
[   89.685574] configure-ovs.sh[1234]: + '[' '' = br-ex ']'
[FAILED] Failed to start Configures…host networking configuration.
[   89.686510] configure-ovs.sh[1234]: + '[' -z '' ']'
See 'systemctl status ovs-configuration.service' for details.
[   89.705049] c[   90.515100] audit: type=1130 audit(1631952598.798:168): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=ovs-configur
ation comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
onfigure-ovs.sh[1234]: + echo 'ERROR: Unable to find default gateway interface'
         Starting Ensure the node h…me is valid for the cluster...
[   89.836303] configure-ovs.sh[1234]: ERROR: Unable to find default gateway interface
[   89.906508] configure-ovs.sh[1234]: + exit 1
[ ***  ] (1 of 2) A start job is running for…n (SSH Keys) (1min 20s / no limit)
[   94.822603] audit: type=1131 audit(1631952603.106:169): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=NetworkManager-dispatcher co
[FAILED] Failed to start Afterburn (SSH Keys).
See 'systemctl status afterburn-sshkeys@core.service' for details.
[DEPEND] Dependency failed for Sync…s@.service template instances.
[DEPEND] Dependency failed for Multi-User System.
[  105.114783] audit: type=1130 audit(1631952613.398:170): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=afterburn-sshkeys@core comm=
"systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
         Starting Record Runlevel Change in UTMP...
[  OK  ] Finished Record Runlevel Change in UTMP.```


It seems we have issue with network. Nodes is avaliable via console (I can't login b/c didn't set password for core user, but I can press enter on login script in user/pass fields - so node is alive. But it absolutely inaccessible from network. (no ping, ssh etc).

If I reboot the node, it may be loaded successfully but goes reboot after some time and got NotReady after the reboot.

I got such bug on t3.2xlarge and t3a.2xlarge.
@JaimeMagiera 
Will collect logs and attach shortly

@yaroslavkasatikov
Copy link

A bit more console log. Seems the Kernel Panic is the reason of Reboots:

[ 1835.781871] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1835.804610] audit: type=1327 audit(1631952425.511:2300): proctitle=2F7362696E2F69707461626C6573002D41004F5554505554002D7000746370002D6D00746370002D2D64706F7274003232363233002D2D73796E002D6A0052454A454354
[ 1835.804623] audit: type=1325 audit(1631952425.511:2301): table=filter family=2 entries=4 op=xt_replace pid=61868 subj=system_u:system_r:spc_t:s0 comm="iptables"
[ 1835.804629] audit: type=1300 audit(1631952425.511:2301): arch=c000003e syscall=54 success=yes exit=0 a0=4 a1=0 a2=40 a3=55f75ca3f890 items=0 ppid=2579 pid=61868 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-legacy-multi" subj=system_u:system_r:spc_t:s0 key=(null)
[ 1835.804634] audit: type=1327 audit(1631952425.511:2301): proctitle=2F7362696E2F69707461626C6573002D41004F5554505554002D7000746370002D6D00746370002D2D64706F7274003232363233002D2D73796E002D6A0052454A454354
[ 1835.804643] audit: type=1325 audit(1631952425.520:2302): table=filter family=2 entries=5 op=xt_replace pid=61869 subj=system_u:system_r:spc_t:s0 comm="iptables"
[ 1835.993912] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1836.002488] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1836.019853] 94d727c361a7823: renamed from veth88b3e2c6
[ 1836.051882] cd1fd95652cb8b9: renamed from veth822fa7be
[ 1836.073230] 75b1e8af25f8fde: renamed from vethf2b3b1e3
[ 1836.087182] IPv6: ADDRCONF(NETDEV_CHANGE): cd1fd95652cb8b9: link becomes ready
[ 1836.104361] device cd1fd95652cb8b9 entered promiscuous mode
[ 1836.110432] device 94d727c361a7823 entered promiscuous mode
[ 1836.141102] device 75b1e8af25f8fde entered promiscuous mode
[ 1836.158862] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1836.179439] a690a063b9c4d8d: renamed from vetha6be4c2b
[ 1836.232536] device a690a063b9c4d8d entered promiscuous mode
[ 1836.270418] device 94d727c361a7823 left promiscuous mode
[ 1837.133951] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1837.149982] 62eb09234222bf7: renamed from veth7c2ad10c
[ 1837.178867] device 62eb09234222bf7 entered promiscuous mode
[ 1839.171275] device 2e580946f8b6f89 left promiscuous mode
[ 1840.163254] device 75b1e8af25f8fde left promiscuous mode
[ 1841.167725] device cd1fd95652cb8b9 left promiscuous mode
[ 1841.171873] kauditd_printk_skb: 162 callbacks suppressed
[ 1841.171876] audit: type=1700 audit(1631952430.919:2357): dev=cd1fd95652cb8b9 prom=0 old_prom=256 auid=4294967295 uid=981 gid=977 ses=4294967295
[ 1841.184783] audit: type=1300 audit(1631952430.919:2357): arch=c000003e syscall=46 success=yes exit=32 a0=46 a1=7ffcbf28d170 a2=0 a3=559ff4a258a8 items=0 ppid=1 pid=1010 auid=4294967295 uid=981 gid=977 euid=981 suid=981 fsuid=981 egid=977 sgid=977 fsgid=977 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null)
[ 1841.206672] audit: type=1327 audit(1631952430.919:2357): proctitle=6F76732D767377697463686400756E69783A2F7661722F72756E2F6F70656E767377697463682F64622E736F636B002D76636F6E736F6C653A656D6572002D767379736C6F673A657272002D7666696C653A696E666F002D2D6D6C6F636B616C6C002D2D75736572006F70656E767377697463683A68756765746C626673002D
[ 1841.210106] device 62eb09234222bf7 left promiscuous mode
[ 1841.231894] audit: type=1700 audit(1631952430.948:2358): dev=62eb09234222bf7 prom=0 old_prom=256 auid=4294967295 uid=981 gid=977 ses=4294967295
[ 1841.240570] audit: type=1300 audit(1631952430.948:2358): arch=c000003e syscall=46 success=yes exit=32 a0=1c a1=7ffcbf28d170 a2=0 a3=559ff4a725c8 items=0 ppid=1 pid=1010 auid=4294967295 uid=981 gid=977 euid=981 suid=981 fsuid=981 egid=977 sgid=977 fsgid=977 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null)
[ 1841.261905] audit: type=1327 audit(1631952430.948:2358): proctitle=6F76732D767377697463686400756E69783A2F7661722F72756E2F6F70656E767377697463682F64622E736F636B002D76636F6E736F6C653A656D6572002D767379736C6F673A657272002D7666696C653A696E666F002D2D6D6C6F636B616C6C002D2D75736572006F70656E767377697463683A68756765746C626673002D
[ 1841.262069] device a690a063b9c4d8d left promiscuous mode
[ 1841.285467] audit: type=1700 audit(1631952430.986:2359): dev=a690a063b9c4d8d prom=0 old_prom=256 auid=4294967295 uid=981 gid=977 ses=4294967295
[ 1841.294656] audit: type=1300 audit(1631952430.986:2359): arch=c000003e syscall=46 success=yes exit=32 a0=1c a1=7ffcbf28d170 a2=0 a3=559ff4a258a8 items=0 ppid=1 pid=1010 auid=4294967295 uid=981 gid=977 euid=981 suid=981 fsuid=981 egid=977 sgid=977 fsgid=977 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null)
[ 1841.316301] audit: type=1327 audit(1631952430.986:2359): proctitle=6F76732D767377697463686400756E69783A2F7661722F72756E2F6F70656E767377697463682F64622E736F636B002D76636F6E736F6C653A656D6572002D767379736C6F673A657272002D7666696C653A696E666F002D2D6D6C6F636B616C6C002D2D75736572006F70656E767377697463683A68756765746C626673002D
[ 1842.606911] audit: type=1325 audit(1631952432.359:2360): table=filter family=2 entries=12 op=xt_unregister pid=42337 subj=system_u:system_r:kernel_t:s0 comm="kworker/u16:1"
[ 1896.752130] a498cfcd5865aaa: renamed from veth6027f331
[ 1896.773530] IPv6: ADDRCONF(NETDEV_CHANGE): a498cfcd5865aaa: link becomes ready
[ 1896.783112] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1896.809969] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1896.828219] 6622f00418ad6ff: renamed from vethcbdd9713
[ 1896.846301] device a498cfcd5865aaa entered promiscuous mode
[ 1896.851398] kauditd_printk_skb: 4 callbacks suppressed
[ 1896.851400] audit: type=1700 audit(1631952486.579:2365): dev=a498cfcd5865aaa prom=256 old_prom=0 auid=4294967295 uid=981 gid=977 ses=4294967295
[ 1896.870891] audit: type=1300 audit(1631952486.579:2365): arch=c000003e syscall=46 success=yes exit=60 a0=40 a1=7ffcbf28cf90 a2=0 a3=7ffcbf28dcc4 items=0 ppid=1 pid=1010 auid=4294967295 uid=981 gid=977 euid=981 suid=981 fsuid=981 egid=977 sgid=977 fsgid=977 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null)
[ 1896.883638] device 6622f00418ad6ff entered promiscuous mode
[ 1896.896462] audit: type=1327 audit(1631952486.579:2365): proctitle=6F76732D767377697463686400756E69783A2F7661722F72756E2F6F70656E767377697463682F64622E736F636B002D76636F6E736F6C653A656D6572002D767379736C6F673A657272002D7666696C653A696E666F002D2D6D6C6F636B616C6C002D2D75736572006F70656E767377697463683A68756765746C626673002D
[ 1896.941096] audit: type=1700 audit(1631952486.622:2366): dev=6622f00418ad6ff prom=256 old_prom=0 auid=4294967295 uid=981 gid=977 ses=4294967295
[ 1896.963448] audit: type=1300 audit(1631952486.622:2366): arch=c000003e syscall=46 success=yes exit=60 a0=1c a1=7ffcbf28cf90 a2=0 a3=7ffcbf28dcc4 items=0 ppid=1 pid=1010 auid=4294967295 uid=981 gid=977 euid=981 suid=981 fsuid=981 egid=977 sgid=977 fsgid=977 tty=(none) ses=4294967295 comm="ovs-vswitchd" exe="/usr/sbin/ovs-vswitchd" subj=system_u:system_r:openvswitch_t:s0 key=(null)
[ 1896.996457] audit: type=1327 audit(1631952486.622:2366): proctitle=6F76732D767377697463686400756E69783A2F7661722F72756E2F6F70656E767377697463682F64622E736F636B002D76636F6E736F6C653A656D6572002D767379736C6F673A657272002D7666696C653A696E666F002D2D6D6C6F636B616C6C002D2D75736572006F70656E767377697463683A68756765746C626673002D
[ 1897.023461] audit: type=1325 audit(1631952486.653:2367): table=filter family=2 entries=0 op=xt_register pid=63733 subj=system_u:system_r:spc_t:s0 comm="iptables"
[ 1897.059721] audit: type=1300 audit(1631952486.653:2367): arch=c000003e syscall=55 success=yes exit=0 a0=4 a1=0 a2=40 a3=7ffd169419f0 items=0 ppid=2579 pid=63733 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="iptables" exe="/usr/sbin/xtables-legacy-multi" subj=system_u:system_r:spc_t:s0 key=(null)
[ 1897.103166] audit: type=1327 audit(1631952486.653:2367): proctitle=2F7362696E2F69707461626C6573002D41004F5554505554002D7000746370002D6D00746370002D2D64706F7274003232363233002D2D73796E002D6A0052454A454354
[ 1897.117377] audit: type=1325 audit(1631952486.653:2368): table=filter family=2 entries=4 op=xt_replace pid=63733 subj=system_u:system_r:spc_t:s0 comm="iptables"
[ 1897.152216] device a498cfcd5865aaa left promiscuous mode
[ 1897.156805] 0dbdb5a05c1a4b1: renamed from vethb724fe48
[ 1897.205631] device 0dbdb5a05c1a4b1 entered promiscuous mode
[ 1897.248965] f248fc7577f5876: renamed from vethad8c4437
[ 1897.355313] ac89e20735172fa: renamed from veth6b1fbd32
[ 1897.377543] device f248fc7577f5876 entered promiscuous mode
[ 1897.452136] device ac89e20735172fa entered promiscuous mode
[ 1897.520499] device ac89e20735172fa left promiscuous mode
[ 1898.445194] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1898.451241] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[ 1898.467892] 4f8cd2c233b4ab7: renamed from veth877096e9
[ 1898.505350] e37555235d2249a: renamed from vethac398c5c
[ 1898.519355] IPv6: ADDRCONF(NETDEV_CHANGE): e37555235d2249a: link becomes ready
[ 1898.532566] device 4f8cd2c233b4ab7 entered promiscuous mode
[ 1898.558253] device e37555235d2249a entered promiscuous mode
[ 1900.229818] general protection fault, probably for non-canonical address 0xee536d3054c44e6b: 0000 [#1] SMP PTI
[ 1900.239002] CPU: 4 PID: 0 Comm: swapper/4 Not tainted 5.13.4-200.fc34.x86_64 #1
[ 1900.246102] Hardware name: Amazon EC2 t3.2xlarge/, BIOS 1.0 10/16/2017
[ 1900.251825] RIP: 0010:update_blocked_averages+0x220/0x660
[ 1900.256319] Code: 4c 89 fe 4c 89 c7 e8 cf 6a 01 00 85 c0 0f 85 6b 02 00 00 49 8b 87 50 01 00 00 48 8b 80 c8 00 00 00 4a 8b 34 28 48 85 f6 74 31 <48> 83 be a0 01 00 00 00 48 8b 86 58 01 00 00 0f 85 b4 02 00 00 48
[ 1900.272650] RSP: 0018:ffffaa2dc0178f20 EFLAGS: 00010082
[ 1900.277594] RAX: ffff9ab1e2a3d1c0 RBX: 0000000000000000 RCX: 0000000000000000
[ 1900.283221] RDX: 0000000000000001 RSI: ee536d3054c44ccb RDI: 6e5045a3d20779b7
[ 1900.289113] RBP: ffff9ab1f2f49800 R08: 00000000000001f3 R09: 00000000d744fcc9
[ 1900.293953] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9ab1e8526d40
[ 1900.425713] R13: 0000000000000020 R14: ffff9ab86972b5d0 R15: ffff9ab1e8526c00
[ 1900.430553] FS:  0000000000000000(0000) GS:ffff9ab869700000(0000) knlGS:0000000000000000
[ 1900.437481] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1900.441695] CR2: 000000c000369000 CR3: 000000019d200001 CR4: 00000000007706e0
[ 1900.447120] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1900.451863] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1900.456476] PKRU: 55555554
[ 1900.459306] Call Trace:
[ 1900.461999]  <IRQ>
[ 1900.464508]  ? __hrtimer_run_queues+0x149/0x250
[ 1900.468060]  run_rebalance_domains+0x44/0x60
[ 1900.471465]  __do_softirq+0xd0/0x28f
[ 1900.474597]  __irq_exit_rcu+0xcc/0x100
[ 1900.477838]  sysvec_apic_timer_interrupt+0x72/0x90
[ 1900.481428]  </IRQ>
[ 1900.483995]  asm_sysvec_apic_timer_interrupt+0x12/0x20
[ 1900.487775] RIP: 0010:native_safe_halt+0xe/0x10
[ 1900.491352] Code: c0 7b 01 00 f0 80 4a 02 20 48 8b 12 83 e2 08 75 c3 e9 7a ff ff ff cc cc cc cc cc cc e9 07 00 00 00 0f 00 2d 26 db 5e 00 fb f4 <c3> 90 e9 07 00 00 00 0f 00 2d 16 db 5e 00 f4 c3 cc cc 0f 1f 44 00
[ 1900.503767] RSP: 0018:ffffaa2dc00a7e88 EFLAGS: 00000246
[ 1900.507596] RAX: 0000000000004000 RBX: 0000000000000001 RCX: 00000000ffffffff
[ 1900.512195] RDX: ffff9ab869700000 RSI: ffff9ab141036800 RDI: ffff9ab141036864
[ 1900.516855] RBP: ffff9ab141036864 R08: ffffffff96e67500 R09: 0000000000000007
[ 1900.521420] R10: 000000000000002a R11: 0000000000000019 R12: 0000000000000001
[ 1900.526012] R13: ffffffff96e67580 R14: 0000000000000001 R15: 0000000000000000
[ 1900.530523]  acpi_idle_do_entry+0x46/0x50
[ 1900.533835]  acpi_idle_enter+0x86/0xc0
[ 1900.537047]  cpuidle_enter_state+0x89/0x350
[ 1900.540421]  cpuidle_enter+0x29/0x40
[ 1900.543519]  do_idle+0x1ce/0x270
[ 1900.546497]  cpu_startup_entry+0x19/0x20
[ 1900.549764]  secondary_startup_64_no_verify+0xc2/0xcb
[ 1900.553521] Modules linked in: nf_conntrack_netlink veth xt_addrtype ipt_REJECT nf_reject_ipv4 xt_nat iptable_raw xt_CT iptable_mangle xt_MASQUERADE xt_conntrack xt_comment iptable_filter xt_mark ip6table_nat ip6_tables iptable_nat rfkill geneve ip6_udp_tunnel udp_tunnel nfnetlink openvswitch nsh nf_conncount nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 overlay intel_rapl_msr intel_rapl_common isst_if_common nfit libnvdimm rapl ppdev parport_pc parport i2c_piix4 drm ip_tables xfs crct10dif_pclmul crc32_pclmul crc32c_intel nvme ena ghash_clmulni_intel nvme_core serio_raw dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua fuse
[ 1900.588648] ---[ end trace 18e8a7510291b57d ]---
[ 1900.592184] RIP: 0010:update_blocked_averages+0x220/0x660
[ 1900.596084] Code: 4c 89 fe 4c 89 c7 e8 cf 6a 01 00 85 c0 0f 85 6b 02 00 00 49 8b 87 50 01 00 00 48 8b 80 c8 00 00 00 4a 8b 34 28 48 85 f6 74 31 <48> 83 be a0 01 00 00 00 48 8b 86 58 01 00 00 0f 85 b4 02 00 00 48
[ 1900.608425] RSP: 0018:ffffaa2dc0178f20 EFLAGS: 00010082
[ 1900.612204] RAX: ffff9ab1e2a3d1c0 RBX: 0000000000000000 RCX: 0000000000000000
[ 1900.616756] RDX: 0000000000000001 RSI: ee536d3054c44ccb RDI: 6e5045a3d20779b7
[ 1900.621319] RBP: ffff9ab1f2f49800 R08: 00000000000001f3 R09: 00000000d744fcc9
[ 1900.625875] R10: 0000000000000000 R11: 0000000000000000 R12: ffff9ab1e8526d40
[ 1900.630488] R13: 0000000000000020 R14: ffff9ab86972b5d0 R15: ffff9ab1e8526c00
[ 1900.635069] FS:  0000000000000000(0000) GS:ffff9ab869700000(0000) knlGS:0000000000000000
[ 1900.641542] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1900.645619] CR2: 000000c000369000 CR3: 000000019d200001 CR4: 00000000007706e0
[ 1900.650244] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1900.654797] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 1900.659412] PKRU: 55555554
[ 1900.662220] Kernel panic - not syncing: Fatal exception in interrupt
[ 1901.732137] Shutting down cpus with NMI
[ 1901.735456] Kernel Offset: 0x14000000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1901.742728] Rebooting in 10 seconds..
�[H�[J�[1;1H�[?25l�[m�[H�[J�[1;1H�[20;7H�[mUse the ^ and v keys to change the selection.                       

      Press 'e' to edit the selected item, or 'c' for a command prompt.   �[4;80H �[7m�[4;1H      Fedora CoreOS 34 47.34.202108121216-0 (ostree:0)                         �[m�[4;79H�[m�[m�[5;1H      Fedora CoreOS 34.20210711.3.0 (ostree:1)                                 �[m�[5;79H�[m�[m�[6;1H                                                                               �[m�[6;79H�[m�[m�[7;1H                                                                               �[m�[7;79H�[m�[m�[8;1H                                                                               �[m�[8;79H�[m�[m�[9;1H                                                                               �[m�[9;79H�[m�[m�[10;1H                                                                               �[m�[10;79H�[m�[m�[11;1H                                                                               �[m�[11;79H�[m�[m�[12;1H                                                                               �[m�[12;79[2021-09-18T08:08:23.540841]H�[m�[m�[13;1H                                                                               �[m�[13;79H�[m�[m�[14;1H                                                                               �[m�[14;79H�[m�[m�[15;1H                                                                               �[m�[15;79H�[m�[m�[16;1H                                                                               �[m�[16;79H�[m�[m�[17;1H                                                                               �[m�[17;79H�[m�[m�[18;1H                                                                               �[m�[18;79H�[m�[18;80H �[4;79H�[22;1H   The selected entry will be started automatically in 1s.                     �[4;79H�[22;1H   The selected entry will be started automatically in 0s.                     �[4;79H�[?25h�[H�[J�[1;1H�[H�[J�[1;1H[    0.000000] Linux version 5.13.4-200.fc34.x86_64 (mockbuild@bkernel01.iad2.fedoraproject.org) (gcc (GCC) 11.1.1 20210531 (Red Hat 11.1.1-3[2021-09-18T08:08:29.440859]), GNU ld version 2.35.1-41.fc34) #1 SMP Tue Jul 20 20:27:29 UTC 2021
[    0.000000] Command line: BOOT_IMAGE=(hd0,gpt3)/ostree/fedora-coreos-f8c4206cdcb01c91cb5d193dff2ac8df01b5399e73ba7230990e014c549d0ff3/vmlinuz-5.13.4-200.fc34.x86_64 console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.0/fedora-coreos/f8c4206cdcb01c91cb5d193dff2ac8df01b5399e73ba7230990e014c549d0ff3/0 ignition.platform.id=aws root=UUID=03a94e54-4675-4e26-86cb-1852306d4ce6 rw rootflags=prjquota systemd.unified_cgroup_hierarchy=0
[    0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x008: 'MPX bounds registers'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x010: 'MPX CSR'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x020: 'AVX-512 opmask'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x040: 'AVX-512 Hi256'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x080: 'AVX-512 ZMM_Hi256'
[    0.000000] x86/fpu: Supporting XSAVE feature 0x200: 'Protection Keys User registers'
[    0.000000] x86/fpu: xstate_offset[2]:  576, xstate_sizes[2]:  256
[    0.000000] x86/fpu: xstate_offset[3]:  832, xstate_sizes[3]:   64
[    0.000000] x86/fpu: xstate_offset[4]:  896, xstate_sizes[4]:   64
[    0.000000] x86/fpu: xstate_offset[5]:  960, xstate_sizes[5]:   64
[    0.000000] x86/fpu: xstate_offset[6]: 1024, xstate_sizes[6]:  512
[    0.000000] x86/fpu: xstate_offset[7]: 1536, xstate_sizes[7]: 1024
[    0.000000] x86/fpu: xstate_offset[9]: 2560, xstate_sizes[9]:    8
[    0.000000] x86/fpu: Enabled xstate features 0x2ff, context size is 2568 bytes, using 'compacted' format.
[    0.000000] BIOS-provided physical RAM map:
[    0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
[    0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffe8fff] usable
[    0.000000] BIOS-e820: [mem 0x00000000bffe9000-0x00000000bfffffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000e0000000-0x00000000e03fffff] reserved
[    0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved
[    0.000000] BIOS-e820: [mem 0x0000000100000000-0x00000008299fffff] usable
[    0.000000] BIOS-e820: [mem 0x0000000829a00000-0x000000083fffffff] reserved
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] SMBIOS 2.7 present.
[    0.000000] DMI: Amazon EC2 t3.2xlarge/, BIOS 1.0 10/16/2017
[    0.000000] Hypervisor detected: KVM
[    0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00
[    0.000000] kvm-clock: cpu 0, msr 60aa01001, primary cpu clock
[    0.000001] kvm-clock: using sched offset of 5851992278 cycles
[    0.000003] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns
[    0.000007] tsc: Detected 2499.998 MHz processor
[    0.000344] last_pfn = 0x829a00 max_arch_pfn = 0x400000000
[    0.000386] x86/PAT: Configuration [0-7]: WB  WC  UC- UC  WB  WP  UC- WT  
[    0.000392] last_pfn = 0xbffe9 max_arch_pfn = 0x400000000
[    0.007400] Using GB pages for direct mapping
[    0.007635] RAMDISK: [mem 0x2ec16000-0x33602fff]
[    0.007647] ACPI: Early table checksum verification disabled
[    0.007653] ACPI: RSDP 0x00000000000F8F40 000014 (v00 AMAZON)
[    0.007659] ACPI: RSDT 0x00000000BFFEE090 000040 (v01 AMAZON AMZNRSDT 00000001 AMZN 00000001)
[    0.007664] ACPI: FACP 0x00000000BFFEFF80 000074 (v01 AMAZON AMZNFACP 00000001 AMZN 00000001)
[    0.007669] ACPI: DSDT 0x00000000BFFEE0D0 0010E9 (v01 AMAZON AMZNDSDT 00000001 AMZN 00000001)
[    0.007672] ACPI: FACS 0x00000000BFFEFF40 000040
[    0.007674] ACPI: SSDT 0x00000000BFFEF480 000AC0 (v01 AMAZON AMZNSSDT 00000001 AMZN 00000001)
[    0.007678] ACPI: APIC 0x00000000BFFEF360 0000A6 (v01 AMAZON AMZNAPIC 00000001 AMZN 00000001)
[    0.007681] ACPI: SRAT 0x00000000BFFEF260 000100 (v01 AMAZON AMZNSRAT 00000001 AMZN 00000001)
[    0.007683] ACPI: SLIT 0x00000000BFFEF1F0 00006C (v01 AMAZON AMZNSLIT 00000001 AMZN 00000001)
[    0.007686] ACPI: WAET 0x00000000BFFEF1C0 000028 (v01 AMAZON AMZNWAET 00000001 AMZN 00000001)
[    0.007691] ACPI: HPET 0x00000000000C9000 000038 (v01 AMAZON AMZNHPET 00000001 AMZN 00000001)
[    0.007693] ACPI: Reserving FACP table memory at [mem 0xbffeff80-0xbffefff3]
[    0.007695] ACPI: Reserving DSDT table memory at [mem 0xbffee0d0-0xbffef1b8]
[    0.007696] ACPI: Reserving FACS table memory at [mem 0xbffeff40-0xbffeff7f]
[    0.007697] ACPI: Reserving SSDT table memory at [mem 0xbffef480-0xbffeff3f]
[    0.007698] ACPI: Reserving APIC table memory at [mem 0xbffef360-0xbffef405]
[    0.007699] ACPI: Reserving SRAT table memory at [mem 0xbffef260-0xbffef35f]
[    0.007700] ACPI: Reserving SLIT table memory at [mem 0xbffef1f0-0xbffef25b]
[    0.007701] ACPI: Reserving WAET table memory at [mem 0xbffef1c0-0xbffef1e7]
[    0.007702] ACPI: Reserving HPET table memory at [mem 0xc9000-0xc9037]
[    0.007753] SRAT: PXM 0 -> APIC 0x00 -> Node 0
[    0.007755] SRAT: PXM 0 -> APIC 0x01 -> Node 0
[    0.007757] SRAT: PXM 0 -> APIC 0x02 -> Node 0
[    0.007759] SRAT: PXM 0 -> APIC 0x03 -> Node 0
[    0.007761] SRAT: PXM 0 -> APIC 0x04 -> Node 0
[    0.007763] SRAT: PXM 0 -> APIC 0x05 -> Node 0
[    0.007765] SRAT: PXM 0 -> APIC 0x06 -> Node 0
[    0.007768] SRAT: PXM 0 -> APIC 0x07 -> Node 0
[    0.007771] ACPI: SRAT: Node 0 PXM 0 [mem 0x00000000-0xbfffffff]
[    0.007774] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x83fffffff]
[    0.007788] NUMA: Node 0 [mem 0x00000000-0xbfffffff] + [mem 0x100000000-0x8299fffff] -> [mem 0x00000000-0x8299fffff]
[    0.007798] NODE_DATA(0) allocated [mem 0x8299d4000-0x8299fefff]
[    0.069690] Zone ranges:
[    0.069694]   DMA      [mem 0x0000000000001000-0x0000000000ffffff]
[    0.069697]   DMA32    [mem 0x0000000001000000-0x00000000ffffffff]
[    0.069699]   Normal   [mem 0x0000000100000000-0x00000008299fffff]
[    0.069701]   Device   empty
[    0.069702] Movable zone start for each node
[    0.069705] Early memory node ranges
[    0.069705]   node   0: [mem 0x0000000000001000-0x000000000009efff]
[    0.069707]   node   0: [mem 0x0000000000100000-0x00000000bffe8fff]
[    0.069709]   node   0: [mem 0x0000000100000000-0x00000008299fffff]
[    0.069714] Initmem setup node 0 [mem 0x0000000000001000-0x00000008299fffff]
[    0.069727] On node 0, zone DMA: 1 pages in unavailable ranges
[    0.069759] On node 0, zone DMA: 97 pages in unavailable ranges
[    0.132074] On node 0, zone Normal: 23 pages in unavailable ranges
[    0.132353] On node 0, zone Normal: 26112 pages in unavailable ranges
[    0.132795] ACPI: PM-Timer IO Port: 0xb008
[    0.132808] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1])
[    0.132846] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23
[    0.132850] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level)
[    0.132852] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level)
[    0.132853] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level)
[    0.132855] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level)
[    0.132861] Using ACPI (MADT) for SMP configuration information
[    0.132862] ACPI: HPET id: 0x8086a201 base: 0xfed00000
[    0.132867] TSC deadline timer available
[    0.132868] smpboot: Allowing 8 CPUs, 0 hotplug CPUs
[    0.132900] PM: hibernation: Registered nosave memory: [mem 0x00000000-0x00000fff]
[    0.132904] PM: hibernation: Registered nosave memory: [mem 0x0009f000-0x0009ffff]
[    0.132906] PM: hibernation: Registered nosave memory: [mem 0x000a0000-0x000effff]
[    0.132909] PM: hibernation: Registered nosave memory: [mem 0x000f0000-0x000fffff]
[    0.132912] PM: hibernation: Registered nosave memory: [mem 0xbffe9000-0xbfffffff]
[    0.132914] PM: hibernation: Registered nosave memory: [mem 0xc0000000-0xdfffffff]
[    0.132917] PM: hibernation: Registered nosave memory: [mem 0xe0000000-0xe03fffff]
[    0.132919] PM: hibernation: Registered nosave memory: [mem 0xe0400000-0xfffbffff]
[    0.132921] PM: hibernation: Registered nosave memory: [mem 0xfffc0000-0xffffffff]
[    0.132924] [mem 0xc0000000-0xdfffffff] available for PCI devices
[    0.132926] Booting paravirtualized kernel on KVM
[    0.132929] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns
[    0.138044] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:8 nr_cpu_ids:8 nr_node_ids:1
[    0.138427] percpu: Embedded 55 pages/cpu s188416 r8192 d28672 u262144
[    0.138960] kvm-guest: stealtime: cpu 0, msr 809a2d080
[    0.138964] kvm-guest: PV spinlocks enabled
[    0.138967] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear)
[    0.138977] Built 1 zonelists, mobility grouping on.  Total pages: 8167041
[    0.138979] Policy zone: Normal
[    0.138980] Kernel command line: BOOT_IMAGE=(hd0,gpt3)/ostree/fedora-coreos-f8c4206cdcb01c91cb5d193dff2ac8df01b5399e73ba7230990e014c549d0ff3/vmlinuz-5.13.4-200.fc34.x86_64 console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.0/fedora-coreos/f8c4206cdcb01c91cb5d193dff2ac8df01b5399e73ba7230990e014c549d0ff3/0 ignition.platform.id=aws root=UUID=03a94e54-4675-4e26-86cb-1852306d4ce6 rw rootflags=prjquota systemd.unified_cgroup_hierarchy=0
[    0.143640] Dentry cache hash table entries: 4194304 (order: 13, 33554432 bytes, linear)
[    0.145922] Inode-cache hash table entries: 2097152 (order: 12, 16777216 bytes, linear)
[    0.145999] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.227258] Memory: 32419772K/33187356K available (16393K kernel code, 3513K rwdata, 9988K rodata, 2792K init, 5048K bss, 767324K reserved, 0K cma-reserved)
[    0.227266] random: get_random_u64 called from __kmem_cache_create+0x2a/0x570 with crng_init=0
[    0.227362] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.227373] Kernel/User page tables isolation: enabled
[    0.227398] ftrace: allocating 46613 entries in 183 pages
[    0.243466] ftrace: allocated 183 pages with 6 groups
[    0.243588] rcu: Hierarchical RCU implementation.
[    0.243590] rcu: 	RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=8.
[    0.243592] 	Trampoline variant of Tasks RCU enabled.
[    0.243593] 	Rude variant of Tasks RCU enabled.
[    0.243593] 	Tracing variant of Tasks RCU enabled.
[    0.243595] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies.
[    0.243596] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8
[    0.247271] NR_IRQS: 524544, nr_irqs: 488, preallocated irqs: 16
[    0.247474] random: crng done (trusting CPU's manufacturer)
[    0.364234] Console: colour VGA+ 80x25
[    0.827575] printk: console [tty0] enabled
[    1.022748] printk: console [ttyS0] enabled
[    1.026023] ACPI: Core revision 20210331
[    1.030440] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 30580167144 ns
[    1.037299] APIC: Switch to symmetric I/O mode setup
[    1.041156] x2apic enabled
[    1.044190] Switched APIC routing to physical x2apic.
[    1.048692] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x240937b9988, max_idle_ns: 440795218083 ns
[    1.056115] Calibrating delay loop (skipped) preset value.. 4999.99 BogoMIPS (lpj=2499998)
[    1.057111] pid_max: default: 32768 minimum: 301
[    1.057111] LSM: Security Framework initializing
[    1.057111] Yama: becoming mindful.
[    1.057111] SELinux:  Initializing.
[    1.057111] LSM support for eBPF active
[    1.057111] landlock: Up and running.
[    1.057111] Mount-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    1.057111] Mountpoint-cache hash table entries: 65536 (order: 7, 524288 bytes, linear)
[    1.057111] Last level iTLB entries: 4KB 64, 2MB 8, 4MB 8
[    1.057111] Last level dTLB entries: 4KB 64, 2MB 0, 4MB 0, 1GB 4
[    1.057111] Spectre V1 : Mitigation: usercopy/swapgs barriers and __user pointer sanitization
[    1.057111] Spectre V2 : Mitigation: Full generic retpoline
[    1.057111] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
[    1.057111] Speculative Store Bypass: Vulnerable
[    1.057111] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
[    1.057111] Freeing SMP alternatives memory: 44K
[    1.057111] smpboot: CPU0: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz (family: 0x6, model: 0x55, stepping: 0x7)
[    1.057444] Performance Events: unsupported p6 CPU model 85 no PMU driver, software events only.
[    1.058208] rcu: Hierarchical SRCU implementation.
[    1.059489] NMI watchdog: Perf NMI watchdog permanently disabled
[    1.060234] smp: Bringing up secondary CPUs ...
[    1.061241] x86: Booting SMP configuration:
[    1.062116] .... node  #0, CPUs:      #1
[    0.809673] kvm-clock: cpu 1, msr 60aa01041, secondary cpu clock
[    1.063669] kvm-guest: stealtime: cpu 1, msr 809a6d080
[    1.072227]  #2
[    0.809673] kvm-clock: cpu 2, msr 60aa01081, secondary cpu clock
[    1.073047] kvm-guest: stealtime: cpu 2, msr 809aad080
[    1.090270]  #3
[    0.809673] kvm-clock: cpu 3, msr 60aa010c1, secondary cpu clock
[    1.091763] kvm-guest: stealtime: cpu 3, msr 809aed080
[    1.102262]  #4
[    0.809673] kvm-clock: cpu 4, msr 60aa01101, secondary cpu clock
[    1.103634] kvm-guest: stealtime: cpu 4, msr 809b2d080
[    1.111281] MDS CPU bug present and SMT on, data leak possible. See https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/mds.html for more details.
[    1.112253]  #5
[    0.809673] kvm-clock: cpu 5, msr 60aa01141, secondary cpu clock
[    1.113027] kvm-guest: stealtime: cpu 5, msr 809b6d080
[    1.121222]  #6
[    0.809673] kvm-clock: cpu 6, msr 60aa01181, secondary cpu clock
[    1.122752] kvm-guest: stealtime: cpu 6, msr 809bad080
[    1.134238]  #7
[    0.809673] kvm-clock: cpu 7, msr 60aa011c1, secondary cpu clock
[    1.136111] kvm-guest: stealtime: cpu 7, msr 809bed080
[    1.155137] smp: Brought up 1 node, 8 CPUs
[    1.156116] smpboot: Max logical packages: 1
[    1.157115] smpboot: Total of 8 processors activated (39999.96 BogoMIPS)
[    1.160465] devtmpfs: initialized
[    1.161164] x86/mm: Memory block size: 128MB
[    1.164201] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns
[    1.165120] futex hash table entries: 2048 (order: 5, 131072 bytes, linear)
[    1.166214] pinctrl core: initialized pinctrl subsystem
[    1.167392] PM: RTC time: 08:08:29, date: 2021-09-18
[    1.168322] NET: Registered protocol family 16
[    1.169230] DMA: preallocated 4096 KiB GFP_KERNEL pool for atomic allocations
[    1.170123] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations
[    1.171124] DMA: preallocated 4096 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations
[    1.172136] audit: initializing netlink subsys (disabled)
[    1.173145] audit: type=2000 audit(1631952508.892:1): state=initialized audit_enabled=0 res=1
[    1.173243] thermal_sys: Registered thermal governor 'fair_share'
[    1.174117] thermal_sys: Registered thermal governor 'bang_bang'
[    1.175114] thermal_sys: Registered thermal governor 'step_wise'
[    1.176115] thermal_sys: Registered thermal governor 'user_space'
[    1.177132] cpuidle: using governor menu
[    1.179558] ACPI: bus type PCI registered
[    1.180116] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5
[    1.181241] PCI: Using configuration type 1 for base access
[    1.183856] Kprobes globally optimized
[    1.184151] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages
[    1.185116] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages
[    1.344128] cryptd: max_cpu_qlen set to 1000
[    1.350189] alg: No test for 842 (842-generic)
[    1.354153] alg: No test for 842 (842-scomp)
[    1.360135] wait_for_initramfs() called before rootfs_initcalls
[    1.377539] DRBG: Continuing without Jitter RNG
[    1.383531] raid6: skip pq benchmark and using algorithm avx512x4
[    1.387116] raid6: using avx512x2 recovery algorithm
[    1.391177] ACPI: Added _OSI(Module Device)
[    1.394115] ACPI: Added _OSI(Processor Device)
[    1.397115] ACPI: Added _OSI(3.0 _SCP Extensions)
[    1.401115] ACPI: Added _OSI(Processor Aggregator Device)
[    1.405115] ACPI: Added _OSI(Linux-Dell-Video)
[    1.408114] ACPI: Added _OSI(Linux-Lenovo-NV-HDMI-Audio)
[    1.412115] ACPI: Added _OSI(Linux-HPI-Hybrid-Graphics)
[    1.415978] ACPI: 2 ACPI AML tables successfully acquired and loaded
[    1.421036] ACPI: Interpreter enabled
[    1.424124] ACPI: (supports S0 S4 S5)
[    1.427114] ACPI: Using IOAPIC for interrupt routing
[    1.430130] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug
[    1.437417] ACPI: Enabled 16 GPEs in block 00 to 0F
[    1.443212] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff])
[    1.447121] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI EDR HPX-Type3]
[    1.453122] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
[    1.460419] acpiphp: Slot [3] registered
[    1.464139] acpiphp: Slot [4] registered
[    1.467130] acpiphp: Slot [5] registered
[    1.470130] acpiphp: Slot [6] registered
[    1.473129] acpiphp: Slot [7] registered
[    1.476128] acpiphp: Slot [8] registered
[    1.480131] acpiphp: Slot [9] registered
[    1.483130] acpiphp: Slot [10] registered
[    1.486128] acpiphp: Slot [11] registered
[    1.489130] acpiphp: Slot [12] registered
[    1.492137] acpiphp: Slot [13] registered
[    1.495129] acpiphp: Slot [14] registered
[    1.498130] acpiphp: Slot [15] registered
[    1.502129] acpiphp: Slot [16] registered
[    1.505130] acpiphp: Slot [17] registered
[    1.508128] acpiphp: Slot [18] registered
[    1.511130] acpiphp: Slot [19] registered
[    1.514128] acpiphp: Slot [20] registered
[    1.517128] acpiphp: Slot [21] registered
[    1.521128] acpiphp: Slot [22] registered
[    1.524130] acpiphp: Slot [23] registered
[    1.527128] acpiphp: Slot [24] registered
[    1.530129] acpiphp: Slot [25] registered
[    1.533129] acpiphp: Slot [26] registered
[    1.536129] acpiphp: Slot [27] registered
[    1.540129] acpiphp: Slot [28] registered
[    1.543128] acpiphp: Slot [29] registered
[    1.546131] acpiphp: Slot [30] registered
[    1.549129] acpiphp: Slot [31] registered
[    1.552126] PCI host bridge to bus 0000:00
[    1.555115] pci_bus 0000:00: root bus resource [io  0x0000-0x0cf7 window]
[    1.559114] pci_bus 0000:00: root bus resource [io  0x0d00-0xffff window]
[    1.564115] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window]
[    1.570114] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window]
[    1.575116] pci_bus 0000:00: root bus resource [bus 00-ff]
[    1.579152] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000
[    1.583550] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100
[    1.589111] pci 0000:00:01.3: [8086:7113] type 00 class 0x000000
[    1.593918] pci 0000:00:01.3: quirk: [io  0xb000-0xb03f] claimed by PIIX4 ACPI
[    1.599130] pci 0000:00:01.3: quirk: [io  0xb100-0xb10f] claimed by PIIX4 SMB
[    1.603177] pci 0000:00:01.3: PIIX4 devres E PIO at fff0-ffff
[    1.607130] pci 0000:00:01.3: PIIX4 devres F MMIO at ffc00000-ffffffff
[    1.611129] pci 0000:00:01.3: PIIX4 devres G PIO at fff0-ffff
[    1.615129] pci 0000:00:01.3: PIIX4 devres H MMIO at ffc00000-ffffffff
[    1.619130] pci 0000:00:01.3: PIIX4 devres I PIO at fff0-ffff
[    1.623130] pci 0000:00:01.3: PIIX4 devres J PIO at fff0-ffff
[    1.627114] pci 0000:00:01.3: quirk_piix4_acpi+0x0/0x140 took 33203 usecs
[    1.631423] pci 0000:00:03.0: [1d0f:1111] type 00 class 0x030000
[    1.635476] pci 0000:00:03.0: reg 0x10: [mem 0xfe400000-0xfe7fffff pref]
[    1.641053] pci 0000:00:03.0: reg 0x30: [mem 0xfebe0000-0xfebeffff pref]
[    1.645432] pci 0000:00:04.0: [1d0f:8061] type 00 class 0x010802
[    1.650598] pci 0000:00:04.0: reg 0x10: [mem 0xfebf0000-0xfebf3fff]
[    1.659705] pci 0000:00:05.0: [1d0f:ec20] type 00 class 0x020000
[    1.663815] pci 0000:00:05.0: reg 0x10: [mem 0xfebf4000-0xfebf7fff]
[    1.672065] ACPI: PCI: Interrupt link LNKA configured for IRQ 10
[    1.676212] ACPI: PCI: Interrupt link LNKB configured for IRQ 10
[    1.680212] ACPI: PCI: Interrupt link LNKC configured for IRQ 11
[    1.684205] ACPI: PCI: Interrupt link LNKD configured for IRQ 11
[    1.688161] ACPI: PCI: Interrupt link LNKS configured for IRQ 9
[    1.692553] iommu: Default domain type: Translated 
[    1.696153] pci 0000:00:03.0: vgaarb: setting as boot VGA device
[    1.697111] pci 0000:00:03.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[    1.706117] pci 0000:00:03.0: vgaarb: bridge control possible
[    1.710114] vgaarb: loaded
[    1.713241] SCSI subsystem initialized
[    1.716155] ACPI: bus type USB registered
[    1.720133] usbcore: registered new interface driver usbfs
[    1.723123] usbcore: registered new interface driver hub
[    1.727122] usbcore: registered new device driver usb
[    1.731141] pps_core: LinuxPPS API ver. 1 registered
[    1.734114] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti <giometti@linux.it>
[    1.741117] PTP clock support registered
[    1.744138] EDAC MC: Ver: 3.0.0
[    1.747312] NetLabel: Initializing
[    1.750115] NetLabel:  domain hash size = 128
[    1.753113] NetLabel:  protocols = UNLABELED CIPSOv4 CALIPSO
[    1.757133] NetLabel:  unlabeled traffic allowed by default
[    1.761117] PCI: Using ACPI for IRQ routing
[    1.765507] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0, 0, 0, 0, 0, 0
[    1.769116] hpet0: 8 comparators, 32-bit 62.500000 MHz counter
[    1.776213] clocksource: Switched to clocksource kvm-clock
[    1.792320] VFS: Disk quotas dquot_6.6.0
[    1.891830] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    1.896120] pnp: PnP ACPI init
[    1.899335] pnp: PnP ACPI: found 5 devices
[    1.910779] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns
[    1.917344] NET: Registered protocol family 2
[    1.921059] IP idents hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    1.928823] tcp_listen_portaddr_hash hash table entries: 16384 (order: 6, 262144 bytes, linear)
[    1.935322] TCP established hash table entries: 262144 (order: 9, 2097152 bytes, linear)
[    1.941743] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes, linear)
[    1.947789] TCP: Hash tables configured (established 262144 bind 65536)
[    1.952215] MPTCP token hash table entries: 32768 (order: 7, 786432 bytes, linear)
[    1.958324] UDP hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    1.962621] UDP-Lite hash table entries: 16384 (order: 7, 524288 bytes, linear)
[    1.968649] NET: Registered protocol family 1
[    1.971934] NET: Registered protocol family 44
[    1.975283] pci_bus 0000:00: resource 4 [io  0x0000-0x0cf7 window]
[    1.979251] pci_bus 0000:00: resource 5 [io  0x0d00-0xffff window]
[    1.983226] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window]
[    1.987440] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window]
[    1.991744] pci 0000:00:00.0: Limiting direct PCI/PCI transfers
[    1.995626] pci 0000:00:01.0: Activating ISA DMA hang workarounds
[    1.999686] pci 0000:00:03.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[    2.006050] PCI: CLS 0 bytes, default 64
[    2.009204] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[    2.009271] Trying to unpack rootfs image as initramfs...
[    2.013262] software IO TLB: mapped [mem 0x00000000bbfe9000-0x00000000bffe9000] (64MB)
[    2.013326] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x240937b9988, max_idle_ns: 440795218083 ns
[    2.031252] clocksource: Switched to clocksource tsc
[    2.047237] Initialise system trusted keyrings
[    2.050761] Key type blacklist registered
[    2.054363] workingset: timestamp_bits=36 max_order=23 bucket_order=0
[    2.060181] zbud: loaded
[    2.063900] integrity: Platform Keyring initialized
[    2.075036] NET: Registered protocol family 38
[    2.078484] xor: automatically using best checksumming function   avx       
[    2.082934] Key type asymmetric registered
[    2.086211] Asymmetric key parser 'x509' registered
[    2.089799] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 245)
[    2.095986] io scheduler mq-deadline registered
[    2.099457] io scheduler kyber registered
[    2.102717] io scheduler bfq registered
[    2.106218] atomic64_test: passed for x86-64 platform with CX8 and with SSE
[    2.110779] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4
[    2.115145] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0
[    2.121286] ACPI: button: Power Button [PWRF]
[    2.124789] input: Sleep Button as /devices/LNXSYSTM:00/LNXSLPBN:00/input/input1
[    2.130966] ACPI: button: Sleep Button [SLPF]
[    2.135708] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[    2.140042] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
[    2.148223] Non-volatile memory driver v1.3
[    2.151496] Linux agpgart interface v0.103
[    2.155728] libphy: Fixed MDIO Bus: probed
[    2.159055] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    2.163195] ehci-pci: EHCI PCI platform driver
[    2.166533] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    2.170510] ohci-pci: OHCI PCI platform driver
[    2.173844] uhci_hcd: USB Universal Host Controller Interface driver
[    2.177933] usbcore: registered new interface driver usbserial_generic
[    2.182021] usbserial: USB Serial support registered for generic
[    2.185941] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12
[    2.192306] i8042: Warning: Keylock active
[    2.196323] serio: i8042 KBD port at 0x60,0x64 irq 1
[    2.199837] serio: i8042 AUX port at 0x60,0x64 irq 12
[    2.203474] mousedev: PS/2 mouse device common for all mice
[    2.207554] rtc_cmos 00:00: RTC can wake from S4
[    2.211836] rtc_cmos 00:00: registered as rtc0
[    2.215376] rtc_cmos 00:00: setting system clock to 2021-09-18T08:08:30 UTC (1631952510)
[    2.221586] rtc_cmos 00:00: alarms up to one day, 114 bytes nvram
[    2.225642] device-mapper: uevent: version 1.0.3
[    2.229237] device-mapper: ioctl: 4.45.0-ioctl (2021-03-22) initialised: dm-devel@redhat.com
[    2.235734] intel_pstate: CPU model not supported
[    2.240032] hid: raw HID events driver (C) Jiri Kosina
[    2.245825] usbcore: registered new interface driver usbhid
[    2.257041] usbhid: USB HID core driver
[    2.265828] drop_monitor: Initializing network drop monitor service
[    2.275380] Initializing XFRM netlink socket
[    2.282653] NET: Registered protocol family 10
[    2.712725] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input2
[    3.042213] Freeing initrd memory: 75700K
[    3.052523] Segment Routing with IPv6
[    3.055767] RPL Segment Routing with IPv6
[    3.060993] mip6: Mobile IPv6
[    3.064407] NET: Registered protocol family 17
[    3.070966] No MBM correction factor available
[    3.075234] IPI shorthand broadcast: enabled
[    3.078848] AVX2 version of gcm_enc/dec engaged.
[    3.084128] AES CTR mode by8 optimization enabled
[    3.118434] sched_clock: Marking stable (2309746137, 808673116)->(3378880864, -260461611)
[    3.127331] registered taskstats version 1
[    3.131651] Loading compiled-in X.509 certificates
[    3.136897] Loaded X.509 cert 'Fedora kernel signing key: 827da2060c3b22c48b38f76f7d3f7bb1e8ca3a27'
[    3.144876] zswap: loaded using pool lzo/zbud
[    3.148782] page_owner is disabled
[    3.152244] Key type ._fscrypt registered
[    3.155746] Key type .fscrypt registered
[    3.159119] Key type fscrypt-provisioning registered
[    3.163300] Btrfs loaded, crc32c=crc32c-generic, zoned=yes
[    3.170965] Key type encrypted registered
[    3.175230] ima: No TPM chip found, activating TPM-bypass!
[    3.180528] Loading compiled-in module X.509 certificates
[    3.184899] Loaded X.509 cert 'Fedora kernel signing key: 827da2060c3b22c48b38f76f7d3f7bb1e8ca3a27'
[    3.191943] ima: Allocated hash algorithm: sha256
[    3.195731] ima: No architecture policies found
[    3.199207] evm: Initialising EVM extended attributes:
[    3.203039] evm: security.selinux
[    3.206026] evm: security.ima
[    3.209103] evm: security.capability
[    3.212340] evm: HMAC attrs: 0x1
[    3.216589] PM:   Magic number: 5:283:119
[    3.220528] RAS: Correctable Errors collector initialized.
[    3.226724] Freeing unused decrypted memory: 2036K
[    3.232971] Freeing unused kernel image (initmem) memory: 2792K
[    3.243142] Write protecting the kernel read-only data: 28672k
[    3.248087] Freeing unused kernel image (text/rodata gap) memory: 2036K
[    3.252511] Freeing unused kernel image (rodata/data gap) memory: 252K
[    3.347187] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    3.351857] rodata_test: all tests were successful
[    3.356024] x86/mm: Checking user space page tables
[    3.382237] input: ImPS/2 Generic Wheel Mouse as /devices/platform/i8042/serio1/input/input4
[    3.451735] x86/mm: Checked W+X mappings: passed, no W+X pages found.
[    3.456372] Run /init as init process
[    3.476387] systemd[1]: systemd v248.6-1.fc34 running in system mode. (+PAM +AUDIT +SELINUX -APPARMOR +IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 +PWQUALITY +P11KIT +QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP +SYSVINIT default-hierarchy=unified)
[    3.503374] systemd[1]: Detected virtualization kvm.
[    3.507814] systemd[1]: Detected architecture x86-64.
[    3.512139] systemd[1]: Running in initial RAM disk.

Welcome to �[0;38;2;60;110;180mFedora CoreOS 34 dracut-055-3.fc34 (Initramfs)�[0m!

[    3.528154] systemd[1]: No hostname configured, using default hostname.
[    3.533485] systemd[1]: Hostname set to <localhost>.
[    3.541732] systemd[1]: Initializing machine ID from KVM UUID.
[    3.661293] systemd[1]: Queued start job for default target Initrd Default Target.
[    3.668805] systemd[1]: Started Forward Password Requests to Clevis Directory Watch.
[�[0;32m  OK  �[0m] Started �[0;1;39mForward Password R…sts to Clevis Directory Watch�[0m.
[    3.677913] systemd[1]: Started Dispatch Password Requests to Console Directory Watch.
[�[0;32m  OK  �[0m] Started �[0;1;39mDispatch Password …ts to Console Directory Watch�[0m.
[    3.687381] systemd[1]: Reached target Local Encrypted Volumes (Pre).
[�[0;32m  OK  �[0m] Reached target �[0;1;39mLocal Encrypted Volumes (Pre)�[0m.
[    3.694387] systemd[1]: Reached target Local Encrypted Volumes.
[�[0;32m  OK  �[0m] Reached target �[0;1;39mLocal Encrypted Volumes�[0m.
[    3.699986] systemd[1]: Reached target Ignition Subsequent Boot Disk Setup.
[�[0;32m  OK  �[0m] Reached target �[0;1;39mIgnition Subsequent Boot Disk Setup�[0m.
[    3.707613] systemd[1]: Reached target Path Units.
[�[0;32m  OK  �[0m] Reached target �[0;1;39mPath Units�[0m.
[    3.712881] systemd[1]: Reached target Slice Units.
[�[0;32m  OK  �[0m] Reached target �[0;1;39mSlice Units�[0m.
[    3.718388] systemd[1]: Reached target Swaps.
[�[0;32m  OK  �[0m] Reached target �[0;1;39mSwaps�[0m.
[    3.723194] systemd[1]: Reached target Timer Units.
[�[0;32m  OK  �[0m] Reached target �[0;1;39mTimer Units�[0m.
[    3.728374] systemd[1]: Listening on D-Bus System Message Bus Socket.
[�[0;32m  OK  �[0m] Listening on �[0;1;39mD-Bus System Message Bus Socket�[0m.
[    3.734249] systemd[1]: Listening on Journal Audit Socket.
[�[0;32m  OK  �[0m] Listening on �[0;1;39mJournal Audit Socket�[0m.
[    3.739699] systemd[1]: Listening on Journal Socket (/dev/log).
[�[0;32m  OK  �[0m] Listening on �[0;1;39mJournal Socket (/dev/log)�[0m.
[    3.751498] systemd[1]: Listening on Journal Socket.
[�[0;32m  OK  �[0m] Listening on �[0;1;39mJournal Socket�[0m.
[    3.757529] systemd[1]: Listening on udev Control Socket.
[�[0;32m  OK  �[0m] Listening on �[0;1;39mudev Control Socket�[0m.
[    3.763050] systemd[1]: Listening on udev Kernel Socket.
[�[0;32m  OK  �[0m] Listening on �[0;1;39mudev Kernel Socket�[0m.
[    3.768872] systemd[1]: Reached target Socket Units.
[�[0;32m  OK  �[0m] Reached target �[0;1;39mSocket Units�[0m.
[    3.777829] systemd[1]: Condition check resulted in Check that initrd matches kernel being skipped.
[    3.787292] systemd[1]: Condition check resulted in Persist osmet files (ISO) being skipped.
[    3.816974] systemd[1]: Starting CoreOS: Touch /run/agetty.reload...
         Starting �[0;1;39mCoreOS: Touch /run/agetty.reload�[0m...
[    3.825986] systemd[1]: Starting Create List of Static Device Nodes...
         Starting �[0;1;39mCreate List of Static Device Nodes�[0m...
[    3.974301] systemd[1]: Starting Journal Service...
         Starting �[0;1;39mJournal Service�[0m...
[    3.980085] systemd[1]: Starting Load Kernel Modules...
         Starting �[0;1;39mLoad Kernel Modules�[0m...
[    3.986297] systemd[1]: Condition check resulted in Create System Users being skipped.
[    3.994545] systemd[1]: Starting Setup Virtual Console...
         Starting �[0;1;39mSetup Virtual Console�[0m...
[    4.000722] systemd[1]: Finished CoreOS: Touch /run/agetty.reload.
[�[0;32m  OK  �[0m] Finished �[0;1;39mCoreOS: Touch /run/agetty.reload�[0m.
[    4.006916] audit: type=1130 audit(1631952512.290:2): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=coreos-touch-run-agetty comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.007430] systemd[1]: Finished Create List of Static Device Nodes.
[    4.037971] fuse: init (API version 7.34)
[    4.038492] systemd-modules-load[306]: Inserted module 'fuse'
[    4.038542] systemd-modules-load[306]: Module 'msr' is built in
[�[0;32m  OK  �[0m] Finished �[0;1;39mCreate List of Static Device Nodes�[0m.
[    4.199521] audit: type=1130 audit(1631952512.483:3): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=kmod-static-nodes comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.199761] systemd[1]: Started Journal Service.
[�[0;32m  OK  �[0m] Started �[0;1;39mJournal Service�[0m.
[    4.217481] audit: type=1130 audit(1631952512.501:4): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-journald comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.218709] systemd[1]: Finished Load Kernel Modules.
[�[0;32m  OK  �[0m] Finished �[0;1;39mLoad Kernel Modules�[0m.
[    4.247846] audit: type=1130 audit(1631952512.531:5): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-modules-load comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.248408] systemd[1]: Finished Setup Virtual Console.
[�[0;32m  OK  �[0m] Finished �[0;1;39mSetup Virtual Console�[0m.
[    4.277668] audit: type=1130 audit(1631952512.561:6): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.278884] systemd[1]: Starting dracut ask for additional cmdline parameters...
         Starting �[0;1;39mdracut ask for additional cmdline parameters�[0m...
         Starting �[0;1;39mApply[    4.307199] systemd[1]: Starting Apply Kernel Variables...
 Kernel Variables�[0m...
[    4.315100] systemd[1]: Starting Create Static Device Nodes in /dev...
         Starting �[0;1;39mCreate Static Device Nodes in /dev�[0m...
         Startin[    4.321135] systemd-sysctl[314]: Couldn't write '1' to 'fs/may_detach_mounts', ignoring: No such file or directory
g �[0;1;39mCreate Volatile Files and Directories�[0m...
[    4.330687] systemd[1]: Starting Create Volatile Files and Directories...
[    4.330811] systemd[1]: Finished Apply Kernel Variables.
[�[0;32m  OK  �[0m] Finished �[0;1;39mApply Kernel Variables�[0m.
[    4.355412] audit: type=1130 audit(1631952512.639:7): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-sysctl comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.358920] systemd[1]: Finished dracut ask for additional cmdline parameters.
[�[0;32m  OK  �[0m] Finished �[0;1;39mdracut ask for additional cmdline parameters�[0m.
[    4.377810] audit: type=1130 audit(1631952512.661:8): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dracut-cmdline-ask comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.378568] systemd[1]: Finished Create Static Device Nodes in /dev.
[�[0;32m  OK  �[0m] Finished �[0;1;39mCreate Static Device Nodes in /dev�[0m.
[    4.406325] audit: type=1130 audit(1631952512.690:9): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-tmpfiles-setup-dev comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.410315] systemd[1]: Finished Create Volatile Files and Directories.
[�[0;32m  OK  �[0m] Finished �[0;1;39mCreate Volatile Files and Directories�[0m.
[    4.433904] audit: type=1130 audit(1631952512.717:10): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=systemd-tmpfiles-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    4.435744] systemd[1]: Starting dracut cmdline hook...
         Starting �[0;1;39mdracut cmdline hook�[0m...
[    4.472856] dracut-cmdline[331]: dracut-34 (CoreOS) dracut-055-3.fc34
[    4.484556] dracut-cmdline[331]: Using kernel command line parameters:  rd.driver.pre=btrfs rd.driver.pre=scsi_dh_alua rd.driver.pre=scsi_dh_emc rd.driver.pre=scsi_dh_rdac rd.driver.pre=dm_multipath   BOOT_IMAGE=(hd0,gpt3)/ostree/fedora-coreos-f8c4206cdcb01c91cb5d193dff2ac8df01b5399e73ba7230990e014c549d0ff3/vmlinuz-5.13.4-200.fc34.x86_64 console=tty0 console=ttyS0,115200n8 ostree=/ostree/boot.0/fedora-coreos/f8c4206cdcb01c91cb5d193dff2ac8df01b5399e73ba7230990e014c549d0ff3/0 ignition.platform.id=aws root=UUID=03a94e54-4675-4e26-86cb-1852306d4ce6 rw rootflags=prjquota systemd.unified_cgroup_hierarchy=0
[    4.583974] systemd[1]: Finished dracut cmdline hook.
[�[0;32m  OK  �[0m] Finished �[0;1;39mdracut cmdline hook�[0m.
[    4.590583] systemd[1]: Starting dracut pre-udev hook...
         Starting �[0;1;39mdracut pre-udev hook�[0m...
[    4.632584] alua: device handler registered
[    4.647425] emc: device handler registered
[    4.661609] rdac: device handler registered
[    4.689779] systemd[1]: Finished dracut pre-udev hook.
[�[0;32m  OK  �[0m] Finished �[0;1;39mdracut pre-udev hook�[0m.
[    4.697039] systemd[1]: Starting Rule-based Manager for Device Events and Files...
         Starting �[0;1;39mRule-based Manage…for Device Events and Files�[0m...
[�[0;32m  OK  �[0m] Started �[0;1;39mRule-based Manager for Device Events and Files�[0m.
[    4.724020] systemd[1]: Started Rule-based Manager for Device Events and Files.
[    4.731626] systemd[1]: Starting dracut pre-trigger hook...
         Starting �[0;1;39mdracut pre-trigger hook�[0m...
[    4.762119] dracut-pre-trigger[467]: rd.md=0: removing MD RAID activation
[    4.803815] systemd[1]: Finished dracut pre-trigger hook.
[�[0;32m  OK  �[0m] Finished �[0;1;39mdracut pre-trigger hook�[0m.
[    4.810935] systemd[1]: Starting Coldplug All udev Devices...
         Starting �[0;1;39mColdplug All udev Devices�[0m...
[    4.954653] systemd[1]: Finished Coldplug All udev Devices.
[�[0;32m  OK  �[0m] Finished �[0;1;39mColdplug All udev Devices�[0m.
[    4.972954] ena 0000:00:05.0: ENA device version: 0.10
[    4.981749] ena 0000:00:05.0: ENA controller version: 0.0.1 implementation version 1
[    5.010561] systemd-udevd[531]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[    5.011854] ena 0000:00:05.0: LLQ is not supported Fallback to host mode policy.
[    5.019967] systemd[1]: Starting Wait for udev To Complete Device Initialization...
         Starting �[0;1;39mWait for udev To …plete Device Initialization�[0m...
[    5.040360] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem febf4000, mac addr 02:e9:56:22:43:78
[    5.049692] udevadm[552]: systemd-udev-settle.service is deprecated. Please fix multipathd.service, multipathd-configure.service not to pull it in.
[    5.049937] systemd-udevd[512]: Using default interface naming scheme 'v247'.
[    5.050349] systemd-udevd[512]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[    5.050561] ena 0000:00:05.0 ens5: renamed from eth0
[    5.050704] nvme nvme0: pci function 0000:00:04.0
[    5.051196] ACPI: \_SB_.LNKD: Enabled at IRQ 11
[    5.059194] nvme nvme0: 2/0/0 default/read/poll queues
[    5.062478]  nvme0n1: p1 p2 p3 p4
[    5.076354] systemd-udevd[534]: Using default interface naming scheme 'v247'.
[    5.077617] systemd-udevd[534]: ethtool: autonegotiation is unset or enabled, the speed and duplex are not writable.
[�[0;32m  OK  �[[    5.285899] systemd[1]: Found device Amazon Elastic Block Store root.
0m] Found device �[0;1;39mAmazon Elastic Block Store root�[0m.
[�[0;32m  OK  �[[    5.291311] systemd[1]: Found device Amazon Elastic Block Store root.
0m] Found device �[0;1;39mAmazon Elastic Block Store root�[0m.
[    5.296105] systemd[1]: Finished Wait for udev To Complete Device Initialization.
[�[0;32m  OK  �[0m] Finished �[0;1;39mWait for udev To Complete Device Initialization�[0m.
[�[0;32m  OK  �[[    5.302888] systemd[1]: Reached target Initrd Root Device.
0m] Reached target �[0;1;39mInitrd Root Device�[0m.
[    5.307523] systemd[1]: Condition check resulted in CoreOS: Mount (subsequent) /sysroot being skipped.
         Starting �[0;1;39mDevice-Mapper Multipath Device Controller�[0m...
[�[0;32m  OK  �[[    5.315154] systemd[1]: Condition check resulted in Device-Mapper Multipath Default Configuration being skipped.
0m] Started �[0;1;39mDevice-Mapper Multipath Device Controller�[0m.
[�[0;32m  OK  �[[    5.325413] systemd[1]: Starting Device-Mapper Multipath Device Controller...
0m] Reached target �[0;1;39mPreparation for Local File Systems�[0m.
[�[0;32m  OK  �[0m] Reached targ[    5.330964] systemd[1]: Started Device-Mapper Multipath Device Controller.
et �[0;1;39mLocal File Systems�[[    5.336781] multipathd[558]: --------start up--------
0m.
[�[0;32m  OK  �[0m] Reached target �[0;1;39mSyst[    5.341300] multipathd[558]: read /etc/multipath.conf
em Initialization�[0m.
[    5.346835] multipathd[558]: /etc/multipath.conf does not exist, blacklisting all devices.
[�[0;32m  OK  �[0m] Reached target �[0;1;39mBasic System�[0m.
[    5.355213] multipathd[558]: You can run "/sbin/mpathconf --enable" to create
[�[0;32m  OK  �[0m] Reached target �[0;1;39mNetwork�[0m.
[    5.367359] multipathd[558]: /etc/multipath.conf. See man mpathconf(8) for more details
         Starting �[0;1;39mdracut initqueue hook�[0m...
[    5.374738] multipathd[558]: /etc/multipath.conf does not exist, blacklisting all devices.
[    5.388411] multipathd[558]: You can run "/sbin/mpathconf --enable" to create
[�[0;32m  OK  �[0m] Finished �[0;1;39mdracut initqueue hook�[0m.
[    5.397916] multipathd[558]: /etc/multipath.conf. See man mpathconf(8) for more details
[�[0;32m  OK  �[0m] Reached target �[0;1;39mPreparation for Remote File Systems�[0m.
[    5.406881] multipathd[558]: path checkers start up
[�[0;32m  OK  �[0m] Reached target �[0;1;39mRemote Encrypted Volumes�[0m.
[�[0;32m  OK  �[[    5.413790] systemd[1]: Reached target Preparation for Local File Systems.
0m] Reached target �[0;1;39mRemote File Systems�[0m.
[    5.419634] multipathd[558]: failed to increase buffer size
         Starting �[0;1;39mdracut pre-mount hook�[0m...
[    5.423626] systemd[1]: Reached target Local File Systems.
[    5.428406] systemd[1]: Reached target System Initialization.
[�[0;32m  OK  �[[    5.432927] systemd[1]: Reached target Basic System.
0m] Finished �[0;1;39mdracut pre-mount hook�[0m.
[    5.437138] systemd[1]: System is tainted: cgroupsv1
         Starting �[0;1;39mFile System Check…4675-4e26-86cb-1852306d4ce6�[0m...
[    5.447870] systemd[1]: Condition check resulted in CoreOS Enable Network being skipped.
[    5.462451] systemd[1]: Condition check resulted in nm-initrd.service being skipped.
[    5.471135] systemd[1]: Reached target Network.
[    5.474967] systemd-fsck[585]: /usr/sbin/fsck.xfs: XFS file system.
[    5.480013] systemd[1]: Condition check resulted in nm-wait-online-initrd.service being skipped.
[    5.480090] systemd[1]: Starting dracut initqueue hook...
[�[0;32m  OK  �[0m] Finished �[0;1;39mFile System Check…4-4675-4e26-86cb-1852306d4ce6�[0m.
[    5.480147] systemd[1]: Finished dracut initqueue hook.
[    5.480175] systemd[1]: Reached target Preparation for Remote File Systems.
[    5.480197] systemd[1]: Reached target Remote Encrypted Volumes.
[    5.480220] systemd[1]: Reached target Remote File Systems.
[    5.509845] systemd[1]: Condition check resulted in Acquire live PXE rootfs image being skipped.
         Mounting �[0;1;39m/sysroot�[0m...
[    5.518354] systemd[1]: Condition check resulted in Persist osmet files (PXE) being skipped.
[    5.536782] systemd[1]: Starting dracut pre-mount hook...
[    5.549052] systemd[1]: Finished dracut pre-mount hook.
[    5.560635] systemd[1]: Starting File System Check on /dev/disk/by-uuid/03a94e54-4675-4e26-86cb-1852306d4ce6...
[    5.581279] systemd[1]: Finished File System Check on /dev/disk/by-uuid/03a94e54-4675-4e26-86cb-1852306d4ce6.
[    5.599365] systemd[1]: Mounting /sysroot...
[    5.742557] SGI XFS with ACLs, security attributes, scrub, quota, no debug enabled
[    5.752644] XFS (nvme0n1p4): Mounting V5 Filesystem
[    5.869591] XFS (nvme0n1p4): Starting recovery (logdev: internal)
[    6.259187] XFS (nvme0n1p4): Ending recovery (logdev: internal)
[    6.547482] xfs filesystem being mounted at /sysroot supports timestamps until 2038 (0x7fffffff)
[�[0;32m  OK  �[[    6.556950] systemd[1]: Mounted /sysroot.
0m] Mounted �[0;1;39m/sysroot�[0m.
[    6.560713] systemd[1]: Starting OSTree Prepare OS/...
[    6.567032] ostree-prepare-root[597]: Resolved OSTree target to: /sysroot/ostree/deploy/fedora-coreos/deploy/74dd6506378b1eced4f1bb1cd7fc2bdc45d200c0c6d5fb0feda31408b079b17b.2
         Starting �[0;1;39mOSTree Prepare OS/�[0[    6.634569] ostree-prepare-root[597]: sysroot configured read-only: 1, currently writable: 1
m...
[    6.720526] systemd[1]: Finished OSTree Prepare OS/.
[�[0;32m  OK  �[0m] Finished �[0;1;39mOSTree Prepare OS/�[0m.
[    6.727814] kauditd_printk_skb: 12 callbacks suppressed
[    6.727817] audit: type=1130 audit(1631952515.011:23): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=ostree-prepare-root comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[    6.728000] systemd[1]: Reached target Initrd Root File System.
[�[0;32m  OK  �[0m] Reached target �[0;1;39mInitrd Root File System�[0m.
[    6.935975] systemd[1]: Condition check resulted in CoreOS Propagate Multipath Configuration being skipped.
[�[0;32m  OK  �[0m] Reached target �[0;1;39mSubsequent (Not Ignition) boot complete�[0m.
[    7.014395] systemd[1]: Reached target Subsequent (Not Ignition) boot complete.
         Starting �[0;1;39mReloa[    7.020486] systemd[1]: Starting Reload Configuration from the Real Root...
d Configuration from the Real Root�[0m...
[    7.122651] audit: type=1334 audit(1631952515.406:24): prog-id=6 op=UNLOAD
[    7.222143] systemd[1]: Reloading.
[    7.226274] audit: type=1334 audit(1631952515.406:25): prog-id=5 op=UNLOAD
[    7.234175] audit: type=1334 audit(1631952515.513:26): prog-id=4 op=UNLOAD
[    7.313996] audit: type=1334 audit(1631952515.597:27): prog-id=3 op=UNLOAD

@yaroslavkasatikov
Copy link

after the reboot the node shows no default route messages on console:

[ 4532.621035] configure-ovs.sh[9846]: + sleep 5
[ 4537.622606] configure-ovs.sh[9846]: + '[' 10 -lt 12 ']'
[ 4537.623843] configure-ovs.sh[9921]: ++ ip route show default
[ 4537.624884] configure-ovs.sh[9922]: ++ awk '{ if ($4 == "dev") { print $5; exit } }'
[ 4537.626633] configure-ovs.sh[9846]: + iface=
[ 4537.627206] configure-ovs.sh[9846]: + [[ -n '' ]]
[ 4537.627387] configure-ovs.sh[9924]: ++ ip -6 route show default
[ 4537.627664] configure-ovs.sh[9925]: ++ awk '{ if ($4 == "dev") { print $5; exit } }'
[ 4537.629646] configure-ovs.sh[9846]: + iface=
[ 4537.630317] configure-ovs.sh[9846]: + [[ -n '' ]]
[ 4537.630355] configure-ovs.sh[9846]: + counter=11
[ 4537.630391] configure-ovs.sh[9846]: + echo 'No default route found on attempt: 11'
[ 4537.630425] configure-ovs.sh[9846]: No default route found on attempt: 11
[ 4537.630579] configure-ovs.sh[9846]: + sleep 5
[ 4542.631141] configure-ovs.sh[9846]: + '[' 11 -lt 12 ']'
[ 4542.632289] configure-ovs.sh[9928]: ++ ip route show default
[ 4542.633615] configure-ovs.sh[9929]: ++ awk '{ if ($4 == "dev") { print $5; exit } }'
[ 4542.636070] configure-ovs.sh[9846]: + iface=
[ 4542.636761] configure-ovs.sh[9846]: + [[ -n '' ]]
[ 4542.636937] configure-ovs.sh[9931]: ++ ip -6 route show default
[ 4542.637137] configure-ovs.sh[9932]: ++ awk '{ if ($4 == "dev") { print $5; exit } }'
[ 4542.639228] configure-ovs.sh[9846]: + iface=
[ 4542.639482] configure-ovs.sh[9846]: + [[ -n '' ]]
[ 4542.639713] configure-ovs.sh[9846]: + counter=12
[ 4542.639762] configure-ovs.sh[9846]: + echo 'No default route found on attempt: 12'
[ 4542.639789] configure-ovs.sh[9846]: No default route found on attempt: 12
[ 4542.639816] configure-ovs.sh[9846]: + sleep 5
[ 4547.640829] configure-ovs.sh[9846]: + '[' 12 -lt 12 ']'
[ 4547.641899] configure-ovs.sh[[ 4548.452400] audit: type=1130 audit(1631957056.734:279): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='u
nit=ovs-configuration comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
9846]: + '[' '' = br-ex ']'
[ 4547.658762] configure-ovs.sh[9846]: + '[' -z '' ']'
[ 4547.672546] configure-ovs.sh[9846]: + echo 'ERROR: Unable to find default gateway interface'
[ 4547.673523] configure-ovs.sh[9846]: ERROR: Unable to find default gateway interface
[ 4547.673900] configure-ovs.sh[9846]: + exit 1```

@yaroslavkasatikov
Copy link

I have rebooted the node with AWS Console. It has been loaded fine. So, I'm waiting for new kernel panic ;)

@vrutkovs
Copy link
Member

I got the same issue

How do we know its the same issue? In any case, kernel panics should be reported to Fedora bugzilla

@yaroslavkasatikov
Copy link

yaroslavkasatikov commented Sep 18, 2021 •
edited
Loading

Ok, it may be different case, but with the same symptoms.
All happens in the same way like @gialloguitar wrote. Nodes become NotReady and don't reply on ping.
It happens time to time with different nodes. The period of 'Ready' status is different. (some nodes are active for several days, some may be rebooted each 10-20mins)

The different is I have clear OKD 4.7.0-0.okd-2021-08-22-163618 cluster, not from upgrade.

The main question for me now - is it possible to downgrade cluster to previous version?

@JaimeMagiera @vrutkovs should I upload must-gather here? it's 125Mb logs

@JaimeMagiera
Copy link
Contributor

Yes, please upload a must-gather. Also, please file a Bugzilla since this is a kernel panic how-to

@gialloguitar
Copy link
Author

As I found out, it seems related with folowing issue from Fedora CoreOS project
coreos/fedora-coreos-tracker#957

I have replaced kernel version of all worker node at one of our cluster and will watch to it

@m-yosefpor
Copy link

This is a critical issue which seems to have impacted many users.

I think all of these refer to the same issue?

@openshift-bot
Copy link

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

@openshift-ci openshift-ci bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Dec 30, 2021
@openshift-bot
Copy link

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

@openshift-ci openshift-ci bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Jan 29, 2022
@openshift-bot
Copy link

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

@openshift-ci
Copy link

openshift-ci bot commented Feb 28, 2022

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@openshift-ci openshift-ci bot closed this as completed Feb 28, 2022
@achilles-git
Copy link

@gialloguitar how did you replace the kernel version of all worker nodes ??

I have replaced kernel version of all worker node at one of our cluster and will watch to it

is it using machineconfig ??

@gialloguitar
Copy link
Author

gialloguitar commented Apr 22, 2022 •
edited
Loading

@gialloguitar how did you replace the kernel version of all worker nodes ??

I have replaced kernel version of all worker node at one of our cluster and will watch to it

is it using machineconfig ??

Yes, all new nodes apply machineconfig when provisioning

    apiVersion: machineconfiguration.openshift.io/v1
    kind: MachineConfig
    metadata:
      labels:
        machineconfiguration.openshift.io/role: worker
      name: 99-worker-kernel-version-lock
    spec:
      config:
        ignition:
          version: 3.2.0
        systemd:
          units:
            - contents: |
                [Unit]
                Description=Machine Config to set up a Kernel Version
                # Make sure it runs only on OSTree booted system
                ConditionPathExists=/run/ostree-booted
                # Make sure it only one time when firstboot
                ConditionPathExists=!/var/local/kernel-version.lock
                Wants=network-online.target
                After=network-online.target
                [Service]
                Environment=KERNEL_REPO=https://kojipkgs.fedoraproject.org/packages/kernel
                Environment=KERNEL_PACKAGE_VERSION=<version>
                Environment=KERNEL_PACKAGE_RELEASE=<release>
                Environment=KERNEL_PACKAGE_ARCH=<arch>
                Type=oneshot
                RemainAfterExit=yes
                ExecStartPre=/usr/bin/touch /var/local/kernel-version.lock
                ExecStart=/bin/sh -c "rpm-ostree override replace --reboot ${KERNEL_REPO}/${KERNEL_PACKAGE_VERSION}/${KERNEL_PACKAGE_RELEASE}/${KERNEL_PACKAGE_ARCH}/kernel{,-core,-modules}-${KERNEL_PACKAGE_VERSION}-${KERNEL_PACKAGE_RELEASE}.${KERNEL_PACKAGE_ARCH}.rpm"
                [Install]
                WantedBy=multi-user.target
              enabled: true
              name: "kernel-version-lock.service"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@vrutkovs @openshift-bot @yaroslavkasatikov @gialloguitar @JaimeMagiera @m-yosefpor @achilles-git