openEuler-23.09配套cloud-init在AWS上存在的严重问题

[wjunLu]

背景

openEuler-23.09创新版发布以后，通过EulerPublisher制作AMI镜像并手动向AWS Marketplace提交发布申请，制作的x86_64和aarch64两种架构版本的镜像均审核失败，提示失败原因如下：

[{"ErrorCode":"INSTANCE_UNREACHABLE","ErrorMessage":"Failed to launch instance or failed to reach port 22. Provide an updated AMI that can be launched and provides access on the correct port [SSH port for Linux AMI. RDP and WinRM port for Windows AMI]. Learn more at: https://docs.aws.amazon.com/marketplace/latest/userguide/best-practices-for-building-your-amis.html#building-an-ami"}]

初步方向是ssh连接失败，但是为什么失败的详细原因还不清楚，只能从控制台起实例观察一下，结果发现相关的系统日志(/var/log/messages)如下：

Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: Cloud-init v. 23.2.2 running 'init' at Mon, 09 Oct 2023 06:44:34 +0000. Up 5.51 seconds.
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: +++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: | Device |   Up  |  Address  |    Mask   | Scope |     Hw-Address    |
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: |  ens5  | False |     .     |     .     |   .   | 02:37:a8:07:b7:51 |
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: |   lo   |  True | 127.0.0.1 | 255.0.0.0 |  host |         .         |
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: |   lo   |  True |  ::1/128  |     .     |  host |         .         |
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: +--------+-------+-----------+-----------+-------+-------------------+
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: +-------+-------------+---------+-----------+-------+
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: | Route | Destination | Gateway | Interface | Flags |
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: +-------+-------------+---------+-----------+-------+
Oct  9 06:44:34 ip-172-31-9-209 cloud-init[580]: ci-info: +-------+-------------+---------+-----------+-------+

可以明显看到实例的网卡ens5没有拉起，故而无法发起通信，这也就能解释为什么ssh无法连接，但是为什么无法启动网卡，原因还不得而知。

初步猜测

结合以上信息大概有个判断：cloud-init有问题或者其配置文件没有正确配置，原因如下：

1. EulerPublisher在制作AMI时，首先从openEuler Repo获取基础镜像，从基础镜像起实例且通过ssh访问安装和修改配置，这些过程都正常（不存在ssh无法访问的问题，而且通过单独起基础镜像实例观察日志，也验证了网卡启动正常），那么问题就出在二次定制时安装和配置的内容上，值得注意的是基础镜像本身没有cloud-init，二次定制过程最关键的就是安装了cloud-init
2. 上文提到制作好的AMI实例系统日志显示网卡没有启动的信息也是在cloud-init阶段
3. 进行初步实验，在基础镜像实例中仅通过yum install -y cloud-init安装cloud-init这一个软件，在实例重启时就会复现网卡不能启动的问题，观察失败情况下的系统日志，发现如下细节：

Oct  9 06:44:34 ip-172-31-9-209 systemd[1]: Starting Network Manager Script Dispatcher Service...
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.3750] dhcp: init: Using DHCP client 'dhclient'
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.3756] manager: (lo): new Loopback device (/org/freedesktop/NetworkManager/Devices/1)
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.3801] device (lo): state change: unmanaged -> unavailable (reason 'connection-assumed', sys-iface-state: 'external')
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.3846] device (lo): state change: unavailable -> disconnected (reason 'connection-assumed', sys-iface-state: 'external')
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.3859] device (lo): Activation: starting connection 'lo' (d0b93703-703c-46f2-9916-52ca9eedb15e)
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.3875] manager: (ens5): new Ethernet device (/org/freedesktop/NetworkManager/Devices/2)
Oct  9 06:44:34 ip-172-31-9-209 dbus-daemon[518]: [system] Successfully activated service 'org.freedesktop.nm_dispatcher'
Oct  9 06:44:34 ip-172-31-9-209 systemd[1]: Started Network Manager Script Dispatcher Service.
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.3905] ovsdb: disconnected from ovsdb
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.3906] device (lo): state change: disconnected -> prepare (reason 'none', sys-iface-state: 'external')
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.3909] device (lo): state change: prepare -> config (reason 'none', sys-iface-state: 'external')
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.3912] device (lo): state change: config -> ip-config (reason 'none', sys-iface-state: 'external')
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.3914] device (lo): state change: ip-config -> ip-check (reason 'none', sys-iface-state: 'external')
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.3993] device (lo): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'external')
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.4042] device (lo): state change: secondaries -> activated (reason 'none', sys-iface-state: 'external')
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.4048] device (lo): Activation: successful, device activated.
Oct  9 06:44:34 ip-172-31-9-209 NetworkManager[553]: <info>  [1696833874.4056] manager: startup complete
Oct  9 06:44:34 ip-172-31-9-209 systemd[1]: Finished Network Manager Wait Online.

这一段日志显示了NetworkManager激活环回网口lo的过程，但是并没有激活ens5的日志（ens5是真正与外部通信的网口）。

cloud-init存在缺陷

问题定位

分析到此处，cloud-init的嫌疑已经非常大了，根因还需要进一步定位，在AWS控制台串口访问情况下，继续观察cloud-init的日志（/var/log/cloud-init.log），通过仔细对比发现cloud-init和网络配置的一条关联信息：

[DEBUG]: Writing to /etc/sysconfig/network-scripts/ifcfg-ens5 - wb: [644] 159 bytes

顺藤摸瓜，看看/etc/sysconfig/network-scripts/ifcfg-ens5修改后的内容：

# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=ens5
HWADDR=02:42:95:63:e1:75
NM_CONTROLLED=no
ONBOOT=yes
TYPE=Ethernet
USERCTL=no

至此，基本上问题就找到——NM_CONTROLLED=no决定了ens5网卡将不受NetworkManager控制，那么激活网卡过程也就将忽略ens5，最终导致ens5网卡未能启动。简单地将NM_CONTROLLED=no改为NM_CONTROLLED=yes之后，重启实例网卡正常拉起，AWS检查成功。

根本原因

这个定位过程找到了cloud-init的维护组织，找了相关的开发人员，发现cloud-init的master代码分支删除了cloud-init-20.4-nm-controlled.patch补丁（PR见：https://gitee.com/src-openeuler/cloud-init/pulls/68/files），其中关键一步内容：

@@ -503,7 +503,6 @@ class TestNetCfgDistroRedhat(TestNetCfgDistroBase):
                 GATEWAY=192.168.1.254
                 IPADDR=192.168.1.5
                 NETMASK=255.255.255.0
-                NM_CONTROLLED=no
                 ONBOOT=yes
                 TYPE=Ethernet
                 USERCTL=no

这个补丁就是去掉了NM_CONTROLLED=no，使得网卡默认可以被NetworkManager控制，进而可以被激活；该补丁被删除后就产生了现在的问题。

问题修复和规避

问题修复

23.09版本的cloud-init问题已经提issue进行修复，见https://gitee.com/src-openeuler/cloud-init/issues/I86OE1。

问题规避

尽管master分支已经修复了该问题，但创新版无update, 只能通过master源码编译或直接获取build件的方式获取正常的cloud-init来规避问题，yum源的cloud-init未进修复仍然不可用。