Skip to content
This repository has been archived by the owner on Dec 7, 2023. It is now read-only.

Fix boot on recent distribution like Ubuntu >=19.10 #526

Merged

Conversation

PaulGrandperrin
Copy link
Contributor

@PaulGrandperrin PaulGrandperrin commented Feb 17, 2020

Trying to boot an Ubuntu 19.10 or 20.04 would result in this error:

$ sudo ignite run paulgrandperrin/ignite-ubuntu:20.04 --name test -i
...
[    2.295577] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
[    2.303564] VFS: Mounted root (ext4 filesystem) on device 254:0.
[    2.313282] devtmpfs: mounted
[    2.319174] Freeing unused decrypted memory: 2040K
[    2.333506] Freeing unused kernel image memory: 1352K
[    2.344741] Write protecting the kernel read-only data: 14336k
[    2.371993] Freeing unused kernel image memory: 2020K
[    2.383185] Freeing unused kernel image memory: 548K
[    2.388744] Run /sbin/init as init process
[    2.406294] Run /etc/init as init process
[    2.410840] Run /bin/init as init process
[    2.415518] Run /bin/sh as init process
[    2.426454] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[    2.430159] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.47 #1
[    2.430159] Call Trace:
[    2.430159]  dump_stack+0x63/0x85
[    2.430159]  ? rest_init+0x40/0xb0
[    2.430159]  panic+0xdf/0x239
[    2.430159]  ? putname+0x47/0x50
[    2.430159]  ? rest_init+0xb0/0xb0
[    2.430159]  kernel_init+0xf0/0x110
[    2.430159]  ret_from_fork+0x1f/0x40
[    2.430159] Kernel Offset: disabled
[    2.430159] Rebooting in 1 seconds..
INFO[0004] firecracker exited: status=0

Installing busybox and setting init=/bin/static-sh as kernel-args correctly spawns a shell but then
trying to execute /bin/bash (or any dynamic executable) results in this error:

$ sudo ignite run paulgrandperrin/ignite-ubuntu:20.04 --name test --kernel-args "pci=off ip=dhcp console=ttyS0 reboot=k panic=1 root=/dev/vda rw virtio_mmio.device=4K@0xd0000000:5 virtio_mmio.device=4K@0xd0001000:6 init=/bin/static-sh" -i
...
[    2.338103] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
[    2.347059] VFS: Mounted root (ext4 filesystem) on device 254:0.
[    2.357809] devtmpfs: mounted
[    2.364615] Freeing unused decrypted memory: 2040K
[    2.379725] Freeing unused kernel image memory: 1352K
[    2.390496] Write protecting the kernel read-only data: 14336k
[    2.419060] Freeing unused kernel image memory: 2020K
[    2.430295] Freeing unused kernel image memory: 548K
[    2.435784] Run /bin/static-sh as init process
[    2.464800] random: fast init done

BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu5) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/bin/static-sh: can't access tty; job control turned off
/ # stat /bin/bash
  File: /bin/bash
  Size: 1183448         Blocks: 2312       IO Block: 4096   regular file
Device: fe00h/65024d    Inode: 10266       Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-02-17 17:38:12.000000000
Modify: 2019-11-13 00:54:47.000000000
Change: 2020-02-17 17:38:12.000000000

/ # /bin/bash
/bin/static-sh: /bin/bash: not found

The "not found" error is not about the executable itself, but about the
dynamic linker.
The dynamic linker address is /lib64/ld-linux-x86-64.so.2 which is itself
a symlink to /lib/x86_64-linux-gnu/ld-2.30.so.

Unfortunatly on the firecracker VM image, /lib/ only contains the modules
directory from the ignite-kernel container image and nothing else.

This is because on those distributions, /lib is supposed to be a
symlink to usr/lib but when the /lib directory from the kernel is
extracted, it overrides this symlink, breaking dynamic linking.

From the gnu-tar manual, the --keep-directory-symlink flag provides exactly
the semantic we are looking for.

This patch simply adds this flag where the kernel tree is extracted over
the OS tree.

Trying to boot an Ubuntu 19.10 or 20.04 would result in this error:

```
$ sudo ignite run paulgrandperrin/ignite-ubuntu:20.04 --name test -i
...
[    2.295577] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
[    2.303564] VFS: Mounted root (ext4 filesystem) on device 254:0.
[    2.313282] devtmpfs: mounted
[    2.319174] Freeing unused decrypted memory: 2040K
[    2.333506] Freeing unused kernel image memory: 1352K
[    2.344741] Write protecting the kernel read-only data: 14336k
[    2.371993] Freeing unused kernel image memory: 2020K
[    2.383185] Freeing unused kernel image memory: 548K
[    2.388744] Run /sbin/init as init process
[    2.406294] Run /etc/init as init process
[    2.410840] Run /bin/init as init process
[    2.415518] Run /bin/sh as init process
[    2.426454] Kernel panic - not syncing: No working init found.  Try passing init= option to kernel. See Linux Documentation/admin-guide/init.rst for guidance.
[    2.430159] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.47 weaveworks#1
[    2.430159] Call Trace:
[    2.430159]  dump_stack+0x63/0x85
[    2.430159]  ? rest_init+0x40/0xb0
[    2.430159]  panic+0xdf/0x239
[    2.430159]  ? putname+0x47/0x50
[    2.430159]  ? rest_init+0xb0/0xb0
[    2.430159]  kernel_init+0xf0/0x110
[    2.430159]  ret_from_fork+0x1f/0x40
[    2.430159] Kernel Offset: disabled
[    2.430159] Rebooting in 1 seconds..
INFO[0004] firecracker exited: status=0
```

Installing busybox and setting `init=/bin/static-sh` as kernel-args correctly spawns a shell but then
trying to execute `/bin/bash` (or any dynamic executable) results in this error:

```
$ sudo ignite run paulgrandperrin/ignite-ubuntu:20.04 --name test --kernel-args "pci=off ip=dhcp console=ttyS0 reboot=k panic=1 root=/dev/vda rw virtio_mmio.device=4K@0xd0000000:5 virtio_mmio.device=4K@0xd0001000:6 init=/bin/static-sh" -i
...
[    2.338103] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
[    2.347059] VFS: Mounted root (ext4 filesystem) on device 254:0.
[    2.357809] devtmpfs: mounted
[    2.364615] Freeing unused decrypted memory: 2040K
[    2.379725] Freeing unused kernel image memory: 1352K
[    2.390496] Write protecting the kernel read-only data: 14336k
[    2.419060] Freeing unused kernel image memory: 2020K
[    2.430295] Freeing unused kernel image memory: 548K
[    2.435784] Run /bin/static-sh as init process
[    2.464800] random: fast init done

BusyBox v1.30.1 (Ubuntu 1:1.30.1-4ubuntu5) built-in shell (ash)
Enter 'help' for a list of built-in commands.

/bin/static-sh: can't access tty; job control turned off
/ # stat /bin/bash
  File: /bin/bash
  Size: 1183448         Blocks: 2312       IO Block: 4096   regular file
Device: fe00h/65024d    Inode: 10266       Links: 1
Access: (0755/-rwxr-xr-x)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-02-17 17:38:12.000000000
Modify: 2019-11-13 00:54:47.000000000
Change: 2020-02-17 17:38:12.000000000

/ # /bin/bash
/bin/static-sh: /bin/bash: not found
```

The "not found" error is not about the executable itself, but about the
dynamic linker.
The dynamic linker address is `/lib64/ld-linux-x86-64.so.2` which is itself
a symlink to `/lib/x86_64-linux-gnu/ld-2.30.so`.

Unfortunatly on the firecracker VM image, /lib/ only contains the `modules`
directory from the `ignite-kernel` container image and nothing else.

This is because on those distributions, `/lib` is supposed to be a
symlink to `usr/lib` but when the `/lib` directory from the kernel is
extracted, it overrides this symlink, breaking dynamic linking.

From the gnu-tar manual, the `--keep-directory-symlink` provides exactly
the semantic we are looking for.

This patch simply adds this flag where the kernel tree is extracted over
the OS tree.
@PaulGrandperrin
Copy link
Contributor Author

I'd like to add too things:

  • --keep-directory-symlink is only available on GNU tar, not on BSD tar. -k is available on both and might do the job.
  • I tried to change the /lib symlink destination from usr/lib to /usr/lib to check if it could allow an untrusted OS image to make the process overwrite files on the host OS. Fortunately, it does not, but the resulting microvm does not have the /lib/modules from the kernel image.

@stealthybox
Copy link
Contributor

@PaulGrandperrin 👋 Thanks for the detailed documentation of this issue and the communication in slack that you provided about not being able to form a kernel image that used paths that were not /boot/ or /lib/ (such as /usr/lib/modules/)
Your well written comments really helped us understand the problem.

@luxas and I have done some experimentation and we plan to merge your PR and do a follow-up to make the tar command more portable as well as fix a bug where the kernel image importer was allowing /lib/ directories other than /lib/modules.

We really appreciate your patch and willingness to play with and improve ignite :)

@stealthybox stealthybox merged commit 2c61783 into weaveworks:master Apr 10, 2020
luxas added a commit that referenced this pull request Apr 10, 2020
@luxas luxas added this to the v0.7.0 milestone Jun 2, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants