Creating PPC64LE containers from X86-64 (or ARM?) hosts

[ptheywood]

Bede requires containers built for PPC64LE architecture CPUs, and most users will not have access to PPC64LE systems where they have the required privileges (root) to create containers.

MIT's Satori system includes some documentation on creating docker containers from X86-64 hosts:

https://mit-satori.github.io/tips-and-tricks/singularity-tips/index.html?highlight=singularity#set-up-to-run-docker-in-ppc64le-mode-on-an-x86-machine

Alternatively, @loveshack has suggested using Charliecloud and QEMU:

It basically works the same as natively on
bede, apart from not necessarily being able to pull an image directly.
You do probably need a version >=0.24 for the --arch support, which is
packaged for EPEL but not Debian 11

$ # This is amd64 Debian with Charliecloud 0.25 installed in ~/.local:
$ cat /etc/debian_version 
11.2
$ # For dockerfile RUN:
$ sudo apt install -y qemu-user-static
 [...]
$ # We have a trivial Dockerfile ...
$ cat Dockerfile
FROM debian:11
RUN apt install fakeroot file -y
$ # ... from which we can make a ppc64le container image (d11p9):
$ ch-build2dir -t d11p9 --force --arch ppc64le . .
 [...]
$ # For some reason the ld.so link is missing
$ ln -s /lib/powerpc64le-linux-gnu/ld64.so.2 d11p9/lib64
$ # You can check the architecture ...
$ file d11p9/bin/bash 
d11p9/bin/bash: ELF 64-bit LSB pie executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, interpreter /lib64/ld64.so.2, BuildID[sha1]=f639d8b9f2cf4466946c6b33e4015e13fd447d43, for GNU/Linux 3.10.0, stripped
$ # ... and do it inside the container, where /usr/bin/file is ppc64le:
$ ch-run d11p9 -- file /bin/bash
/bin/bash: ELF 64-bit LSB pie executable, 64-bit PowerPC or cisco 7500, version 1 (SYSV), dynamically linked, interpreter /lib64/ld64.so.2, BuildID[sha1]=f639d8b9f2cf4466946c6b33e4015e13fd447d43, for GNU/Linux 3.10.0, stripped


$ # You can't pull a ppc64le CentOS image from docker hub, and I'd
$ # have to check how to get one otherwise (pending Alma's ppc64le
$ # coming out of beta?), but you can make one another way.
$ # Here's one I made earlier on bede (which takes a while):
$ cat Dockerfile.c8p9
FROM ppc64le/centos:8
$ # For installs inside container:
RUN dnf install -y fakeroot epel-release
RUN dnf install -y --setopt=install_weak_deps=false epel-release \
                'dnf-command(config-manager)' \
 && dnf config-manager --enable powertools \
 && dnf config-manager --enable epel \
 && dnf config-manager --enable appstream \
 && dnf copr enable loveshack/livhpc -y \
 && dnf install -y --setopt=install_weak_deps=false \
        cube-devel opari2 bash-completion libdwarf-devel cube scorep-openmpi scorep \
 && dnf clean all
# lustre mount point
RUN mkdir /nobackup
$ # images.linuxcontainers.org currently gets me an empty UK mirror, 
$ # specify a mirror site.
$ # Adjust the URL for image du jour -- why don't they link "latest"?
$ wget https://us.lxd.images.canonical.com/images/centos/8/ppc64el/default/20220113_07:08/rootfs.tar.xz
 [...]
$ ch-image import rootfs.tar.xz ppc64le/centos:8
 [...]
$ ch-build2dir --force -t c8-ppc64le -f Dockerfile.c8p9 . .
 [...]
$ ch-run c8-ppc64le -- head -5 /usr/share/scorep/scorep.summary
Configure command:
  ../configure                  '--build=powerpc64le-redhat-linux-gnu' \
                                '--host=powerpc64le-redhat-linux-gnu' \
                                '--program-prefix=' \
                                '--disable-dependency-tracking' \
$ 

[ptheywood]

Singularities --fakeroot option might make this process much simpler, allowing users to build (some/most) containers on Bede directly, rather than having to find an extra Power9 system / use QEMU.

https://sylabs.io/guides/3.5/user-guide/fakeroot.html

This requires a recent-ish kernel, and must be enabled for the singularity install. I do not know if this is currently enabled on Bede, or if this would require changes to the installation (if this is desired).

[jhodrien]

Fakeroot I've used it on CentOS 7 machines, and from a user perspective, it's lovely. It's enabled in the EPEL singularity build, since that's the version I've tested (on EL7/8).

It adds some filesystem management complexity, as files can be created by users owned by any of their subordinate uids/gids.

[loveshack]

I updated the write-up of building images unprivileged on a foreign architecture for current versions: https://fx.srht.site/posts/2022-01-21-building-foreign-images.html
https://hpc.github.io/charliecloud/ has documentation about multi-node use.