Singularity should allow any user to build from a definition file.

[preney]

Version of Singularity:

What version of Singularity are you using?

$ singularity version
3.7.3

but the version is irrelevant as this is a feature request.

Expected behavior

Singularity should allow any user to build from a definition file. Not all definition files require root/fakeroot permissions. Clearly users can already build images in user space --not all images will succeed-- and if they don't build errors are reported. Building from definition files shouldn't be any different.

Actual behavior

The actual behaviour unnecessarily mandates root, fakeroot, or remote permissions, e.g.,

$ singularity build file.sif file.def 
FATAL:   You must be the root user, however you can use --remote or --fakeroot to build from a Singularity recipe file

By not allowing users to build form definition files strongly encourages the non-use of Singularity definition files.

Steps to reproduce this behavior

N/A.

What OS/distro are you running

N/A.

How did you install Singularity

N/A.

[dtrudg]

Hi @preney - this is a discussion that has been had a few times in Singularity's history. The position is as follows:

Almost all definition files we see in use will include apt or yum installation of packages into the container. This generally requires a real root or fakeroot root user in the container build in order to succeed.

It's not possible to explain simply to non-expert Linux users what would / would not require root or fakeroot in a definition file. Resulting random failures would be confusing and frustrating and lead to a poor user experience.

Since all modern distributions support unprivileged user namespace we advocate for use of --fakeroot, and remote builds are available where fakeroot cannot be configured.

Edit - to reconsider this we'd need to have evidence of a number of builds that explicitly are possible without root privilege, and that are being performed regularly by multiple users.

[preney]

Hi @dtrudg and thanks for your reply! :-)

Not all modern Linux distributions require root to install packages, etc. Gentoo and NixOS are two that do not need root. (I have a definition file for Gentoo (it builds the latest Gentoo release completely) and a work-in-progress for NixOS --but they are useless for users since root is required to use them.) Some might argue those are niche distributions --but that is not the point: they can tolerate not being root, function properly, and can be used to move forward within user space only Singularity.

Moreover, using programs like fakeroot one can easily successfully install packages as a user in Debian, Ubuntu, etc. provided those programs are somehow available in the image (i.e., previously installed or as statically-linked binaries bind mounted in to the image). This is very easy to demonstrate...

# Pull down Debian v10 say from DockerHub (i.e., user space)
$ singularity build -s debian.dir docker://debian:10
# Unless we have static binaries for fakeroot (if we did then only user space would be required), we'll need root to install fakeroot...
$ sudo singularity shell -w debian.dir/
Singularity> apt update
# Might as well install fakechroot while we are at it...
Singularity> apt install fakeroot fakechroot
Singulartiy> exit
# Flatten the filesystem permissions back to user space since the previous command used sudo...
$ sudo chown -R $(id -u):$(id -g) debian.dir
# Enter a user space writable Singularity shell...
$ singularity shell -w debian.dir/
# The next line will fail since user is not root...
Singularity> apt install build-essential
# But will succeed if one runs it with fakeroot...
Singularity> fakeroot apt install build-essential
# etc.

i.e., once fakeroot is somehow present in the image, the rest of the above is 100% user-space.

Certainly if Singularity provided a fakeroot-program-like ability, then one could fake being root without requiring --fakeroot (which requires UID:GID mappings) or root permissions completely within user space. But even without such a feature, if the fakeroot program is provided in the base image (e.g., bind-mounted as a statically linked binary or somehow installed), then users can do virtually everything they need in user space. The majority of programs can be installed this way --one just needs to be able to install them. A small number of programs won't run because of special permissions --but those programs won't run properly in user space anyways.

---

With respect to Singularity definition files, it would be straight-forward to have a line, e.g., say immediately after the "Bootstrap:" line, such as "UserSpaceBuildingPermitted: true" to "mark" such a Singularity definition file as being okay to run as a normal user and its omission would mean root/fakeroot is required with suitable quick-error messages to the user. This would make it very easy to check before executing any script commands whether or not it can be run in user space.

--

That said, if such was discussed and settled earlier and won't change, at least I've an answer: essentially the Singularity project's position is that Singularity definition file feature is simply not a good time investment for users: I/others should script things using third-party programs, custom scripts, etc. instead of using Singularity definition files so things can be done in user space. Using third-party programs, custom scripts, etc. for user space needs implies that the same should be used for root space when/where needed. (Why bring in even more scripts/tools/syntax involved --even if it is Singularity's?) Thus, it makes sense to conclude Singularity definition file feature is not a good time investment. If I am incorrect about this, please advise me (and future readers of this).

(Restated, Singularity enables user-space-only executions which helps, for example, deal with security issues. Things that don't address such in any way, e.g., definition files, means those things are of no value to user-space needs and therefore tools need to be used/built to address any short-comings of such.)

[dtrudg]

Hmmm, okay, over a cup of tea :-) I can see a potential argument for it with a special line in the definition file to enable this. I have experimented with Guix and Nix myself in the past, and had considered them somewhat orthogonal to containers... more that you'd create your environment with whatever you need without involving containers at all.

To my mind the number of people familiar with this userspace fakeroot wrapping etc. is likely an extremely small number of Singularity users, and in the container world the user namespace mapping approach is definitely where things are generally going, e.g. podman and other tooling operates.

It's perhaps something that we'd consider a contribution for as an optional feature enabled in that way.

Going to page the project lead @gmkurtzer and other maintainers @cclerget @ikaneshiro here... and let them give opinions.

Cheers.

[Hoeze]

I happen to have exactly this issue in our lab.
We would like to use the IntOGen pipeline but we cannot build the necessary containers on our cluster.

• The only sudo command I could find in the whole repo was sudo singulartiy build.
• We tried the fakeroot feature but this obviously fails on shared file systems.

Podman has podman --storage-opt ignore_chown_errors=true --storage-opt mount_program=/usr/bin/fuse-overlayfs. Using this, I can run virtually anything that does not exceed my user privileges.
I would expect the same to work with singularity by default.

It's a shame that most container runtimes require user namespaces that do not work on shared file systems...

[dtrudg]

Podman has podman --storage-opt ignore_chown_errors=true --storage-opt mount_program=/usr/bin/fuse-overlayfs. Using this, I can run virtually anything that does not exceed my user privileges.
I would expect the same to work with singularity by default.

A singularity build to build a container is a bit different than execution of the container. The comparison would be with buildah rather than podman for this.... which admittedly may also be able to work in the same way.

For running a container using singularity run/exec/shell it should be the case that you "can run virtually anything that does not exceed my user privileges"... since you are the same user in the container. If you have things that don't work we'd be interested to know more detail.

[Hoeze]

Thanks for your answer @dtrudg. Should I open a separate issue then?

---

For running a container using singularity run/exec/shell it should be the case that you "can run virtually anything that does not exceed my user privileges"... since you are the same user in the container. If you have things that don't work we'd be interested to know more detail.

The step that fails is the build/containers/build.sh file:

# [~/intogen/build]$ make
INFO:    Creating SIF file...
INFO:    Build complete: ../containers/vep.simg
Building bgSignature container
containers/build.sh containers/signature ../containers/signature.simg
<user> is not allowed to run sudo on <host>.  This incident will be reported.
make: *** [containers/signature/signature.mk:10: ../containers/signature.simg] Error 1

I tried replacing the sudo command in build.sh:

# [~/intogen/build/containers/]$ cat build.sh
<...>
(cd ${FOLDER} && singularity build ${tmpdir}/$(basename ${IMAGE}) Singularity)
mv ${tmpdir}/$(basename ${IMAGE}) ${IMAGE}
# sudo chown ${USER}: ${IMAGE}

This failed due to:
FATAL: You must be the root user, however you can use --remote or --fakeroot to build from a Singularity recipe file.

I replaced singularity build with singularity build --fakeroot and tried again:

# [~/intogen/build]$ make
Building bgSignature container
containers/build.sh containers/signature ../containers/signature.simg
FATAL:   could not use fakeroot: no mapping entry found in /etc/subuid for <user>
make: *** [containers/signature/signature.mk:10: ../containers/signature.simg] Error 255

I do not want to change the uid, I need to keep the current user permissions (=> no namespaces).
Otherwise I cannot read/write the shared file system.

[dtrudg]

Hi @Hoeze - The process you show there that fails is a container build with singularity build, rather than execution of an existing container, so it's expected it will fail like you see. This is not really like using podman to run a container. It's more like using buildah to build a container.

The --fakeroot option will map root in the container to your own user's uid on the host.... so as long as the build process only uses root it can actually work for some situations on a shared filesystem, as the mapping is root in container to uid on host. root access in the container occurs as your user on the shared fs. The subuid/subgid values are used for other non-root uids/gids in the container, which must be mapped to the subuid/subgid ranges that are configured. Those are the things that don't work on a shared fs.

Also, we generally find people use --fakeroot on systems with parallel / network fs without much issue as Singularity will perform the build in a local temporary filesystem that supports subuid/subgid mapping. Once the build of a container into a SIF file is complete it doesn't matter where the resulting container image resides, as the container is loopback mounted out of the image file, avoiding issues with permissions on the shared filesystem. If there is no local filesystem, then on large memory nodes it can be possible to use an in-ram tmpfs and set the TMPDIR for singularity build to point there.

The proposal from .@preney here may not actually help in this case. It would allow builds that are explicitly written to not require root to succeed. That may not be the case for the build occurring here. It is possible that it could be re-written by the pipeline developers to work, however. The build for these pipeline containers is performing apt installs which need to run as root, or the build adapted to use a fakeroot wrapper inside the build definition, as preney is advocating. Just as you cannot directly run an apt install on your host fs as a normal user, you cannot do so inside a container so some uid mapping or root-emulating wrapper is required.

[carterpeel]

Hello,

This is a templated response that is being sent out to all open issues. We are working hard on 'rebuilding' the Singularity community, and a major task on the agenda is finding out what issues are still outstanding.

Please consider the following:

1. Is this issue a duplicate, or has it been fixed/implemented since being added?
2. Is the issue still relevant to the current state of Singularity's functionality?
3. Would you like to continue discussing this issue or feature request?

Thanks,
Carter

[preney]

This issue is still open and I believe relevant. Personally, I am waiting for responses per this message #5941 (comment) earlier in this issue to discuss, etc. Thank you.

[stale]

This issue has been automatically marked as stale because it has not had activity in over 60 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because no response was provided within 7 days.

[preney]

While I get a bot closing things, it really shouldn't if HPCng is backlogged, etc. given recent goings-on. @carterpeel asked a question, I responded and there has been no further response from HPCng.

[DrDaveD]

I see this as being related to issue #3612. I don't think there should need to be an option in the definition file at all, I think it should be the default for the singularity build command to silently invoke the --fakeroot mode if it is available. @cclerget @ikaneshiro do you agree that's a logical thing to do? Of course, having it be a priority for someone to implement is separate matter.

[preney]

Thanks for your response. While silently invoking --fakeroot could be done (but know such does not fix the issue I raised), there are cases where everything can be built without root, sudo, --fakeroot, etc. For example, one can completely download and install Gentoo Linux using only normal user perms, e.g., consider this singularity-minimal-stage3-gentoo-latest-current-amd64.def file of mine:

Bootstrap: scratch

%setup
# Gentoo Base URL to download from...
GENTOO_BASE_URL="https://bouncer.gentoo.org/fetch/root/all/releases/amd64/autobuilds"

# Establish a temporary working directory...
INSTALL_TMPDIR="$(mktemp -d -t singularity-gentoo.XXXXXXXXXX)"
cd "$INSTALL_TMPDIR"

# Determine the latest release stage3 tarball...
curl -L "$GENTOO_BASE_URL"/latest-stage3-amd64.txt | grep -v '^#' | awk '{ print $1 }' >gentoo-latest-stage3.txt
GENTOO_TARBALL_URL="$GENTOO_BASE_URL"/"$(cat gentoo-latest-stage3.txt)"

# Download the tarball...
curl -L "$GENTOO_TARBALL_URL" -o "$SINGULARITY_ROOTFS"/gentoo-tarball

# Untar the tarball...
cd "$SINGULARITY_ROOTFS"
tar xpf ./gentoo-tarball --xattrs-include='*.*' --numeric-owner

# Remove the tarball...
rm -f "$SINGULARITY_ROOTFS"/gentoo-tarball

# Remove temporary directory...
rm -rf "$INSTALL_TMPDIR"

%post
#!/bin/bash
THIS_ENV=/.singularity.d/env/A0-this_env.sh
cat >$THIS_ENV <<ZZEOF
#!/bin/bash
source /etc/profile
export PROMPT_COMMAND='echo -n "Singularity(\$SINGULARITY_NAME):$PS1 "'
ZZEOF
chmod 755 $THIS_ENV

# Run emerge-webrsync...
source /etc/profile
emerge-webrsync

# Enable the first default profile...
echo "Selecting profile 1..."
eselect profile set 1

# Select and generate locale(s)...
#cat /usr/share/i18n/SUPPORTED >/etc/locale.gen
echo "C.UTF-8 UTF-8" >/etc/locale.gen
locale-gen

# OpenRC: Set timezone to Etc/UTC...
echo "Etc/UTC" >/etc/timezone
emerge --config sys-libs/timezone-data

# Updating environment...
source /etc/profile
env-update

The reason for this issue was to ask that .def files be allowed to be run by any user (implicitly using the permissions of the user running such). This would only require removing checks for root (or turning those checks in to warning messages that not using root / --fakeroot may not work), i.e., no elevated permissions use at all. (Alternatively, a special option could be added to "override" the root / --fakeroot check to permit running things 100% in user space.)

[florianblume]

I'd be interested in this feature, too. I'm running singularity on our university's cluster where we don't have root privileges. Therefore I cannot use singularity build and instead am building a Dockerfile on a different machine where I am root, push it to Docker hub and pull that image using Singularity. I'm using the resulting .sif file with MLFlow which copies the code to a new layer of the image.

I.e. the build manifest looks like this

Bootstrap: localimage

From: [path to local sif file].sif
%files
. /mlflow/projects/code/
%runscript
cd /mlflow/projects/code/`

This doesn't require root privileges, right? As of now I've modified MLFlow to not build but just run the image, losing the feature that the code is persisted within the image.

[smokhov]

Came to find here to my surprise regular users can't build containers as non-root users from .def file. That presents a problem to our users on our HPC cluster too where people are forced to build things outside and then load up the images to find that they don't work due to kernel mismatches and whatnot.

[ngam]

Any update on this issue?

[preney]

I am still waiting for an update on this issue.

[DrDaveD]

I don't think there would be any objection to such a feature if someone were interested in submitting a pull request.

[smokhov]

On Mon, 13 Dec 2021, DrDaveD wrote: Date: Mon, 13 Dec 2021 10:01:50 -0800 I don't think there would be any objection to such a feature if someone were interested in submitting a pull request.

What does this entail? Any shortcut pointers as to where start looking at for any possible PRs?

…

-s

[smokhov]

I also don't understand the need for /etc/subuid and /etc/subgid when using the --fakeroot option. With hundreds of users added to the cluster and being added or removed from the dozens of nodes doesn't scale. Can a single /etc/subgid cover all the users who are members of the same group?

[DrDaveD]

On Mon, 13 Dec 2021, DrDaveD wrote: Date: Mon, 13 Dec 2021 10:01:50 -0800 I don't think there would be any objection to such a feature if someone were interested in submitting a pull request.
What does this entail? Any shortcut pointers as to where start looking at for any possible PRs?

I haven't touched that part of the code before but I would just start with searching for the error message you saw and/or the keywords in definition files.

I also don't understand the need for /etc/subuid and /etc/subgid when using the --fakeroot option. With hundreds of users added to the cluster and being added or removed from the dozens of nodes doesn't scale. Can a single /etc/subgid cover all the users who are members of the same group?

A single /etc/subuid file can, yes, but there needs to be an entry in the file for each individual. This is a standard from the rootless containers people, for example podman. The Linux kernel doesn't allow unprivileged user namespaces to map more than one user id, and this feature enables having more user ids in a fakeroot type of mode via an increased privilege helper program.

[ShamrockLee]

I generate a definition file and build the Singularity image with a Nix expression. The generated definition file looks like:

Boostrap: scratch

%setup
    # Copy the whole dependency closure
    nix-store --export $(nix-env -qR "/nix/store/xfb60lr4bh1dgn93cswif4j3gadfpjyp-xAna_monoZ_preselect-0.0.1")
    | nix-store --store-path "${SINGULARITY_ROOTFS}" --import

%environment
    export PATH="/nix/store/xfb60lr4bh1dgn93cswif4j3gadfpjyp-xAna_monoZ_preselect-0.0.1/bin:$PATH"

There seems no reason for it to require root privileges.

It would be great to add a --no-root flag, while add cautions inside the documentations and help messages about the potential risk to fail.

Motivation

Nix is not only a package manager for NixOS, but a cross-platform package/project manager that allows the developers to build packages and setup environments in a declarative and reproducible manner, which shares similar pursuit with Singularity.

https://indico.cern.ch/event/712739/contributions/2927853/attachments/1613818/2578668/chris_burr.pdf

Different approaches have been developed to make Nix portable and work in restricted environments while enjoying the prebuilt binaries at the same time. However, the huge overhead of many-small-file operations on Network-based filesystems make it hard to copy/extract the whole store onto grid. That's where the SquashFS-based Singularity container comes in handy.

[ngam]

https://indico.cern.ch/event/712739/contributions/2927853/attachments/1613818/2578668/chris_burr.pdf

tagging @chrisburr to offer any thoughts on this topic

[ShamrockLee]

The generation is now stuck with apptainer/apptainer#205, but that seems to be related to some other technical issue.

BTW, I've heard that developments have now moved to https://github.com/apptainer/apptainer . Is there any plan for the issues to get migrated?

[DrDaveD]

Yes the plan is to create new issues in the apptainer repository referring back to old issues here. There are a couple of people going through the issues to decide which ones to do that to and which ones to just close, but I'll go ahead and do it for this one.

[DrDaveD]

Continued in apptainer/apptainer#215.