singularity-tools: refactor and add definition-based image buider #224636

ShamrockLee · 2023-04-04T11:15:08Z

Description of changes

Status: work in progress

The original buildImage implementation is commented out to make discussion easier.

Also update trivial-builders.nix:
writeMultipleReferencesToFile: init (#178717)
writeScriptBin, writeShellScriptBin: add meta.mainProgram automatically

Things done

This builder takes a list `path` and reduces the `writeMultipleReferencesToFile` result of each element to a file. It would be helpful when building a container image from a list of packages.

…suffix

ShamrockLee · 2023-04-04T11:44:31Z

Questions:

What does runScript do?
What does runAsRoot do? Is it an analogy to the setup section in the Apptainer/Singularity definition file?
If we would like to make the buildImage interface consistent with that of dockTools, which builder are we going to align with? Key difference between Apptainer and Docker, IIUC:
- Apptainer/Singularity images does not have layers
- Apptainer/Singularity images are usually run immutably, without daemon or process isolation.
Should we replace buildImage with the definition-based implementation? Should we keep the sandbox-based implementation?
I'm personally optimistic about Apptainer's progress in unprivileged image building. We could hopefully build the image without additional virtualization eventually. Considering the current situation, is there a lighter chroot-like mounter to provide /var/singularity/mnt/{container,final,overlay,session,source}? Should we maintain the manually-image-composition script by @posch (Beat singularity-tools up to shape #177908 (comment))?

Cc: @jbedo @SomeoneSerge @dmadisetti

pkgs/build-support/singularity-tools/default.nix

ShamrockLee

Additional questions about the original implementation.

jbedo · 2023-04-04T22:45:37Z

Questions:

What does runScript do?

It's the contents of the runscript script singularity uses when invoked by a singularity run.

What does runAsRoot do? Is it an analogy to the setup section in the Apptainer/Singularity definition file?

Not quite, it's executed inside the container via a chroot, but has the host store available. So it allows arbitrary setup of things outside of the nix store prior to instantiating the container specific store. This is analagous to dockerTools.

If we would like to make the buildImage interface consistent with that of dockTools, which builder are we going to align with? Key difference between Apptainer and Docker, IIUC:

Apptainer/Singularity images does not have layers

Apptainer/Singularity images are usually run immutably, without daemon or process isolation.

Not sure how much isolation affects the process of building an image, more of a runtime thing.

Should we replace buildImage with the definition-based implementation? Should we keep the sandbox-based implementation?

I'm personally optimistic about Apptainer's progress in unprivileged image building. We could hopefully build the image without additional virtualization eventually. Considering the current situation, is there a lighter chroot-like mounter to provide /var/singularity/mnt/{container,final,overlay,session,source}? Should we maintain the manually-image-composition script by @posch (Beat singularity-tools up to shape #177908 (comment))?

One thing to consider is that many places have namespaces disabled so lighter alternatives are not possible. The squashfs approach of @posch is great and wasn't possible when I wrote the initial implementation, but runAsRoot isn't possible with this. This probably doesn't matter for a majority of use cases.

ShamrockLee · 2023-04-05T00:18:30Z

One thing to consider is that many places have namespaces disabled so lighter alternatives are not possible.

What if we have user namespaces? buildFHSUserEnvBubblewrap are common, and there's a TODO by atemu in all-packages.nix about switching to buildFHSUserEnv = buildFHSUserEnvBubblewrap eventually.

posch's script is great, but it's a bit hacky as the neither Apptainer nor SingularityCE have published the specification about the image format. (Hand-made sandbox build also has similar problem.)

SomeoneSerge · 2023-04-05T00:22:04Z

What if we have user namespaces? buildFHSUserEnvBubblewrap are common

(I haven't processed all of the previous notifications, but) I just wanted to note that this isn't too often the case for HPC clusters (Aalto's "Triton" only offers setuid)...

Split the build steps used by buildImage into * buildSandboxFromContents * buildImageFromSandbox * buildImageInLinuxVM

ShamrockLee · 2023-04-05T22:18:00Z

Just noticed that the layout of an Apptainer/Singularity "sandbox" is very similar to the extracted content of a SIF image.

If we could figure out a way to create a sandbox, maybe we could just pack it into a squashfs image.

On the other hand, I'm facing a strange runInLinuxVM bug (#224889) while testing the sandbox production.

SomeoneSerge · 2023-04-06T21:39:30Z

pkgs/build-support/singularity-tools/default.nix

+        "${coreutils}/bin/ln" -s "${coreutils}/bin/env" "''${SINGULARITY_ROOTFS}/usr/bin/env"
+      '';
+      environment = {
+        PATH = "${lib.makeBinPath contents}:\${PATH:-}";


I suppose this is the alternative to the previous approach of generating a FHS-like tree (/bin and such). We can address this in a separate PR, but I wanted to explore (ab)using ext3 and populate both /nix/store and /bin using hardlinks. This way I could hide the container's /nix with --bind /tmp:/nix and still having working sh and nix (assuming I've built them statically linked)

I suppose this is the alternative to the previous approach of generating a FHS-like tree (/bin and such).

Yes. The PATH-based approach seems cleaner for me, which is also how nix shell works. As both of them has its use cases (Nixy idiom vs. FHS PATH compatibility), we could allow users to decide which to use.

We can address this in a separate PR, but I wanted to explore (ab)using ext3 and populate both /nix/store and /bin using hardlinks. This way I could hide the container's /nix with --bind /tmp:/nix and still having working sh and nix (assuming I've built them statically linked)

HPC might be based upon a distributed filesystems; great chance that they don't support hard links. Former versions of Singularity simply extracts the SquashFS image on the host. If the machine have some sort of TMPDIR where local filesystems like ext4 or tmpfs are mounted, you could have it extract there and (hopefully) enjoy the hard links. If you don't have access hard-link supported spaces, there might be problems extracting an image like that.

A good news is that both Apptainer and SingularityCE now tries to mount the image with the squashfuse command from the environment before falling back to extracting, and that SquashFS does support hard links on its own. I added squashfuse to the defaultPath and wrapper PATH of both apptainer and singularity along with the version bump #224683. Let's make it land on the stable release of Nixpkgs.

great chance that they don't support hard links

I guess I don't really care if they're extracted as copies on the host (in fact, only one would be extracted, and the other would get hidden), as long as they are represented as hardlinks inside the image that we create.

I added squashfuse to the defaultPath and wrapper PATH of both apptainer and singularity along with the version bump

Awesome! I should have a look tomorrow!

Let's make it land on the stable release of Nixpkgs

Sounds great! But just in case it takes longer, we can and should backport later

posch · 2023-04-28T19:48:07Z

posch's script is great, but it's a bit hacky as the neither Apptainer nor SingularityCE have published the specification about the image format. (Hand-made sandbox build also has similar problem.)

Today I've tried to use "apptainer build" instead of "mksquashfs" to create the container from the sandbox-like tree. It worked with only a small change to apptainer (see apptainer/apptainer#1312 ). Maybe that's a little less hacky.

ShamrockLee · 2023-04-29T03:21:15Z

It worked with only a small change to apptainer (see apptainer/apptainer#1312 ). Maybe that's a little less hacky.

Great! That would be more likely to be megred by the upstrem than mine (apptainer/apptainer#1284).

ShamrockLee · 2023-08-25T10:20:14Z

During the work on #224828, I found that apptainer run turns out to invoke a non-interctive Bash session, which results in a painful debugging experience. It's caused by the default runscript calling /bin/sh, which links to the non-interactive runtimeShell in images built by singulatity-tools.buildImage.

As most of the distributions (I have heard of) uses an interactive shell as its /bin/sh, would it be better if we link bashInteractive to /bin/sh? Would we care about the increasing closure size?

SomeoneSerge · 2024-10-21T16:05:06Z

Right now it seems like the contents of `.singularity.d/runscript.sh are overwritten by singularity/apptainer

ShamrockLee added 6 commits April 3, 2023 22:17

trivial-builders: add writeMultipleReferencesToFile

341c521

This builder takes a list `path` and reduces the `writeMultipleReferencesToFile` result of each element to a file. It would be helpful when building a container image from a list of packages.

writeScriptBin, writeShellScriptBin: add meta.mainProgram automatically

165335e

singularity-tools: rearrarnge the input parameters

1218d32

singularity-tools: replace mkLayer with writeMultipleReferenceseToFile

a20cda6

singularity-tools.buildImage: unify storeDir access

97eccd8

singularity-tools.buildImage: use .sif instead of .img as image name …

56f66ad

…suffix

ShamrockLee commented Apr 4, 2023

View reviewed changes

pkgs/build-support/singularity-tools/default.nix Outdated Show resolved Hide resolved

ShamrockLee commented Apr 4, 2023

View reviewed changes

ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux labels Apr 4, 2023

ShamrockLee added 4 commits April 6, 2023 02:22

singularity-tools.buildImage: remove gawk from buildInputs

042d2c4

singularity-tools: move defaultSingularity let-in to the top

0771b7e

singularity-tools: split buildImage into reusable parts

36432b8

Split the build steps used by buildImage into * buildSandboxFromContents * buildImageFromSandbox * buildImageInLinuxVM

singularity-tools.buildImageFromDef: init

53a1065

ShamrockLee force-pushed the singularity-tools branch from 5a88f78 to 53a1065 Compare April 5, 2023 18:28

ShamrockLee mentioned this pull request Apr 5, 2023

Building singularity container very slow #171551

Closed

SomeoneSerge reviewed Apr 6, 2023

View reviewed changes

ShamrockLee mentioned this pull request Apr 13, 2023

Beat singularity-tools up to shape #177908

Open

10 tasks

ShamrockLee mentioned this pull request Nov 18, 2023

singularity-tools: refactor and add runImageInLinuxVM; apptainer.passthru.tests.exec-image-in-linux-vm: init #268199

Draft

13 tasks

wegank added 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 2.status: merge conflict This PR has merge conflicts with the target branch labels Mar 19, 2024

stale bot removed the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Mar 20, 2024

wegank added the 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md label Jul 4, 2024

ShamrockLee mentioned this pull request Aug 14, 2024

singularity-tools: miscellaneous fixes (2nd round) #333843

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

singularity-tools: refactor and add definition-based image buider #224636

singularity-tools: refactor and add definition-based image buider #224636

ShamrockLee commented Apr 4, 2023 •

edited

Loading

ShamrockLee commented Apr 4, 2023 •

edited

Loading

ShamrockLee left a comment

jbedo commented Apr 4, 2023

ShamrockLee commented Apr 5, 2023

SomeoneSerge commented Apr 5, 2023 •

edited

Loading

ShamrockLee commented Apr 5, 2023 •

edited

Loading

SomeoneSerge Apr 6, 2023 •

edited

Loading

ShamrockLee Apr 8, 2023

SomeoneSerge Apr 9, 2023 •

edited

Loading

posch commented Apr 28, 2023 •

edited

Loading

ShamrockLee commented Apr 29, 2023 •

edited

Loading

ShamrockLee commented Aug 25, 2023

SomeoneSerge commented Oct 21, 2024

singularity-tools: refactor and add definition-based image buider #224636

Are you sure you want to change the base?

singularity-tools: refactor and add definition-based image buider #224636

Conversation

ShamrockLee commented Apr 4, 2023 • edited Loading

Description of changes

Things done

ShamrockLee commented Apr 4, 2023 • edited Loading

ShamrockLee left a comment

Choose a reason for hiding this comment

jbedo commented Apr 4, 2023

ShamrockLee commented Apr 5, 2023

SomeoneSerge commented Apr 5, 2023 • edited Loading

ShamrockLee commented Apr 5, 2023 • edited Loading

SomeoneSerge Apr 6, 2023 • edited Loading

Choose a reason for hiding this comment

ShamrockLee Apr 8, 2023

Choose a reason for hiding this comment

SomeoneSerge Apr 9, 2023 • edited Loading

Choose a reason for hiding this comment

posch commented Apr 28, 2023 • edited Loading

ShamrockLee commented Apr 29, 2023 • edited Loading

ShamrockLee commented Aug 25, 2023

SomeoneSerge commented Oct 21, 2024

ShamrockLee commented Apr 4, 2023 •

edited

Loading

ShamrockLee commented Apr 4, 2023 •

edited

Loading

SomeoneSerge commented Apr 5, 2023 •

edited

Loading

ShamrockLee commented Apr 5, 2023 •

edited

Loading

SomeoneSerge Apr 6, 2023 •

edited

Loading

SomeoneSerge Apr 9, 2023 •

edited

Loading

posch commented Apr 28, 2023 •

edited

Loading

ShamrockLee commented Apr 29, 2023 •

edited

Loading