Bug: Cannot build a tar layer on top of a squashfs layer

[smoser]

stacker version

v1.0.0-rc4-8e267fc

Describe the bug

Building the following stacker.yaml as root or non-root will fail with the stacktrace below.
The problem is with trying to build a tar layer from a source layer that is only published as squashfs.
You can avoid the problem by using --layer-type=squashfs

stacker.yaml:

base:
  from:
    type: docker
    url: "${{ZOTHUB_BASE:docker://zothub.io}}/machine/bootkit/rootfs:0.0.5.230327-squashfs"
  run: |
    echo hello world

To reproduce

$ stacker --debug build
stacker version v1.0.0-rc4-8e267fc
usernsexec-ing [u 0 1000 1 1 100001 65535 g 0 1000 1 1 100001 65535 -- /usr/local/bin/stacker --internal-userns --debug build]
stacker version v1.0.0-rc4-8e267fc
no previous storage type detected
initializing stacker recipe: stacker.yaml
substituting $STACKER_ROOTFS_DIR to /tmp/roots
substituting $STACKER_STACKER_DIR to /tmp/.stacker
substituting $STACKER_OCI_DIR to /tmp/oci
substituting $STACKER_WORK_DIR to 
stacker build order:
0 build /tmp/stacker.yaml: requires: []
building: 0 /tmp/stacker.yaml
substituting $STACKER_ROOTFS_DIR to /tmp/roots
substituting $STACKER_STACKER_DIR to /tmp/.stacker
substituting $STACKER_OCI_DIR to /tmp/oci
substituting $STACKER_WORK_DIR to 
Dependency Order [base]
preparing image base...
overlay-dirs, possibly modified after import: []
loading docker://zothub.io/machine/bootkit/rootfs:0.0.5.230327-squashfs
Copying blob c2c670f1c1af done  
Copying blob bcd9b263edd3 done  
Copying blob 25c2c00faedd done  
Copying config 3bd19ef6e2 done  
Writing manifest to image destination
Storing signatures
unpacking to /tmp/roots/base
maybeKernelSquashMount(/tmp/.stacker/layer-bases/oci/blobs/sha256/25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d) exited 32: mount: /tmp/roots/sha256_25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d/overlay: failed to setup loop device for /tmp/.stacker/layer-bases/oci/blobs/sha256/25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d.
maybeKernelSquashMount(/tmp/.stacker/layer-bases/oci/blobs/sha256/c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9) exited 32: mount: /tmp/roots/sha256_c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9/overlay: failed to setup loop device for /tmp/.stacker/layer-bases/oci/blobs/sha256/c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9.
Extracting /tmp/.stacker/layer-bases/oci/blobs/sha256/25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d -> /tmp/roots/sha256_25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d/overlay with /usr/bin/squashfuse_ll [/tmp/roots/sha256_25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d/.overlay-squashfuse.log]
maybeKernelSquashMount(/tmp/.stacker/layer-bases/oci/blobs/sha256/bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad) exited 32: mount: /tmp/roots/sha256_bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad/overlay: failed to setup loop device for /tmp/.stacker/layer-bases/oci/blobs/sha256/bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad.
Extracting /tmp/.stacker/layer-bases/oci/blobs/sha256/c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9 -> /tmp/roots/sha256_c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9/overlay with /usr/bin/squashfuse_ll [/tmp/roots/sha256_c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9/.overlay-squashfuse.log]
Extracting /tmp/.stacker/layer-bases/oci/blobs/sha256/bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad -> /tmp/roots/sha256_bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad/overlay with /usr/bin/squashfuse_ll [/tmp/roots/sha256_bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad/.overlay-squashfuse.log]
lxc rootfs overlay arg overlayfs:/tmp/roots/sha256_c2c670f1c1af96cab5865c25ba566095513887b1d2be75375930cc7b553465c9/overlay:/tmp/roots/sha256_bcd9b263edd3ed4c6c7f5f5e91bcfcc4887c65c6570ccd63d0fee59484fde4ad/overlay:/tmp/roots/sha256_25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d/overlay:/tmp/roots/base/overlay
stacker version v1.0.0-rc4-8e267fc
stacker subcommand: [/usr/local/bin/stacker --oci-dir /tmp/oci --roots-dir /tmp/roots --stacker-dir /tmp/.stacker --storage-type overlay --internal-userns --debug internal-go check-aa-profile lxc-container-default-cgns]
bind mounting /tmp/.stacker/imports/base into container
+ echo hello world
hello world
converting between {squashfs true} and {tar false}
error: chmod /tmp/roots/sha256_25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d/overlay: function not implemented
chmod +r
github.com/opencontainers/umoci/pkg/unpriv.Open.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:134
github.com/opencontainers/umoci/pkg/unpriv.Wrap
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:75
github.com/opencontainers/umoci/pkg/unpriv.Open
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:125
github.com/opencontainers/umoci/pkg/unpriv.foreachSubpath
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:318
github.com/opencontainers/umoci/pkg/unpriv.walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:541
github.com/opencontainers/umoci/pkg/unpriv.Walk.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:574
github.com/opencontainers/umoci/pkg/unpriv.Wrap
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:75
github.com/opencontainers/umoci/pkg/unpriv.Walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:569
github.com/opencontainers/umoci/oci/layer.GenerateInsertLayer.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/layer/generate.go:153
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598
unpriv.open
github.com/opencontainers/umoci/pkg/unpriv.Open
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:142
github.com/opencontainers/umoci/pkg/unpriv.foreachSubpath
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:318
github.com/opencontainers/umoci/pkg/unpriv.walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:541
github.com/opencontainers/umoci/pkg/unpriv.Walk.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:574
github.com/opencontainers/umoci/pkg/unpriv.Wrap
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:75
github.com/opencontainers/umoci/pkg/unpriv.Walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:569
github.com/opencontainers/umoci/oci/layer.GenerateInsertLayer.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/layer/generate.go:153
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598
github.com/opencontainers/umoci/pkg/unpriv.foreachSubpath
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:320
github.com/opencontainers/umoci/pkg/unpriv.walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:541
github.com/opencontainers/umoci/pkg/unpriv.Walk.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:574
github.com/opencontainers/umoci/pkg/unpriv.Wrap
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:75
github.com/opencontainers/umoci/pkg/unpriv.Walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:569
github.com/opencontainers/umoci/oci/layer.GenerateInsertLayer.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/layer/generate.go:153
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598
unpriv.walk
github.com/opencontainers/umoci/pkg/unpriv.Walk.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:579
github.com/opencontainers/umoci/pkg/unpriv.Wrap
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:75
github.com/opencontainers/umoci/pkg/unpriv.Walk
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/pkg/unpriv/unpriv.go:569
github.com/opencontainers/umoci/oci/layer.GenerateInsertLayer.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/layer/generate.go:153
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598
generate insert layer
github.com/opencontainers/umoci/oci/layer.GenerateInsertLayer.func1.1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/layer/generate.go:140
github.com/opencontainers/umoci/oci/layer.GenerateInsertLayer.func1
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/layer/generate.go:153
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598
copy to temporary blob
github.com/opencontainers/umoci/oci/cas/dir.(*dirEngine).PutBlob
	/stacker-tree/.build/gopath/pkg/mod/github.com/project-stacker/umoci@v0.0.0-20230130205906-2f7d2b39ff9f/oci/cas/dir/dir.go:173
stackerbuild.io/stacker/pkg/overlay.ociPutBlob
	/stacker-tree/pkg/overlay/pack.go:301
stackerbuild.io/stacker/pkg/overlay.ConvertAndOutput
	/stacker-tree/pkg/overlay/pack.go:152
stackerbuild.io/stacker/pkg/overlay.(*overlay).initializeBasesInOutput
	/stacker-tree/pkg/overlay/pack.go:236
stackerbuild.io/stacker/pkg/overlay.(*overlay).Repack
	/stacker-tree/pkg/overlay/pack.go:266
stackerbuild.io/stacker/pkg/stacker.(*Builder).build
	/stacker-tree/pkg/stacker/build.go:493
stackerbuild.io/stacker/pkg/stacker.(*Builder).BuildMultiple
	/stacker-tree/pkg/stacker/build.go:568
main.doBuild
	/stacker-tree/cmd/stacker/build.go:117
github.com/urfave/cli.HandleAction
	/stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli@v1.22.12/app.go:524
github.com/urfave/cli.Command.Run
	/stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli@v1.22.12/command.go:175
github.com/urfave/cli.(*App).Run
	/stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli@v1.22.12/app.go:277
main.main
	/stacker-tree/cmd/stacker/main.go:324
runtime.main
	/usr/lib/go/src/runtime/proc.go:250
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598
error: exit status 1
stackerbuild.io/stacker/pkg/container.MaybeRunInNamespace
	/stacker-tree/pkg/container/userns.go:102
main.main.func3
	/stacker-tree/cmd/stacker/main.go:319
github.com/urfave/cli.(*App).Run
	/stacker-tree/.build/gopath/pkg/mod/github.com/urfave/cli@v1.22.12/app.go:264
main.main
	/stacker-tree/cmd/stacker/main.go:324
runtime.main
	/usr/lib/go/src/runtime/proc.go:250
runtime.goexit
	/usr/lib/go/src/runtime/asm_amd64.s:1598

Expected behavior

No response

Additional context

No response

[hallyn]

Honestly this makes no sense:

ociPutBlob: called on types.StackerConfig{WorkDir:"", StackerDir:"/home/serge/sandbox/smoser4/.stacker", OCIDir:"/home/serge/sandbox/smoser4/oci", RootFSDir:"/home/serge/sandbox/smoser4/roots", Debug:false, StorageType:"overlay", EmbeddedFS:embed.FS{files:(*[]embed.file)(0x3482da0)}}
ociPutBlob: opened layout "/home/serge/sandbox/smoser4/oci"
ociPutBlob: calling putblob on &io.PipeReader{p:(*io.pipe)(0xc000701f80)}
error: copy to temporary blob: generate layer: unpriv.walk: unpriv.open: chmod +r: chmod /home/serge/sandbox/smoser4/roots/sha256_25c2c00faedd2c307ea94c1146337e8aabd53edc3085b0f26486f955ede5eb4d/overlay: function not implemented

We are opening the ${topdir}/oci as ocidir in umoci, then calling oci.PubBlob(), but it ends up trying to chown ${RootfsDir}/sha256_{blob}

[hallyn]

Ok, yeah, so the problem is in pkg/overlay/pack.go:generateBlob(), we take in the target layerType. If we are building a tar layer, then we call umoci/oci/layer.GenerateInsertLayer(), if squash then mksquashfs. But if we are building a tar layer from a squashfs source layer, then the overlay/ dir where we start is a mountpoint. Now umoci tries to chmod the parent dir +r, which fails bc it's a mountpoint.

This only happens when we are re-building an intermediate layer - one which we already have as squash layer, but are now generating a tar layer for.

Easiest is probably to detect the situation (using IsMountpoint()) and then convert slightly differently.

[hallyn]

Actually hallyn/umoci@922fdb3 fixes it for me