snapshotter doesn't like paths in ko images

[mattmoor]

In github.com/google/ko built images, the paths we use within the tarball seem to be causing problems for the estargz snapshotter. These images run fine against a standard containerd, so it would be good to harden this snapshotter to support them:

Dec 20 21:36:45 ubuntu containerd-stargz-grpc[1179]: {"error":"failed to cache prefetched layer: invalid child path \"/ko-app\"; must be child of \"\"","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/28/fs","msg":"failed to prefetched layer","time":"2020-12-20T21:36:45.919228015Z"}
...
Dec 20 21:36:52 ubuntu containerd-stargz-grpc[1179]: {"error":"invalid child path \"/var\"; must be child of \"\"","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/27/fs","msg":"failed to fetch whole layer","time":"2020-12-20T21:36:52.422678402Z"}

You can produce an image with ko + estargz by:

1. clone github.com/google/ko
2. go install ./cmd/ko
3. GGCR_EXPERIMENT_ESTARGZ=1 KO_DOCKER_REPO=docker.io/{username} ko publish -B ./cmd.foo
4. kubectl run foo --image={digest from above}

[mattmoor]

It looks like the distroless images (run through crane optimize) exhibit a similar error:

Dec 20 22:04:02 ubuntu containerd-stargz-grpc[1179]: {"error":"invalid child path \"/var\"; must be child of \"\"","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/46/fs","msg":"failed to fetch whole layer","time":"2020-12-20T22:04:02.521706097Z"}

[mattmoor]

If I manually force ko to use relative paths for files within the tarball then it seems to stop complaining about that layer, but this is likely going to be a problem for a whole class of images, so it'd be best to try and fix this.

[mattmoor]

cc @itsmurugappan @ktock @AkihiroSuda

[ktock]

@mattmoor Thanks for reporting this.
There seems several issues for lazy pulling distroless images. I'll fix them today.

• Issue 1 : ko's default base images aren't eStargz => solvable by using eStargz-formatted base images
• Issue 2 (estargz: Clean entry names when read and unmarshal blob #218): Stargz snashotter's prefetching & background fetching doesn't support absolute path
• Issue 3 (estargz: Support duplicated entries during sort #219): There is a distroless image that cannot be optimized

Issue 1: ko's default base images aren't eStargz

Currently, containerd doesn't allow the snapshotter to use lazy layer stacked on non-lazy layers.
So image provided by ko with default (non-estargz) base images cannot be lazily pulled.

git clone https://github.com/google/ko
go install ./cmd/ko
GGCR_EXPERIMENT_ESTARGZ=1 KO_DOCKER_REPO=registry2:5000/demo ko publish --insecure-registry -B ./cmd/ko

sha256:643df82f335953adb1ab8c7fc37f424572142eb2b6da2c3ee29964ccca6d39d2 isn't eStargz.

crane manifest --insecure registry2:5000/demo/ko@sha256:f1146ac8be267b8724eeb8d60ce9aa68da64db9bc8af1ab9f704d669a286c342 | jq 
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 1009,
    "digest": "sha256:643df82f335953adb1ab8c7fc37f424572142eb2b6da2c3ee29964ccca6d39d2"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 641550,
      "digest": "sha256:e59bd8947ac7af2c8f4403b183326986ab554bbc262739cf5d9d596c7c7a3aca"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 604,
      "digest": "sha256:e1af7412b94538289c65daaf5990ea8fe8b79ec37ca909bfe024a9e6da67a2ed",
      "annotations": {
        "containerd.io/snapshot/stargz/toc.digest": "sha256:8565f82ef8ef48132b96f20ebd2d569286962650da5f8a688c599a2441e21776"
      }
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 22754507,
      "digest": "sha256:6607eb8a25887d318ad5b0756b4b75ed2e7b0c2712edc1272e91513fc89c8713",
      "annotations": {
        "containerd.io/snapshot/stargz/toc.digest": "sha256:f776c40c43ef6e612edd05841feb38d0957ce6b7056a15b5e085445a93dec5da"
      }
    }
  ]
}

This can be solved by using eStargz base image.

echo "defaultBaseImage: ghcr.io/stargz-containers/distroless-static:nonroot-esgz" > /tmp/koconf/.ko.yaml
GGCR_EXPERIMENT_ESTARGZ=1 KO_CONFIG_PATH=/tmp/koconf/ KO_DOCKER_REPO=registry2:5000/demo ko publish --insecure-registry -B ./cmd/ko

crane manifest --insecure registry2:5000/demo/ko@sha256:0053c914ff2cf7412ec50a4fd1c0bca4eb0961111e51f2d457c22f72e44d3e77 | jq
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 961,
    "digest": "sha256:5ef5237e7f4e4389874d01da9b37b393a4eb934c8a1174511d0d5c1d798cea44"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1093216,
      "digest": "sha256:d0d627437df53d07d2bc1ab1be3b8ce87939285f32f34ea9d586cdb25c2afab1",
      "annotations": {
        "containerd.io/snapshot/stargz/toc.digest": "sha256:d2cdc477fbf26b7686541e58871c7622dc2c6c39e8b59bd786c8c7b3fd64da2f"
      }
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 604,
      "digest": "sha256:e1af7412b94538289c65daaf5990ea8fe8b79ec37ca909bfe024a9e6da67a2ed",
      "annotations": {
        "containerd.io/snapshot/stargz/toc.digest": "sha256:8565f82ef8ef48132b96f20ebd2d569286962650da5f8a688c599a2441e21776"
      }
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 22754507,
      "digest": "sha256:6607eb8a25887d318ad5b0756b4b75ed2e7b0c2712edc1272e91513fc89c8713",
      "annotations": {
        "containerd.io/snapshot/stargz/toc.digest": "sha256:f776c40c43ef6e612edd05841feb38d0957ce6b7056a15b5e085445a93dec5da"
      }
    }
  ]
}

Issue2: Stargz snashotter's prefetching & background fetching doesn't support absolute path

Prefetching & background fetching images provided by ko fail because the image contains absolute path.
We should fix the snapshotter to support absolute path.

ctr-remote i rpull --plain-http registry2:5000/demo/ko@sha256:0053c914ff2cf7412ec50a4fd1c0bca4eb0961111e51f2d457c22f72e44d3e77
... (omit) ...
{"error":"invalid child path \"/ko-app\"; must be child of \"\"","level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/7/fs","msg":"failed to fetch whole layer","time":"2020-12-21T01:12:11.932087841Z"}

$ crane blob --insecure registry2:5000/demo/ko@sha256:6607eb8a25887d318ad5b0756b4b75ed2e7b0c2712edc1272e91513fc89c8713 | tar -z --list
tar: Removing leading `/' from member names
/ko-app/ko
.prefetch.landmark
/ko-app
stargz.index.json

Issue3: There is a distroless image that cannot be optimized

There are duplicated entries /usr and ./usr that isn't currently allowed by estargz lib.
We should allow duplicate entries by overriding instead of returning error.

$ ctr-remote i optimize --no-optimize --plain-http gcr.io/distroless/python2.7 http://registry2:5000/demo/distroless-python:1
WARN[0000] 2020/12/21 00:45:24 No matching credentials were found, falling back on anonymous 
WARN[0001] Platform mismatch or optimization disabled; converting without optimization  platform=linux/amd64
ctr: failed to convert layer to stargz: failed to sort: Duplicated entry("./usr/") is not supported

$ crane blob gcr.io/distroless/python2.7@sha256:71f896c24ff1d7bcf234cda2d588ca1dba02ef6d3d1797392c0bcbe3fbd1f3cb | tar -z --list | grep usr
tar: Removing leading `/' from member names
/usr/
/usr/bin/
/usr/bin/python
./usr/
./usr/share/
./usr/share/doc/
./usr/share/doc/dash/

[mattmoor]

Ack issue #1, but isn't the lazy loading per-layer?

[ktock]

From snapshotter's perspective, yes it is.
But from containerd's perspective, it queries chains (snapshots; stacked view of layers) to the snapshotter.
When containerd gets non-lazy chain, it immediately starts to pull all upper layers.

Docs about snapshots:

• Snapshots
• Remote Snapshotter

[mattmoor]

Yeah, I am familiar. My point is that a layer can still be lazy even if the upper layers are pulled eagerly.

[ktock]

a layer can still be lazy even if the upper layers are pulled eagerly.

Yes, I think it's theoretically possible.

[ktock]

All issues are fixed (#218, #219) and now image produced by ko can be lazily pulled.
@mattmoor Could you try the latest commit on your environment?

# echo "defaultBaseImage: ghcr.io/stargz-containers/distroless-static:nonroot-esgz" > /tmp/koconf/.ko.yaml
# GGCR_EXPERIMENT_ESTARGZ=1 KO_CONFIG_PATH=/tmp/koconf/ KO_DOCKER_REPO=registry2:5000/demo ko publish --insecure-registry -B ./cmd/ko
# ctr-remote i rpull --plain-http registry2:5000/demo/ko@sha256:fa55570e6376057ce68d1226055d695ff26ff57890f90e2be90f6b05a2893129
... (omit) ...
{"level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/1/fs","msg":"completed to prefetch","time":"2020-12-22T13:06:25.043179837Z"}
{"level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/2/fs","msg":"completed to prefetch","time":"2020-12-22T13:06:25.315508487Z"}
{"level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/3/fs","msg":"completed to prefetch","time":"2020-12-22T13:06:25.853265661Z"}
{"level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/2/fs","msg":"completed to fetch all layer data in background","time":"2020-12-22T13:06:30.854436638Z"}
{"level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/3/fs","msg":"completed to fetch all layer data in background","time":"2020-12-22T13:06:30.854708019Z"}
{"level":"debug","mountpoint":"/var/lib/containerd-stargz-grpc/snapshotter/snapshots/1/fs","msg":"completed to fetch all layer data in background","time":"2020-12-22T13:06:31.069902797Z"}
... (omit) ...

[mattmoor]

I'll sync with @itsmurugappan and try to repro this today

[itsmurugappan]

sure, will give it a try today

[ktock]

@mattmoor @itsmurugappan Is this still an issue?

[mattmoor]

I haven't checked the logs recently, but I do not think this is a problem anymore.

[ktock]

Closing this. Feel free to reopen this whenever issues related to this are found.