Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lchown failures on img pull when running on k8s #40

Open
jbeda opened this issue Feb 24, 2018 · 12 comments
Open

lchown failures on img pull when running on k8s #40

jbeda opened this issue Feb 24, 2018 · 12 comments
Labels

Comments

@jbeda
Copy link

jbeda commented Feb 24, 2018

This may be an issue with my setup so all bets are off...

My setup:

  • Ubuntu 16.04 cluster set up via kubeadm (via Heptio AWS quickstart)[https://aws.amazon.com/quickstart/architecture/heptio-kubernetes/]
  • echo 1 > /proc/sys/kernel/unprivileged_userns_clone run on each node (not sure if it is necessary).

YAML file for workload:

apiVersion: v1
kind: Pod
metadata:
  labels:
    run: img
  name: img
  annotations:
    container.apparmor.security.beta.kubernetes.io/img: unconfined
spec:
  containers:
  - image: r.j3ss.co/img
    imagePullPolicy: IfNotPresent
    name: img
    resources: {}
    command:
    - sleep
    - "600"
  restartPolicy: Never

Then:

workstation$ kubectl exec -t -i img -- ash
container# unshare -m -U --map-root-user
ns# img pull alpine
Pulling alpine...
INFO[0000] resolving docker.io/library/alpine:latest
INFO[0001] unpacking docker.io/library/alpine:latest
INFO[0001] Apply failure, attempting cleanup             error="mount callback failed on /tmp/containerd-mount735446913: lchown /tmp/containerd-mount735446913/etc/shadow: invalid argument" key="extract-337652914-e-vv sha256:cd7100a72410606589a54b932cabd804a17f9ae5b42a1882bd56d263e02b6215"
failed to extract layer sha256:cd7100a72410606589a54b932cabd804a17f9ae5b42a1882bd56d263e02b6215: mount callback failed on /tmp/containerd-mount735446913: lchown /tmp/containerd-mount735446913/etc/shadow: invalid argument

I haven't had time to debug further.

@AkihiroSuda
Copy link
Collaborator

We need to wait for opencontainers/runc#1693 or use ptrace hack.

Please refer to README for further info

@jbeda
Copy link
Author

jbeda commented Feb 24, 2018

Ah! I asssumed that r.j3ss.co/img already had a hacked runc in there to work around the setgroup issue.

@jessfraz
Copy link
Collaborator

jessfraz commented Feb 24, 2018 via email

@jessfraz
Copy link
Collaborator

We actually not with that one with the other one... I'll do them all tho later

@AkihiroSuda
Copy link
Collaborator

@AkihiroSuda
Copy link
Collaborator

AkihiroSuda commented Feb 25, 2018

or this tag: https://github.com/AkihiroSuda/runc/tree/demo-rootless.20180116-0
(commits are same; using tag rather than branch would be more deterministic)

@jbeda
Copy link
Author

jbeda commented Feb 25, 2018

We actually not with that one with the other one... I'll do them all tho later

I'm not sure which one you are talking about... but I just tried with your new image and I'm seeing the same behavior. I'm a bit confused still about what patches y'all are talking about and the necessary setup for those to work. But I'll let y'all figure this out. @jessfraz if this is being tracked in a different place feel free to close this issue.

@jessfraz
Copy link
Collaborator

I'm going to make some k8s examples and carry the patches on my flight this afternoon, sorry was doing saturday things :)

@jessfraz
Copy link
Collaborator

then we can work out all the rough edges, I also hope that then we can help with them testing this upstream (re comment here: moby/moby#32925 (comment)) I know the last time the builder in docker was replaced we had to test a lot to find the weird bashisms that broke and odd bugs in the old builder that were being used as features, etc

@jessfraz jessfraz added the bug label Mar 6, 2018
@AkihiroSuda
Copy link
Collaborator

seems closable now?

@ccremer
Copy link

ccremer commented Mar 13, 2020

I'm having a similar issue (Operation not permitted), but only when trying to use a different state directory (need build cache with PVC):

apiVersion: batch/v1
kind: Job
metadata:
  name: img-build
  namespace: amze-2029
spec:
  template:
    metadata:
      annotations:
        container.apparmor.security.beta.kubernetes.io/build: unconfined
        container.seccomp.security.alpha.kubernetes.io/build: unconfined
    spec:
      initContainers:
      - name: git-clone
        image: r.j3ss.co/img:v0.5.7
        command:
        - /bin/sh
        args:
        - -c
        - git clone https://github.com/amazeeio/drupal-example.git
        workingDir: /home/user/src
        volumeMounts:
        - mountPath: /home/user/src
          name: src
      containers:
      - name: build
        image: r.j3ss.co/img:v0.5.7
        command:
        - /bin/sh
        args:
        - -c
        - >
          id &&
          echo "RUN apk add --no-cache tcpdump" >> drupal-example/lagoon/php.dockerfile &&
          img build -s /tmp -t registry.ch-gva-2.exo.appuio.ch/amze-2029/drupal-example:latest -f drupal-example/lagoon/php.dockerfile
          --build-arg CLI_IMAGE=registry.ch-gva-2.exo.appuio.ch/amze-2029/drupal-example:cli drupal-example &&
          img push -s /tmp registry.ch-gva-2.exo.appuio.ch/amze-2029/drupal-example:latest
        securityContext:
        #   privileged: true
          #runAsUser: 1000
          #runAsGroup: 0
          procMount: Unmasked
        workingDir: /home/user/src
        volumeMounts:
        - mountPath: /home/user/src
          name: src
        - mountPath: /tmp
          name: cache
        - mountPath: /home/user/.docker/config.json
          subPath: config.json
          name: registries
      restartPolicy: Never
      volumes:
      - name: src
        emptyDir: {}
      - name: cache
        persistentVolumeClaim:
          claimName: img-cache
      - name: registries
        secret:
          secretName: regcred
          items:
          - key: .dockerconfigjson
            path: config.json
  backoffLimit: 0

log excerpt:

#7 sha256:831a6d750410aeacea063fbe7db0e87c626b3019ae8f56be0c9aefd3d6b8af76 14.71MB / 14.71MB done

#7 unpacking registry.ch-gva-2.exo.appuio.ch/amze-2029/drupal-example:cli@sha256:14630f6378aa48eb2e32bcdffafc5c6eec8d49dca50d4b50f89cd61db8c40195

time="2020-03-12T18:35:32Z" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:77cae8ab23bf486355d1b3191259705374f4a11d483b24964d2f729dd8c076a0: mount callback failed on /run/user/1000/containerd-mount827883887: lchown /run/user/1000/containerd-mount827883887/etc/shadow: operation not permitted" key="extract-844920920-8DtT sha256:77cae8ab23bf486355d1b3191259705374f4a11d483b24964d2f729dd8c076a0"

time="2020-03-12T18:35:32Z" level=info msg="apply failure, attempting cleanup" error="failed to extract layer sha256:77cae8ab23bf486355d1b3191259705374f4a11d483b24964d2f729dd8c076a0: mount callback failed on /run/user/1000/containerd-mount475060601: context canceled" key="extract-88219689-NE5W sha256:77cae8ab23bf486355d1b3191259705374f4a11d483b24964d2f729dd8c076a0"

#6 ...

I started trying different parameters (was using /cache before /tmp) and also played with the securityContext user/group ids, but to no avail.
Kubernetes is v1.14.9-eks-502bfb

@exherb
Copy link

exherb commented Mar 15, 2020

some issue here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants