make KanikoDir a configuration option

[ulrichSchreiner]

At the moment the directory /kaniko is hard coded in constants.go as

KanikoDir = "/kaniko"

kaniko stores snapshots in this directory. this is a problem, if the volume storage is limited in the build environment. we use gitlab-ci in a k8s environment and connect the default /builds directory to a ephemeral volume, so when running a ci job, the build directory has >10GB storage, but the other directories are normal directories in the job-pod and so they use the disk storage of the k8s-nodes.

when building big images (and also have a lot of parallel jobs running), we sometimes have no space left on device because the snapshot in /kaniko is to big.

It would be great to have a configuration option, so kaniko stores the snapshots in another directory (/builds in our case) so we can use the mounted ephemeral storage. i tried the --tarPath option but it looks like that in addition to the tar, the snapshots are also written to /kaniko.

in #1881 it was also mentioned to make this directory configurable, but the whole issue has another aspect so i created a new issue.

[very-doge-wow]

This would also be a needed feature in order to make kaniko work on clusters with limited access (for example when containers are always started with limited read-only access for the root dirs such as /kaniko) as mentioned in #1363.

[imjasonh]

This would also be a needed feature in order to make kaniko work on clusters with limited access (for example when containers are always started with limited read-only access for the root dirs such as /kaniko) as mentioned in #1363.

Yep! This would be a really useful feature, and if there's anybody interested in working on it, I'd be happy to help however I can.

[jdockerty]

I'd like to take a shot at this, please! @tejal29

@imjasonh I hope you don't mind if I ping you if I get blocked on it for some pointers 😄

[imjasonh]

I'd like to take a shot at this, please! @tejal29

@imjasonh I hope you don't mind if I ping you if I get blocked on it for some pointers 😄

Sounds great! Thanks!

[cmdjulian]

Hey guys, I was thrilled to see this feature merged. Thanks for the work to both of you, I'm really was really looking forward for that feature. Maybe you could help me out a bit.
I'm trying to run the kaniko container now with a read only fs with:

docker run --rm --name kaniko \
  --memory=2G \
  --read-only \
  --cap-drop=all \
  --cap-add=chown --cap-add=fowner --cap-add=setgid --cap-add=setuid --cap-add=dac_override \
  -v "$(pwd)/Dockerfile:/workspace/Dockerfile:ro" \
  -v "$(pwd)/requirements.txt:/kaniko/requirements.txt:ro" \
  --mount type=tmpfs,destination=/tmp \
  gcr.io/kaniko-project/executor:v1.8.1 \
  --dockerfile /workspace/Dockerfile \
  --context dir:///workspace/ \
  --log-format=color \
  --log-timestamp=true \
  --kaniko-dir /tmp \
  --no-push

My requiremnts.txt should not be part of the final image and is therefore mounted inside of the kaniko dir. However trying to run the above docker command yields: Error: error resolving dockerfile path: copying dockerfile: creating file: open /kaniko/Dockerfile: read-only file system.
When omitting the -read-onlyflag it works. When I check the image I see, that kaniko moves the Dockerfile to /kaniko and also downloads the base images to /kaniko. To me it seems like the -kaniko-dir option is not taken into account. When using the env, I get a different error message when I run:

docker run --rm --name kaniko \
  --memory=2G \
  --cap-drop=all \
  --cap-add=chown --cap-add=fowner --cap-add=setgid --cap-add=setuid --cap-add=dac_override \
  -v "$(pwd)/Dockerfile:/workspace/Dockerfile:ro" \
  -v "$(pwd)/requirements.txt:/kaniko/requirements.txt:ro" \
  -e "KANIKO_DIR=/not-kaniko/" \
  gcr.io/kaniko-project/executor:v1.8.1 \
  --dockerfile /workspace/Dockerfile \
  --context dir:///workspace/ \
  --log-format=color \
  --log-timestamp=true \
  --no-push

Error: rename /kaniko /not-kaniko: file exists. It doesn't matter which path I'm choosing, existent or non-existent I always see that error.

My goal is to make the root fs read only and only allow a certain path for kaniko to run on, so the final docker run command should look something like that, mind the tmpfs mount:

docker run --rm --name kaniko \
  --memory=2G \
  --read-only \
  --cap-drop=all \
  --cap-add=chown --cap-add=fowner --cap-add=setgid --cap-add=setuid --cap-add=dac_override \
  -v "$(pwd)/Dockerfile:/workspace/Dockerfile:ro" \
  -v "$(pwd)/requirements.txt:/kaniko/requirements.txt:ro" \
  --mount type=tmpfs,destination=/tmp \
  -e "KANIKO_DIR=/not-kaniko" \
  gcr.io/kaniko-project/executor:v1.8.1 \
  --dockerfile /workspace/Dockerfile \
  --context dir:///workspace/ \
  --log-format=color \
  --log-timestamp=true \
  --no-push

This unfortunately yields the same error as above. Any ideas what I might do wrong?

Additional Context:
Dockerfile:

FROM python:3.7-slim
RUN useradd --uid=1000 --user-group --home-dir=/home/worker --create-home worker
USER 1000:1000
RUN mkdir -p /home/worker/script /home/worker/results /home/worker/tmp
WORKDIR /home/worker/script
RUN pip install --user --no-cache-dir --no-warn-script-location --disable-pip-version-check -r /kaniko/requirements.txt

requirements.txt:

keras==2.4.3
pandas
keras_pickle_wrapper
sklearn
tensorflow==2.2.0

Thanks in advance.

[Quad-Plex]

Error: rename /kaniko /not-kaniko: file exists. It doesn't matter which path I'm choosing, existent or non-existent I always see that error.

I am seeing the same thing, trying to run the v1.8.1-debug image in a gitlab-ci environment. It's a very simple setup, just the kaniko image as default and a KANIKO_DIR env variable. It doesn't seem to matter which path I choose, everything comes back as file exists.

[mtz29]

I am seeing the same thing, trying to run the v1.8.1-debug image in a gitlab-ci environment. It's a very simple setup, just the kaniko image as default and a KANIKO_DIR env variable. It doesn't seem to matter which path I choose, everything comes back as file exists.

The reason for that is due to the fact kaniko is trying to rename /kaniko directory. If the path is mounted or there's no permissions it fails.

https://github.com/GoogleContainerTools/kaniko/blob/main/cmd/executor/cmd/root.go#L244L253

I have no idea what's the idea behind that.