Actual behavior
When WORKDIR is called on a non-existent directory, kaniko is kind enough to create that directory for you, resulting in a layer being added. However, kaniko does not cache that layer, which means that on every invocation a completely new image is emitted from that point onwards. Inside the same stage this is non-obvious as caching mechanism still pulls, so you get a 100% cache hitrate thereafter, but the image is completely new. In multistage builds or builds that depend on the newly emitted image, this is catastrophic, as they do consider the entire image's sha when determining whether a cache is hit or not, so this will invalidate the entire cache.

Workaround is simple but silly, we just need to make sure that the directory exists before calling WORKDIR

RUN mkdir /app
WORKDIR /app

Expected behavior

When WORKDIR is called on a non-existent directory, that directory is implicitly created and the resulting layer is pushed into cache.

To Reproduce

FROM ubuntu:latest
WORKDIR /app
RUN echo "whatever"

run this build multiple times, with cache, and observe that the layers change every time

docker inspect <image>

Triage Notes for the Maintainers