kaniko does not honor copy regex

[flixpae]

Actual behavior
A clear and concise description of what the bug is.

I am in the process of integrating kaniko in a custom image for use in gitlab-ci

I tried to exclude contents with a filepath.Match pattern as stated in the docker docs
https://docs.docker.com/engine/reference/builder/#copy

But Kaniko copys all contents regardless of the preceding match pattern

Expected behavior
A clear and concise description of what you expected to happen.

Kaniko should honor the supplied match pattern as "docker build" does.

To Reproduce
Steps to reproduce the behavior:

1. create a Dockerfile with

  COPY --from=gcr.io/kaniko-project/executor:v0.11.0 ./kaniko/[^.]* /kaniko/

2. run docker build on the Dockerfile and check if files are excluded
3. run kaniko executor on the same Dockerfile and see if files are excluded

Additional Information

• Dockerfile
  Please provide either the Dockerfile you're trying to build or one that can reproduce this error.
• Build Context
  Please provide or clearly describe any files needed to build the Dockerfile (ADD/COPY commands)
• Kaniko Image : docker inspect gcr.io/kaniko-project/executor:v0.11.0

[
    {
        "Id": "sha256:acfda54308fba3edd86d587ffdc5b7b31ae4b2205d66d2065da3e640762de37c",
        "RepoTags": [
            "gcr.io/kaniko-project/executor:v0.11.0"
        ],
        "RepoDigests": [
            "gcr.io/kaniko-project/executor@sha256:584a8d90679211d9b09465d778990ec15965cf78f57f197e973d57d14b08eb81"], 

[tejal29]

@flixpae Currently kaniko does not support reg-ex in copy command.
We welcome any PRs to support this. You would need to add this here
https://github.com/GoogleContainerTools/kaniko/blob/master/pkg/commands/copy.go#L54

[sushant30]

This issue is still not fixed. Any timeline around this?

[JeromeJu]

After some investigation, seems the COPY command does accept wildcards with the mathSources in command_util; similarly for ADD. The wildcard implementations shall precede this issue was raised. Looking into the reason why this actually failed, but with v1.16.0 image.

[JeromeJu]

± |main U:3 ?:2 ✗| → ./run_in_docker.sh /DOCKERFILE /Users/jeromeju/Desktop/kaniko gcr.io/jju-dev/test:latest
INFO[0000] Using dockerignore file: /workspace/.dockerignore 
INFO[0000] Retrieving image manifest gcr.io/kaniko-project/executor:latest 
INFO[0000] Retrieving image gcr.io/kaniko-project/executor:latest from registry gcr.io 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest gcr.io/kaniko-project/executor:latest 
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'gcr.io/kaniko-project/executor:latest' [idx: '0', base-idx: '-1'] 
INFO[0001] Unpacking rootfs as cmd COPY /example?/ /kaniko/ requires it. 
INFO[0005] Resolving srcs [/example?/]...               
INFO[0019] COPY /example?/ /kaniko/examples/                     
INFO[0019] Resolving srcs [/example?/]...               
INFO[0021] Taking snapshot of files...                  
INFO[0021] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0021] Pushing image to gcr.io/jju-dev/test:latest  
INFO[0024] Pushed gcr.io/jju-dev/test@sha256:901423a1551a351387fbf53b34c8f0df40f8548a4cfa0c919c529a41a065f03c 

Also tried * and []. Looks like this shall work with what's supported with filepath.Match(), which are {?*[]} according to the documentation provided https://docs.docker.com/engine/reference/builder/#copy. I updated the docString respectively.

[JeromeJu]

Suggesting this could be closed for now with the support of {?*[} which corresponds to the docker build matching patterns.

[aaron-prindle]

Just validated that this functionality indeed works @ HEAD using the repro above:

Repro Dockerfile + file tree:

aprindle@aprindle-ssd ~/kaniko-ctx-dir tree
.
├── Dockerfile
└── files
    └── include_me

2 directories, 2 files
aprindle@aprindle-ssd ~/kaniko-ctx-dir cat Dockerfile; echo
# Use a base image
FROM debian:buster-slim AS base

RUN ls -a /workspace/files
COPY ./files/[^.]* /

aprindle@aprindle-ssd ~/kaniko  [release-automation]./run_in_docker.sh /workspace/Dockerfile /usr/local/google/home/aprindle/kaniko-ctx-dir gcr.io/aprindle-test-cluster/kaniko-test:latest false |& tee ~/out.txt
INFO[0000] Resolved base name debian:buster-slim to base 
INFO[0000] Retrieving image manifest debian:buster-slim 
INFO[0000] Retrieving image debian:buster-slim from registry index.docker.io 
INFO[0001] Built cross stage deps: map[]                
INFO[0001] Retrieving image manifest debian:buster-slim 
INFO[0001] Returning cached image manifest              
INFO[0001] Executing 0 build triggers                   
INFO[0001] Building stage 'debian:buster-slim' [idx: '0', base-idx: '-1'] 
INFO[0001] Unpacking rootfs as cmd RUN ls -a /workspace/files requires it. 
INFO[0002] RUN ls -a /workspace/files                   
INFO[0002] Initializing snapshotter ...                 
INFO[0002] Taking snapshot of full filesystem...        
INFO[0003] Cmd: /bin/sh                                 
INFO[0003] Args: [-c ls -a /workspace/files]            
INFO[0003] Running: [/bin/sh -c ls -a /workspace/files] 
.
..
.exclude_me
include_me
INFO[0003] Taking snapshot of full filesystem...        
INFO[0003] No files were changed, appending empty layer to config. No layer added to image. 
INFO[0003] Resolving srcs [files/[^.]*]...              
INFO[0003] COPY ./files/[^.]* /                         
INFO[0003] Resolving srcs [files/[^.]*]...              
INFO[0003] Taking snapshot of files...                  
INFO[0003] Pushing image to gcr.io/aprindle-test-cluster/kaniko-test:latest 
INFO[0006] Pushed gcr.io/aprindle-test-cluster/kaniko-test@sha256:7078264902254c40122fd3ab3d462df40c032b03fda42de2cc1512b55dc9118a 

aprindle@aprindle-ssd ~/kaniko  [release-automation]docker run -it gcr.io/aprindle-test-cluster/kaniko-test@sha256:7078264902254c40122fd3ab3d462df40c032b03fda42de2cc1512b55dc9118a bash
Unable to find image 'gcr.io/aprindle-test-cluster/kaniko-test@sha256:7078264902254c40122fd3ab3d462df40c032b03fda42de2cc1512b55dc9118a' locally
gcr.io/aprindle-test-cluster/kaniko-test@sha256:7078264902254c40122fd3ab3d462df40c032b03fda42de2cc1512b55dc9118a: Pulling from aprindle-test-cluster/kaniko-test
9f1ecb66bc03: Already exists 
929208e555c0: Pull complete 
Digest: sha256:7078264902254c40122fd3ab3d462df40c032b03fda42de2cc1512b55dc9118a
Status: Downloaded newer image for gcr.io/aprindle-test-cluster/kaniko-test@sha256:7078264902254c40122fd3ab3d462df40c032b03fda42de2cc1512b55dc9118a
root@5e0484aac370:/# ls -a
.  ..  .dockerenv  bin	boot  dev  etc	home  include_me  lib  lib64  media  mnt  opt  proc  root  run	sbin  srv  sys	tmp  usr  var
root@5e0484aac370:/# 

From the above you can see the file .exclude_me is indeed excluded due to the Dockerfile COPY regex/matching syntax used