Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

/run/secrets/<secret-id> file not present when secrets are read from env variables #2302

Closed
shaaga opened this issue Aug 12, 2021 · 15 comments
Closed

/run/secrets/<secret-id> file not present when secrets are read from env variables #2302

shaaga opened this issue Aug 12, 2021 · 15 comments

Comments

@shaaga
Copy link

shaaga commented Aug 12, 2021 •
edited
Loading

Dockerfile content:

# syntax=docker/dockerfile:labs
FROM registry.access.redhat.com/rhel7/rhel:latest
RUN --mount=type=secret,id=username cat /run/secrets/username

Docker build command:

DOCKER_BUILDKIT=1  docker build --no-cache --progress plain \
--secret id=username,env=RHS_USERNAME \
-t image-builder:1
--file "$DOCKERFILE_LOC" .

RHS_USERNAME variable is set:

echo $RHS_USERNAME
abcd

Output of build command:

#1 [internal] load build definition from Dockerfile
#1 sha256:ca8f81fe080bea2a8a9e00d8cc6770484ef9de0c41bb5e35ba0a73985c33b983
#1 transferring dockerfile: 550B done
#1 DONE 0.0s

 

#2 [internal] load .dockerignore
#2 sha256:d7915c51c0034c60580ad54cbfab10254f2992a83a73ded38fa37d8ac831bdef
#2 transferring context: 2B done
#2 DONE 0.1s

 

#3 resolve image config for docker.io/docker/dockerfile:labs
#3 sha256:ae6043fa72988ff5fbc61f95aed2ab3a49d353e5e459f8782ab755d536177982
#3 DONE 1.0s

 

#4 docker-image://docker.io/docker/dockerfile:labs@sha256:03ca0e50aa4b6e76365fa9a5607c3f988bc9284de6a82672eab5ad627324e1fe
#4 sha256:2529344acb43a54bc4c02adfb059b417204b0876186a9cf5c4b073d079cc4308
#4 CACHED

 

#6 [internal] load .dockerignore
#6 sha256:dcc0744fe62b4d681df3e3bd693aebeaf12a5b9cf49b86055dc41e724a78c203
#6 DONE 0.0s

 

#5 [internal] load build definition from Dockerfile
#5 sha256:c9174ba7c649a92015f3647ade97e8a9cc1c9027802e984d6cfe623c1243dc47
#5 DONE 0.0s

 

#7 [internal] load metadata for registry.access.redhat.com/rhel7/rhel:latest
#7 sha256:7bb623c8cb006ce5edcb4845c7475e0dcaf4399de385f184a0eb095b6e6f569b
#7 DONE 1.5s

 

#8 [stage-0 1/5] FROM registry.access.redhat.com/rhel7/rhel:latest@sha256:6d3dea6ba14301d963d75184f91dc1f8f00114b6422b8e11107acc47852a36ad
#8 sha256:7222d3c811633a59fefde21161a878f866cd7c81907084bbbc860cfb41832cd2
#8 CACHED

 

#11 [internal] load build context
#11 sha256:16b45d2752d6f4975ee9d1a79a6beff1549a11649c79e2ea785a26335428190d
#11 transferring context: 73B done
#11 DONE 0.0s

 

#9 [stage-0 2/5] RUN --mount=type=secret,id=username cat /run/secrets/username
#9 sha256:47907de552841457d7a26b778b689477f0de29dbdfe192052c58f8b2c5f454bb
#9 0.503 cat: /run/secrets/username: No such file or directory
#9 ERROR: executor failed running [/bin/sh -c cat /run/secrets/username]: exit code: 1
------
 > [stage-0 2/5] RUN --mount=type=secret,id=username cat /run/secrets/username:
------
executor failed running [/bin/sh -c cat /run/secrets/username]: exit code: 1

When I store the env variable value in a text file (mysecret.txt) and modify the docker build command as below, everything works fine.

DOCKER_BUILDKIT=1  docker build --no-cache --progress plain \
--secret id=username,src=mysecret.txt \
-t image-builder:1
--file "$DOCKERFILE_LOC" .

What am I missing here? Are the secrets not stored in /run/secrets directory when created from an env variable?

Suggested assignee
@tonistiigi

shaaga@osimagecreationvm:/datadrive/Image-builder/scripts$ sudo docker version
Client: Docker Engine - Community
Version: 20.10.7
API version: 1.41
Go version: go1.13.15
Git commit: f0df350
Built: Wed Jun 2 11:56:40 2021
OS/Arch: linux/amd64
Context: default
Experimental: true

Server: Docker Engine - Community
Engine:
Version: 20.10.7
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: b0f5bc3
Built: Wed Jun 2 11:54:48 2021
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.4.6
GitCommit: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc:
Version: 1.0.0-rc95
GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
docker-init:
Version: 0.19.0
GitCommit: de40ad0

shaaga@osimagecreationvm:/datadrive/Image-builder/scripts$ sudo docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 20.10.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.4.0-1051-azure
Operating System: Ubuntu 18.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.776GiB
Name: osimagecreationvm
ID: 35O2:2VPC:Z4HC:GQVU:QYC6:XRTW:O4TF:HPRU:ZTA2:ZXZC:KA5S:TO5V
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
@shaaga shaaga mentioned this issue Aug 12, 2021
Merged
@tonistiigi
Copy link
Member

Do you have RHS_USERNAME defined?

@shaaga
Copy link
Author

shaaga commented Aug 12, 2021

Yes, like I mentioned in the 1st comment, its value is set as 'abcd'

echo $RHS_USERNAME
abcd

@thaJeztah
Copy link
Member

Just to be sure, because I recall Red Hat also using /run/secrets for other purposes; do you see the same if you mount the secret in a different location? e.g.;

RUN --mount=type=secret,id=username,target=/foobar cat /foobar

@tonistiigi
Copy link
Member 

/tmp # cat Dockerfile
#syntax=tonistiigi/dockerfile:test
from alpine
RUN --mount=type=secret,id=username ls -l /run/secrets
/tmp # export RHS_USERNAME=aa
/tmp # docker build --no-cache --progress plain \
> --secret id=username,env=RHS_USERNAME .
#1 [internal] load build definition from Dockerfile
#1 sha256:6ac6ced61631975ccf081bdeef931b4d4139c0b46682e30a80e7beab5bdd94c6
#1 transferring dockerfile: 31B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:c2644cd290def98c7d6ea47f249883ee5021dd445423b09296d4d98312742605
#2 transferring context: 2B done
#2 DONE 0.0s

#3 resolve image config for docker.io/tonistiigi/dockerfile:test
#3 sha256:339b45a3f710d1352d44df2597d8d16cd8cfa77a4132b8469a5fff8bf8a321b6
#3 DONE 0.5s

#4 docker-image://docker.io/tonistiigi/dockerfile:test@sha256:9b0d4e58826f869ac25e046bb2b85470f15d8e901030e1e7a28fcbf9dbaf211e
#4 sha256:28c1bbf37e03d2d17fe4605c3a0e29d6c8633786e7301a77c65aa8c4c90061ea
#4 CACHED

#6 [internal] load build definition from Dockerfile
#6 sha256:ebbff027854ff9316ddf5c8a139a1cee4e331f0ddbab9d8f6a87ac173269e766
#6 DONE 0.0s

#5 [internal] load .dockerignore
#5 sha256:700db7d07b059a418e13798151004e54dcf7e30e6de19f4290074fa645012304
#5 DONE 0.0s

#7 [internal] load metadata for docker.io/library/alpine:latest
#7 sha256:d4fb25f5b5c00defc20ce26f2efc4e288de8834ed5aa59dff877b495ba88fda6
#7 DONE 0.5s

#8 [stage-0 1/2] FROM docker.io/library/alpine@sha256:eb3e4e175ba6d212ba1d6e04fc0782916c08e1c9d7b45892e9796141b1d379ae
#8 sha256:2b581387a274b1c30543dc01e7bfc9a440e08af1286b09556e022d6a4425aed0
#8 CACHED

#9 [stage-0 2/2] RUN --mount=type=secret,id=username ls -l /run/secrets
#9 sha256:d4c7abf0a1ce5a827e91a96fb4cc63dd85e40cd1d21e1d44f236af86697e53d1
#9 0.428 total 4
#9 0.429 -r--------    1 root     root             2 Aug 12 16:49 username
#9 DONE 0.4s

#10 exporting to image
#10 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#10 exporting layers 0.0s done
#10 writing image sha256:b546c5791e6968e6a1a4de8da265319d034d0d8e8d715645c56f75069e7f7d41 done
#10 DONE 0.0s

@shaaga
Copy link
Author

shaaga commented Aug 12, 2021 •
edited by thaJeztah
Loading

Just to be sure, because I recall Red Hat also using /run/secrets for other purposes; do you see the same if you mount the secret in a different location? e.g.;

RUN --mount=type=secret,id=username,target=/foobar cat /foobar

I tried that but its also not working.

#1 [internal] load build definition from Dockerfile
#1 sha256:7caf50a48938a3a70c5d6069f2622be33ffde83a9ec57c9d7ed48c7ff770d6a5
#1 transferring dockerfile: 38B done
#1 DONE 0.0s

 

#2 [internal] load .dockerignore
#2 sha256:9e3356587cf0df009813e5532b88afe14dca8797932ade26a8a308e99833724d
#2 transferring context: 2B done
#2 DONE 0.1s

 

#3 resolve image config for docker.io/docker/dockerfile:labs
#3 sha256:ae6043fa72988ff5fbc61f95aed2ab3a49d353e5e459f8782ab755d536177982
#3 DONE 1.0s

 

#4 docker-image://docker.io/docker/dockerfile:labs@sha256:03ca0e50aa4b6e76365fa9a5607c3f988bc9284de6a82672eab5ad627324e1fe
#4 sha256:2529344acb43a54bc4c02adfb059b417204b0876186a9cf5c4b073d079cc4308
#4 CACHED

 

#6 [internal] load .dockerignore
#6 sha256:c39139793edb85ab06439a0417c95f7c62c0d198a23a77ed24735ba8780ecb02
#6 DONE 0.0s

 

#5 [internal] load build definition from Dockerfile
#5 sha256:24f3430427b2a1c736d445276507d5654cf702ac6b62bf209e2ec4f33e2e4248
#5 DONE 0.0s

 

#7 [internal] load metadata for registry.access.redhat.com/rhel7/rhel:latest
#7 sha256:7bb623c8cb006ce5edcb4845c7475e0dcaf4399de385f184a0eb095b6e6f569b
#7 DONE 2.4s

 

#8 [stage-0 1/5] FROM registry.access.redhat.com/rhel7/rhel:latest@sha256:6d3dea6ba14301d963d75184f91dc1f8f00114b6422b8e11107acc47852a36ad
#8 sha256:7222d3c811633a59fefde21161a878f866cd7c81907084bbbc860cfb41832cd2
#8 CACHED

 

#11 [internal] load build context
#11 sha256:69e7f31ea4ba0ef2b4b8e4a8ad263ddb9114f7dc463e5a734bc3438ea0793c43
#11 transferring context: 73B done
#11 DONE 0.0s

 

#9 [stage-0 2/5] RUN --mount=type=secret,id=username,target=/username cat /username
#9 sha256:cac992157b575a599544e65024e2883c0bbb5c3f7a59e4edc90931763d1a64e4
#9 0.658 cat: /username: No such file or directory
#9 ERROR: executor failed running [/bin/sh -c cat /username]: exit code: 1
------
 > [stage-0 2/5] RUN --mount=type=secret,id=username,target=/username cat /username:
------
executor failed running [/bin/sh -c cat /username]: exit code: 1

@shaaga
Copy link
Author

shaaga commented Aug 12, 2021 •
edited by thaJeztah
Loading 

/tmp # cat Dockerfile
#syntax=tonistiigi/dockerfile:test
from alpine
RUN --mount=type=secret,id=username ls -l /run/secrets
/tmp # export RHS_USERNAME=aa
/tmp # docker build --no-cache --progress plain \
> --secret id=username,env=RHS_USERNAME .
#1 [internal] load build definition from Dockerfile
#1 sha256:6ac6ced61631975ccf081bdeef931b4d4139c0b46682e30a80e7beab5bdd94c6
#1 transferring dockerfile: 31B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:c2644cd290def98c7d6ea47f249883ee5021dd445423b09296d4d98312742605
#2 transferring context: 2B done
#2 DONE 0.0s

#3 resolve image config for docker.io/tonistiigi/dockerfile:test
#3 sha256:339b45a3f710d1352d44df2597d8d16cd8cfa77a4132b8469a5fff8bf8a321b6
#3 DONE 0.5s

#4 docker-image://docker.io/tonistiigi/dockerfile:test@sha256:9b0d4e58826f869ac25e046bb2b85470f15d8e901030e1e7a28fcbf9dbaf211e
#4 sha256:28c1bbf37e03d2d17fe4605c3a0e29d6c8633786e7301a77c65aa8c4c90061ea
#4 CACHED

#6 [internal] load build definition from Dockerfile
#6 sha256:ebbff027854ff9316ddf5c8a139a1cee4e331f0ddbab9d8f6a87ac173269e766
#6 DONE 0.0s

#5 [internal] load .dockerignore
#5 sha256:700db7d07b059a418e13798151004e54dcf7e30e6de19f4290074fa645012304
#5 DONE 0.0s

#7 [internal] load metadata for docker.io/library/alpine:latest
#7 sha256:d4fb25f5b5c00defc20ce26f2efc4e288de8834ed5aa59dff877b495ba88fda6
#7 DONE 0.5s

#8 [stage-0 1/2] FROM docker.io/library/alpine@sha256:eb3e4e175ba6d212ba1d6e04fc0782916c08e1c9d7b45892e9796141b1d379ae
#8 sha256:2b581387a274b1c30543dc01e7bfc9a440e08af1286b09556e022d6a4425aed0
#8 CACHED

#9 [stage-0 2/2] RUN --mount=type=secret,id=username ls -l /run/secrets
#9 sha256:d4c7abf0a1ce5a827e91a96fb4cc63dd85e40cd1d21e1d44f236af86697e53d1
#9 0.428 total 4
#9 0.429 -r--------    1 root     root             2 Aug 12 16:49 username
#9 DONE 0.4s

#10 exporting to image
#10 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#10 exporting layers 0.0s done
#10 writing image sha256:b546c5791e6968e6a1a4de8da265319d034d0d8e8d715645c56f75069e7f7d41 done
#10 DONE 0.0s

Can you please try once with "registry.access.redhat.com/rhel7/rhel:latest" as base image. I tried listing the secrets using ls -l /run/secrets but seems no secrets are present

#1 [internal] load build definition from Dockerfile
#1 sha256:a6f5685e58021489e56d515bd6ef7252acbe05acdd7622c14138576abc620795
#1 transferring dockerfile: 544B done
#1 DONE 0.1s

 

#2 [internal] load .dockerignore
#2 sha256:91ec31d70c6b839166e5f1af1b270d7a902904c7b3fdf410a299caaf03dfe0bf
#2 transferring context: 2B done
#2 DONE 0.1s

 

#3 resolve image config for docker.io/docker/dockerfile:labs
#3 sha256:ae6043fa72988ff5fbc61f95aed2ab3a49d353e5e459f8782ab755d536177982
#3 DONE 2.0s

 

#4 docker-image://docker.io/docker/dockerfile:labs@sha256:03ca0e50aa4b6e76365fa9a5607c3f988bc9284de6a82672eab5ad627324e1fe
#4 sha256:2529344acb43a54bc4c02adfb059b417204b0876186a9cf5c4b073d079cc4308
#4 CACHED

 

#5 [internal] load build definition from Dockerfile
#5 sha256:9f15672a5c39790f9578dd40885b9023d356b2ebe8005e8f9c552003c2eb4f85
#5 DONE 0.0s

 

#6 [internal] load .dockerignore
#6 sha256:87d59c52739857134655fbb805eb26d72943b16ce25b695b21489df21dfdaa9d
#6 DONE 0.0s

 

#7 [internal] load metadata for registry.access.redhat.com/rhel7/rhel:latest
#7 sha256:7bb623c8cb006ce5edcb4845c7475e0dcaf4399de385f184a0eb095b6e6f569b
#7 DONE 1.3s

 

#8 [stage-0 1/5] FROM registry.access.redhat.com/rhel7/rhel:latest@sha256:6d3dea6ba14301d963d75184f91dc1f8f00114b6422b8e11107acc47852a36ad
#8 sha256:7222d3c811633a59fefde21161a878f866cd7c81907084bbbc860cfb41832cd2
#8 CACHED

 

#11 [internal] load build context
#11 sha256:d9b5e18d79aba54d880ecc22e48d4420e28379b7860d14df1cc08b2a5dc2613f
#11 transferring context: 73B done
#11 DONE 0.0s

 

#9 [stage-0 2/5] RUN --mount=type=secret,id=username ls -l /run/secrets/
#9 sha256:26fd3d464ca7277b16db174981d926db6c931278c9755912dedddfaf7391d56e
#9 0.520 total 0
#9 DONE 0.7s

@thaJeztah
Copy link
Member

Could you also update the top description to include docker version and docker info ? If you are using buildkit through the docker cli, then you may need to have the docker 20.10 cli (the option was added in the cli through docker/cli#2656, which is in docker 20.10 and up)

Or you can install buildx to get an updated client for buildkit

@shaaga
Copy link
Author

shaaga commented Aug 12, 2021 •
edited by thaJeztah
Loading

Yes the docker version is 20.10.7. Pasting output of both commands below

shaaga@osimagecreationvm:/datadrive/Image-builder/scripts$ sudo docker version
Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:40 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

 

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:48 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

shaaga@osimagecreationvm:/datadrive/Image-builder/scripts$ sudo docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

 

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-1051-azure
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 7.776GiB
 Name: osimagecreationvm
 ID: 35O2:2VPC:Z4HC:GQVU:QYC6:XRTW:O4TF:HPRU:ZTA2:ZXZC:KA5S:TO5V
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

 

WARNING: No swap limit support

@thaJeztah
Copy link
Member

Thanks for adding that info.

Nope, no ideas then. I tried your example earlier, also with the rhel base image, but wasn't able to reproduce the problem 🤔

@tonistiigi
Copy link
Member

Make sure the value shows up on running env. echo might just show your local shell variable.

@shaaga
Copy link
Author

shaaga commented Aug 13, 2021

Okay, where should I be setting the env variable? Right now, I am setting it in "/home//.bashrc" file and then I did run source "/home/shaaga/.bashrc".
But the docker build command runs with "sudo". Should I be adding anywhere else?

@shaaga
Copy link
Author

shaaga commented Aug 13, 2021

Yes it shows up in env list

@thaJeztah
Copy link
Member

But the docker build command runs with "sudo". Should I be adding anywhere else?

Ah! Your reproduction steps didn't show that sudo is used.

In that case, use sudo -E to preserve environment variables, otherwise they are reset, and you get the environment variables of the root user.

@shaaga
Copy link
Author

shaaga commented Aug 13, 2021

But the docker build command runs with "sudo". Should I be adding anywhere else?

Ah! Your reproduction steps didn't show that sudo is used.

In that case, use sudo -E to preserve environment variables, otherwise they are reset, and you get the environment variables of the root user.

Yes that worked. Thank you so much.

@thaJeztah
Copy link
Member

Great! Let me close this ticket, as (from the above) it looks like there's no bug.

Contributions to improve the documentation (in case it's useful to mention) welcome though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tonistiigi @thaJeztah @shaaga