Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Non-Docker clusters] Permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: #1968

Closed
vinayjain15 opened this issue Aug 27, 2019 · 2 comments
Closed

[Non-Docker clusters] Permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: #1968

vinayjain15 opened this issue Aug 27, 2019 · 2 comments
Assignees
@Ark-kun @IronPan @hongye-sun

Comments

@vinayjain15
Copy link

All sample pipelines in kubeflow failing with error in container wait:
Permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock:

kubeflow 0.5.0 installed on openshift 3.11 (kubernetes v1.11.0+d4cacc)

./oc logs sequential-gzx5k-652412774 -c wait
time="2019-08-27T19:01:34Z" level=info msg="Creating a docker executor"
time="2019-08-27T19:01:34Z" level=info msg="Executor (version: v2.2.0, build_date: 2018-08-30T08:52:54Z) initialized with template:\narchiveLocation:\n s3:\n accessKeySecret:\n key: accesskey\n name: mlpipeline-minio-artifact\n bucket: mlpipeline\n endpoint: minio-service.kubeflow:9000\n insecure: true\n key: artifacts/sequential-gzx5k/sequential-gzx5k-652412774\n secretKeySecret:\n key: secretkey\n name: mlpipeline-minio-artifact\ncontainer:\n args:\n - gsutil cat gs://ml-pipeline-playground/shakespeare1.txt | tee /tmp/results.txt\n command:\n - sh\n - -c\n image: google/cloud-sdk:216.0.0\n name: ""\n resources: {}\ninputs:\n parameters:\n - name: url\n value: gs://ml-pipeline-playground/shakespeare1.txt\nmetadata: {}\nname: download\noutputs:\n artifacts:\n - name: mlpipeline-ui-metadata\n path: /mlpipeline-ui-metadata.json\n s3:\n accessKeySecret:\n key: accesskey\n name: mlpipeline-minio-artifact\n bucket: mlpipeline\n endpoint: minio-service.kubeflow:9000\n insecure: true\n key: runs/e94077b1-c8fc-11e9-a6c0-a0935189926f/sequential-gzx5k-652412774/mlpipeline-ui-metadata.tgz\n secretKeySecret:\n key: secretkey\n name: mlpipeline-minio-artifact\n - name: mlpipeline-metrics\n path: /mlpipeline-metrics.json\n s3:\n accessKeySecret:\n key: accesskey\n name: mlpipeline-minio-artifact\n bucket: mlpipeline\n endpoint: minio-service.kubeflow:9000\n insecure: true\n key: runs/e94077b1-c8fc-11e9-a6c0-a0935189926f/sequential-gzx5k-652412774/mlpipeline-metrics.tgz\n secretKeySecret:\n key: secretkey\n name: mlpipeline-minio-artifact\n parameters:\n - name: download-downloaded\n valueFrom:\n path: /tmp/results.txt\n"
time="2019-08-27T19:01:34Z" level=info msg="Waiting on main container"
time="2019-08-27T19:01:35Z" level=info msg="main container started with container ID: 644746b591f59147bae44b13db74e43684332d7e33646e2006a0d63cb5f5fe2a"
time="2019-08-27T19:01:35Z" level=info msg="Starting annotations monitor"
time="2019-08-27T19:01:35Z" level=info msg="Starting deadline monitor"
time="2019-08-27T19:01:35Z" level=info msg="docker wait 644746b591f59147bae44b13db74e43684332d7e33646e2006a0d63cb5f5fe2a"
time="2019-08-27T19:01:35Z" level=error msg="docker wait 644746b591f59147bae44b13db74e43684332d7e33646e2006a0d63cb5f5fe2a failed: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.38/containers/644746b591f59147bae44b13db74e43684332d7e33646e2006a0d63cb5f5fe2a/wait?condition=: dial unix /var/run/docker.sock: connect: permission denied\n"
time="2019-08-27T19:01:35Z" level=info msg="Main container completed"
time="2019-08-27T19:01:35Z" level=info msg="No sidecars"
time="2019-08-27T19:01:35Z" level=info msg="Saving output artifacts"
time="2019-08-27T19:01:35Z" level=info msg="Annotations monitor stopped"
time="2019-08-27T19:01:35Z" level=info msg="Saving artifact: mlpipeline-ui-metadata"
time="2019-08-27T19:01:35Z" level=info msg="Archiving 644746b591f59147bae44b13db74e43684332d7e33646e2006a0d63cb5f5fe2a:/mlpipeline-ui-metadata.json to /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz"
time="2019-08-27T19:01:35Z" level=info msg="sh -c docker cp -a 644746b591f59147bae44b13db74e43684332d7e33646e2006a0d63cb5f5fe2a:/mlpipeline-ui-metadata.json - | gzip > /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz"
time="2019-08-27T19:01:35Z" level=info msg="Archiving completed"
time="2019-08-27T19:01:35Z" level=info msg="Creating minio client minio-service.kubeflow:9000 using static credentials"
time="2019-08-27T19:01:35Z" level=info msg="Saving from /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz to s3 (endpoint: minio-service.kubeflow:9000, bucket: mlpipeline, key: runs/e94077b1-c8fc-11e9-a6c0-a0935189926f/sequential-gzx5k-652412774/mlpipeline-ui-metadata.tgz)"
time="2019-08-27T19:01:35Z" level=info msg="Successfully saved file: /argo/outputs/artifacts/mlpipeline-ui-metadata.tgz"
time="2019-08-27T19:01:35Z" level=info msg="Saving artifact: mlpipeline-metrics"
time="2019-08-27T19:01:35Z" level=info msg="Archiving 644746b591f59147bae44b13db74e43684332d7e33646e2006a0d63cb5f5fe2a:/mlpipeline-metrics.json to /argo/outputs/artifacts/mlpipeline-metrics.tgz"
time="2019-08-27T19:01:35Z" level=info msg="sh -c docker cp -a 644746b591f59147bae44b13db74e43684332d7e33646e2006a0d63cb5f5fe2a:/mlpipeline-metrics.json - | gzip > /argo/outputs/artifacts/mlpipeline-metrics.tgz"
time="2019-08-27T19:01:35Z" level=info msg="Archiving completed"
time="2019-08-27T19:01:35Z" level=info msg="Creating minio client minio-service.kubeflow:9000 using static credentials"
time="2019-08-27T19:01:35Z" level=info msg="Saving from /argo/outputs/artifacts/mlpipeline-metrics.tgz to s3 (endpoint: minio-service.kubeflow:9000, bucket: mlpipeline, key: runs/e94077b1-c8fc-11e9-a6c0-a0935189926f/sequential-gzx5k-652412774/mlpipeline-metrics.tgz)"
time="2019-08-27T19:01:35Z" level=info msg="Successfully saved file: /argo/outputs/artifacts/mlpipeline-metrics.tgz"
time="2019-08-27T19:01:35Z" level=info msg="Saving output parameters"
time="2019-08-27T19:01:35Z" level=info msg="Saving path output parameter: download-downloaded"
time="2019-08-27T19:01:35Z" level=info msg="[sh -c docker cp -a 644746b591f59147bae44b13db74e43684332d7e33646e2006a0d63cb5f5fe2a:/tmp/results.txt - | tar -ax -O]"
time="2019-08-27T19:01:35Z" level=error msg="[sh -c docker cp -a 644746b591f59147bae44b13db74e43684332d7e33646e2006a0d63cb5f5fe2a:/tmp/results.txt - | tar -ax -O] stderr:\nGot permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.38/containers/644746b591f59147bae44b13db74e43684332d7e33646e2006a0d63cb5f5fe2a/archive?path=%2Ftmp%2Fresults.txt: dial unix /var/run/docker.sock: connect: permission denied\ntar: This does not look like a tar archive\ntar: Exiting with failure status due to previous errors\n"
time="2019-08-27T19:01:35Z" level=info msg="Alloc=5429 TotalAlloc=13142 Sys=12714 NumGC=5 Goroutines=10"
time="2019-08-27T19:01:35Z" level=fatal msg="exit status 2\ngithub.com/argoproj/argo/errors.Wrap\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:87\ngithub.com/argoproj/argo/errors.InternalWrapError\n\t/root/go/src/github.com/argoproj/argo/errors/errors.go:70\ngithub.com/argoproj/argo/workflow/executor/docker.(*DockerExecutor).GetFileContents\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/docker/docker.go:40\ngithub.com/argoproj/argo/workflow/executor.(*WorkflowExecutor).SaveParameters\n\t/root/go/src/github.com/argoproj/argo/workflow/executor/executor.go:343\ngithub.com/argoproj/argo/cmd/argoexec/commands.waitContainer\n\t/root/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:49\ngithub.com/argoproj/argo/cmd/argoexec/commands.glob..func4\n\t/root/go/src/github.com/argoproj/argo/cmd/argoexec/commands/wait.go:19\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).execute\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:766\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).ExecuteC\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:852\ngithub.com/argoproj/argo/vendor/github.com/spf13/cobra.(*Command).Execute\n\t/root/go/src/github.com/argoproj/argo/vendor/github.com/spf13/cobra/command.go:800\nmain.main\n\t/root/go/src/github.com/argoproj/argo/cmd/argoexec/main.go:15\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:198\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:2361"
@Ark-kun
Copy link
Contributor

Ark-kun commented Aug 28, 2019

Thanks for the report.
This issue seems to be duplicate of #1654
At this moment we have some issues running on Docker-less clusters.

@Ark-kun Ark-kun changed the title Permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: [Non-Docker clusters] Permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Aug 28, 2019
@Ark-kun Ark-kun self-assigned this Sep 24, 2019
@Ark-kun
Copy link
Contributor

Ark-kun commented Sep 24, 2019

Closing as probable duplicate of #1654

@Ark-kun Ark-kun closed this as completed Sep 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Ark-kun Ark-kun

@IronPan IronPan

@hongye-sun hongye-sun

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Ark-kun @IronPan @hongye-sun @vinayjain15