Image pull error: User "system:serviceaccount:argo:argo-helm-argo-workflows-workflow-controller" cannot get resource "secrets" in API group "" in the namespace "mynamespace"
#9630
Comments
|
@vitalyrychkov can you provide your k8s version? |
|
There is PR for supporting v1.24 service account secret change. #9620 |
|
@sarabala1979 |
sarabala1979
added
the
P1
|
@terrytangyuan will work on this. |
|
Hi @sarabala1979 and @terrytangyuan We have tried to use a private image registry with anonymous pull enabled. We use the same image to start a pod (service) and to submit a task in Argo. We have tested the following scenarios:
Seems that if the imagePullSecret is specified in the deployment, the workflow-controller always tries to authenticate instead of anonymous pull? |
Created a separate issue for this: #9802 |
sarabala1979
added
the
type/regression
This comment was marked as resolved.
This comment was marked as resolved.
|
Bumping this issue, not stale. Also: https://drewdevault.com/2021/10/26/stalebot.html |
This comment was marked as resolved.
This comment was marked as resolved.
1 similar comment
This comment was marked as resolved.
This comment was marked as resolved.
|
Feel like I have a similar situation. When running on kind cluster using Tiltdev to build container into private nonsecure docker docker registry(with ctlptl). There is no issue pulling from public registries, but argo workflow just cannot seem to figure out the local registry bit, (while directly pushing a k8s Job there are no issue.) https://docs.docker.com/registry/deploying/ Scenario available here: |
|
Just add my case for reference: Context:
I configured the imagePullSecrets for submitted my workflow to this namespace and got this error: Add a new role, and grant the apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: secret-reader
namespace: default
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: argo-secret-reader
namespace: default
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: secret-reader
subjects:
- kind: ServiceAccount
name: argo
namespace: argoThe above error gone, but got a new one: Tried to figure out what's wrong for almost 4 days, even looked at the source code. Finally, found this issue: crossplane/crossplane#3023 (comment) For anyone who has the same setup as me, you must grant permissions for the Default Compute Engine service account of the project that runs Argo on GKE (Project A) to access the Container Registry/Artifact Registry in the project that hosts the container registry (Project B) For team, please update to a newer version of go-containerregistry that respects the chain order (k8s first). Thanks. |
Signed-off-by: Son Bui <sonbv00@gmail.com>
Signed-off-by: Son Bui <sonbv00@gmail.com>
Signed-off-by: Son Bui <sonbv00@gmail.com>
agilgur5
added
area/upstream
agilgur5
changed the title
Image pull error: User "system:serviceaccount:argo:argo-helm-argo-workflows-workflow-controller" cannot get resource "secrets" in API group "" in the namespace "mynamespace"
argoproj#11614) Signed-off-by: Son Bui <sonbv00@gmail.com>
Pre-requisites
:latestWhat happened/what you expected to happen?
We are storing container images of our application in a private image registry. We are deploying Argo using Helm. It seems that the workflow server in the v3.4 tries to read the container image manifest (to lookup the cmd/args) using the
argo-helm-argo-workflows-workflow-controllerservice account from theargonamespace.Reading the manifest requires registry access credentials in case of a private image registry and we provide the secret with credentials in deployments:
When we submit a workflow the workflow controller's service account fails to read the registry access credentials from the secret located in the namespace of the application:
Earlier, we have tested one of the latest 3.3.9 builds and it could pull and read the image successfully, see the issue #9139
We are using argo service account in the application's namespace to submit workflows (
--serviceaccountoption) which can read the secret in the same namespace. Would it be possible to use this service account to pull the image manifest? Otherwise the usersystem:serviceaccount:argo:argo-helm-argo-workflows-workflow-controllermust be able to read secrets in all namespaces where an application is deployed?Please explain how to use images from a private registry with access credentials in the v.3.4.0.
Version
3.4.0
Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.
[The issue seems to be specific to accessing credentials for private registries from the secret in the application's namespace.]Logs from the workflow controller
Logs from in your workflow's wait container
[no output, as the workflow could not be submitted due to manifest pull error]
The text was updated successfully, but these errors were encountered: