Cannot pull images from AWS ECR: no basic auth credentials (v0.27.0 minikube) #65
Comments
|
I'm having a similar issue with ECR creds on minikube v0.24.1 (registry-creds image upmcenterprises/registry-creds:1.8). The first time it happened, after trying to disable/re-enable registry-creds, I decided to I then ran The initial logs I saw when the registry-creds pod came up: I deployed an app that uses our private ECR registry, and voila, it worked. I then rebuilt the image and pushed it to my ECR repo with a new tag, and re-deployed my app to the minikube cluster. After that I got the dreaded I also deployed the same image and tag to a KOPS cluster and it pulled the image just fine, so I know the image tag exists. At this point, there are no new logs in registry creds to help diagnose the issue, and there appears to be no verbosity option to pass to the image to help debug. It works the first time, fails the second time. I'm wondering if it has something to do with this log line: Strange to see this considering the name of the secret that |
|
I am facing the same issue as @erstaples. @stevesloka do you have any ideas what may've gone wrong? I see a lot of Pull Requests with reasonable changes (the docs changes with info for minikube setup seems quite useful, for example) - is this repo still being supported/developed? |
|
Hey, @nicroto yes this repo is still maintained, just hasn't needed many updates recently. Let me give this a shot, something might have changed upstream with the aws sdk, but I doubt that's really the issue. I had someone else recently use this on To confirm you're doing the same steps that @erstaples did to first configure, then enable? |
|
Hey @stevesloka, thanks for the quick reply. Yes, I am doing the same thing. minikube delete
minikube start
minikube addons configure registry-creds
# then I would enter my creds from AWS
minikube addons enable registry-credsThen I would install a helm chart which has a deployment.yaml looking roughly like this: apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ template "app.fullname" . }}
labels:
app: {{ template "app.name" . }}
...
spec:
replicas: {{ .Values.replicaCount }}
template:
metadata:
labels:
app: {{ template "app.name" . }}
release: {{ .Release.Name }}
spec:
imagePullSecrets:
- name: {{ .Values.image.pullSecrets }}
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
ports:
- containerPort: {{ .Values.service.internalPort }}
livenessProbe:
httpGet:
path: /
port: {{ .Values.service.internalPort }}
initialDelaySeconds: 240
periodSeconds: 5
readinessProbe:
httpGet:
path: /
port: {{ .Values.service.internalPort }}
resources:
...
nodeSelector:
...and in my values.yaml file I would have: replicaCount: 1
image:
repository: <omitted my account id>.dkr.ecr.us-east-1.amazonaws.com/<omitted repo name>
tag: latest
pullPolicy: IfNotPresent
pullSecrets: awsecr-cred
...If it does work on your end - maybe we are making some kind of mistake when entering the creds?
|
|
@stevesloka Did you manage to check this out? |
|
Sorry I upgraded minikube and now latest doesn't work. I'll keep troubleshooting, if not I have an older build which should work. By the way, what version of minikube are you using? 0.27? |
|
Thanks. I am currently using |
|
minikube v0.28.0 is working fine. Just put to your deployment awsecr-cred instead of registry-creds-ecr
|
|
@nicroto I didn't get your step 5 in addon configuration. What dashes in your account id? Account id is just 12 numbers, so just type xxxxxxxxxxxx, but for few accounts you can split them using commas |
|
@ahanoff Maybe this is it. That is why I posted every detail of my setup, so a mistake can be ruled out. |
|
@ahanoff doesn't work for me, v0.28.2 with awsecr-cred. I have this log: Unable to retrieve pull secret default/awsecr-cred for default/data-service-7ccb57c46d-662h7 due to secrets "awsecr-cred" not found |
|
@sylvain-rouquette can you check if this secret exists using kubectl? It shoud be in kube-system namespace |
|
@ahanoff I have registry-creds-ecr running in kube-system, but I get the same error if I set this for imagePullSecrets. isn't the problem the "default/" at the beginning, shouldn't it be "kube-system/" instead? edit: I checked the content of registry-creds-ecr and it seems correctly configured.
edit2: it seems the problem could be in the addon:
"caused by: Post https://ecr.eu-west-1.amazonaws.com/: dial tcp: lookup ecr.eu-west-1.amazonaws.com on 10.96.0.10:53: read udp 172.17.0.8:33304->10.96.0.10:53: i/o timeout" edit3: enabling the ingress addon fixed that. But now it says my credentials are invalid. I specified my AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY I use somewhere else. |
|
@sylvain-rouquette can you pull image to your local environment using those credentials? Just docker pull. Thanks. So there is either really invalid credentials which is easy to check, or something wrong with setting up registry-creds. Edit1: name of secret is awsecr-cred, you can search in readme |
|
yes it works locally. now awsecr-cred doesn't show an error anymore. You were right, I had to use awsecr-cred in imagePullSecrets. But now I have this error: no basic auth credentials. |
|
You can try kill pod of registry-creds 😄 or try reconfigure registry creds again. |
|
yeah I restarted multiple times minikube, disabling the addon ("disable" is broken right now, so I do it by editing the config file), the pod is re-created after configuring and enabling the addon. But I'll try again to recreate everything from scratch and see. Thanks your your help :) edit: it works :) I had to follow very specific steps in order: In you Deployment: in your console: if you deployed before configuring registry-creds, it won't work, I guess secrets won't be refreshed in the existing pods. If registry-creds is already enabled and you can't disable it, check in $HOME/.minikube/config and disable it here, and restart minikube. |
|
If i run minikube without any driver it continuous giving this error even ingress addon enabled: if i run Minikube with VirtualBox it doesn't give any error. edit: i understand that it is about dns resolver of minikube: kubernetes/minikube#2302 |
|
Good to hear you got it working @guemues! To everyone on this thread, I'm going to close as it seems all issues are ok? If not feel free to open a new one or reopen this one. Thanks! |
|
I am still getting the "no basic auth credentials", even after following @sylvain-rouquette's procedure and having all tools upgraded to latest AND using my Account ID in "xxxxxxxxxxxx" form. How can I further debug this to give you more info on what's going wrong, here? Here is a simplification of my deployment that fails to pull an image from ECR: apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: {{ template "chart-name.fullname" . }}
labels:
app: {{ template "chart-name.name" . }}
chart: {{ template "chart-name.chart" . }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
replicas: {{ .Values.replicaCount }}
selector:
matchLabels:
app: {{ template "chart-name.name" . }}
release: {{ .Release.Name }}
template:
metadata:
labels:
app: {{ template "chart-name.name" . }}
release: {{ .Release.Name }}
spec:
imagePullSecrets:
- name: awsecr-cred
containers:
- name: {{ .Chart.Name }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }} |
|
OK, finally got it working. There probably was more than one issue in my case, but after upgrading everything to latest and getting the error I last posted, I checked the logs for the addon pod and I found that it couldn't resolve the aws dns. My account should be assigned to the "us-east-1a", but constructing the dns with the "a" at the end didn't properly resolve. Changing the region from "us-east-1a" to "us-east-1" resolved the issue with pulling images on my end. |
|
Strange, for me I'm seeing the registry-creds pod failing to start with: I'm not trying to use |
|
Ah, I found that when I ran Once I disabled the addon, then ran: I was able to pull images using a format like: spec:
template:
spec:
containers:
- name: my-container
image: ACCOUNT_ID.dkr.ecr.us-east-1.amazonaws.com/ECR_REPO:latest
imagePullSecrets:
- name: awsecr-cred |
This work for me, thanks ! |
Environment:
cat ~/.minikube/machines/minikube/config.json | grep DriverName): virtualboxcat ~/.minikube/machines/minikube/config.json | grep -i ISOorminikube ssh cat /etc/VERSION): v0.26.0What happened:
I cannot pull images from the ECR registry: "no basic auth credentials" error
What you expected to happen:
I expected to pull the image from the ECR registry after having configured registry-creds with my ID, KEY, TOKEN and AWS Region, and activating the registry-creds addon and using PullSecrets
How to reproduce it (as minimally and precisely as possible):
minikube startminikube addons configure registry-creds=> configure only with AWS ECRminikube addons enable registry-credskubectl create -f deployment.yaml=> The error occured: cannot start the container due to no basic auth credentials error.
kubectl get secrets --all-namespaces=> we can see that the secret created is in kube-system and called registry-creds-ecr. I never found the awsecr-cred name for the secret as mentioned in the documentation https://github.com/upmc-enterprises/registry-credsdeployement.yaml content:
Output of
minikube logs(if applicable):The text was updated successfully, but these errors were encountered: