sort out the private registry auth story on draft init

[bacongobbler]

Right now the user story to inject registry auth details into draftd is to echo a JSON string, pipe it to base64 and use the output as the registry.authtoken field in the chart.

We should test out the following scenarios:

1. mimic docker login on draft init somehow, asking users for their credentials
2. fetch the auth tokens from ~/.docker/config.json, if present

That way we can "hide" this process from the end user and do it for them without any terminal hackery.

Of course, this process of attaching your own base64-encoded string as the auth token might still be useful for those that want to do things the hard way or have their own custom setup.

[bacongobbler]

So the current user story is that the user calls draft init. At this state, the program checks ~/.docker/config.json for credentials. It then branches based on information found:

1. creds are noticed in ~/.docker/config.json
2. ~/.docker/config.json not found

For the former, draft will display the following information:

><> draft init
Reading Docker Client configuration...
It appears you are logged into at least one Docker Registry. Which would you like to use with Draft?

1) docker.io
2) https://index.docker.io/v1/
3) quay.io
4) None. I want to enter my own information.

Please enter a number: 

If it's the latter (or the user chose option 4), then the following information is displayed to the user:

Reading Docker Client configuration...
It appears that you are not logged into a Docker Registry. Please login.
registry URL: docker.io
username: bacongobbler
password: ************
organization (optional, leave blank to use your username):

This prompt window can be bypassed by supplying --registry-url, --username, and --password to draft init.

Once the information is gathered from the client configuration, it will generate the auth token and inject it into the draft helm chart as a secret.

NOTE: short-lived token auth support will not be part of the initial rollout of this feature. For users running on Google Container Registry or Amazon EC2 Container Registry, the recommended way to log in would be to use the JSON key file via

><> draft init --registry-url "gcr.io" --username "_json_key" -password "$(cat keyfile.json)"

[squillace]

@bacongobbler @technosophos @michelleN

I'm having this problem. I've done everything I think I know how to get this running on ACR -- using the provided domains so not messing with any custom domains.

Here is the calls that I make to draft init:

draft init --set registry.url=$acrname.azurecr.io,registry.org=$acrname,registry.authtoken=$authtoken,basedomain=azurecr.io

where

• $acrname is "draftacs"
• the $authtoken is generated by

  password=$(az acr credential show -n $acrname --output tsv --query "passwords[0].value")
  authtoken=$(echo \{\"username\":\"$acrname\",\"password\":\"$password\",\"email\":\"rasquill@microsoft.com\"\} | base64)
  

Everything works fine but then I get: 

c59fa6cbcbd9: Preparing
8d4d1ab5ff74: Preparing
ab0825ad21e8: Waiting
e7b0b4cd055a: Waiting
445ed6ee6867: Waiting
c59fa6cbcbd9: Waiting
8d4d1ab5ff74: Waiting
Error encountered streaming JSON response: no basic auth credentials
Error: there was an error running 'draft up': websocket: close 1001 (going away)

Any ideas?

[squillace]

It seems an auth issue with the registry. BTW, minikube seems to work perfectly, but the same workflow for minikube is:

8d4d1ab5ff74: Preparing
Error encountered streaming JSON response: no basic auth credentials
Error: there was an error running 'draft up': websocket: close 1001 (going away)

Uh, these will both work if someone knows how to identify what I'm doing wrong to auth to the registry..... :-) I will be happy to write a doc for azure/draft showing minikube if I get the registry auth working fine.

[squillace]

Again, I can log on directly to my registry with the SP that I had to create with the registry. I get, however, the same error when directly connecting using THOSE credentials as my username and password, too. Sigh. Writing up the doc, because I'm sure it's a simple thing and as soon as we figure it out, we can push both ACS/ACR doc and a minikube to azure/draft.

[squillace]

Again, I can log on directly to my registry with the SP that I had to create with the registry. I get, however, the same error when directly connecting using THOSE credentials as my username and password, too. Sigh. Writing up the doc, because I'm sure it's a simple thing and as soon as we figure it out, we can push both ACS/ACR doc and a minikube to azure/draft.

[technosophos]

So you got to the point where draft (or draftd) is running in your cluster? That is, draft init succeeded, but it's draft up that is failing?

You might check kubectl get deployment -o yaml -n kube-system and make sure that the args sent to the Draft pod have the right registry information

[squillace]

Ignore my issue. #UserError. :-) That said, the issue is a good one.

[jackzampolin]

So I'm running into this too. How would you do it for GKE? Acording to the docs and this issue the following should (?) work but doesn't:

draft init --set 
  registry.url=gcr.io,
  registry.org=influx-perf-testing,
  registry.authtoken=$(echo '{"username":"_json_key","password":"$(cat ~/.draft/keyfile.json)"}' | base 64),
  basedomain=my.tld```

[DazWilkin]

@jackzampolin I was able to get this to work on GKE using Application Default Credentials:

gcloud auth application-default login
gcloud auth application-default print-access-token

The draft init command that works for me is then:

ACCESS_TOKEN=$(gcloud auth application-default print-access-token)
AUTH_TOKEN=$(echo "{\"registrytoken\":\"$ACCESS_TOKEN\"}" | base64 --wrap=0)
 
draft init \
--set \
registry.url=gcr.io,\
registry.org=${PROJECT},\
registry.authtoken=${AUTH_TOKEN},\
basedomain=${DOMAIN}

[DazWilkin]

@jackzampolin my approach was incorrect.

I spoke with one of the Engineers working on Container Registry and here's the correction:

AUTH_TOKEN=$(echo "{\"username\":\"_token\",\"password\":\"$ACCESS_TOKEN\"}" | base64 --wrap=0)

Otherwise unchanged.

[Disturbing]

One other suggestion - After init has been called once, second time it should auto --upgrade without the flag?

If this is poor standardization, then I'd suggest disabling init and stating 'already initialized, don't forget to use --upgrade flag'. I was in a horrible loop for 1 hour because I was forgetting the upgrade flag when I needed to update my credentials yesterday.

[bacongobbler]

I disagree that init should handle upgrades. In fact I would rather we limit the scope to literally initializing the cluster for the first time. After that the user can call draft up on draft itself to upgrade. I'd like to see that as a more solid "Turing complete" upgrade workflow.

[Disturbing]

Agreed, what's your proposal for updating credentials / basedomain / etc. That's the key UX that needs to be solved as from what I found, the only way I could update my credentials was draft init {updated config} --upgrade.

I did not see any user stories around that on this thread, so would love to know what's the game plan there.

In short-term, would suggest updating https://github.com/Azure/draft/blob/master/docs/install.md - to share information on how to update your configuration. It would have saved me tons of time after a silly typo.

[bacongobbler]

Subsequent calls to draft up will update the application. You can either supply your custom registry information through draft.toml or though values.yaml. I use this during development when hacking on changes to draftd.

In short-term, would suggest updating https://github.com/Azure/draft/blob/master/docs/install.md - to share information on how to update your configuration. It would have saved me tons of time after a silly typo.

Right now the recommended way is to use draft init --update as you mentioned. I intentionally avoided documenting it as that functionality is going to change sooner rather than later through a PR to close out this issue.

[bacongobbler]

part one of the plan is to improve the minikube story, starting by contributing a registry addon to minikube. Once that's done, we can recommend users to use the following workflow:

$ minikube addons install ingress
$ minikube addons install registry
$ minikube start
$ helm init
$ draft init

Which at that point, draft will read the local kube config and notice that the current context name is called "minikube". It will then install draftd and point it towards the in-cluster registry.

As for Azure, we will do the same operation:

• draft will read the local kube config and notice the current context is pointing at an ACS cluster
• draft will read AZURE_RG_NAME and other envvars to search the ACS cluster's resource group for an ACR instance. If so, it will configure itself to automatically use that cluster. No authentication token will be required because ACS can talk to ACR clusters within the same resource group, as I understand it.

[bacongobbler]

My chicken-scratch writing on a whiteboard:

The one problem I have right now is "how do we install an ingress controller onto ACS?" Right now there's no solid story other than helm install stable/nginx-ingress. I'd like this to be one less step but I think it's suitable for the moment.

[squillace]

@bacongobbler, good line on minikube, azure. Will this approach work for all registries, generally? I'll follow up with the ACR/ACS access.....

[bacongobbler]

Generally speaking, if someone were to write a Google Cloud bootstrap workflow for draft then yes. I'm limiting scope for v0.5.0 to just minikube and Azure for now as that happens to be the large majority of users trying out draft. All the other unrecognized providers will need to go through the manual approach we do today until someone can pick up the work for GKE, Amazon etc.

[bacongobbler]

(that someone likely being myself)

[squillace]

gotcha: just wanna make sure we have a good path for everyone. checking on acr/acs access issues now to be certain about our assumptions.

[bacongobbler]

closing in favour of #152. The discussion in that ticket more clearly demonstrates the issue and proposed solution going forward.

[bacongobbler]

checking on acr/acs access issues now to be certain about our assumptions.

My assumptions were wrong. Either way we can still automate this process to fetch the admin username/password and pass that to draftd.