Authorization token expired

[arleif-dfactory]

Describe the bug

After a few hours of working perfectly Gitpod (self-hosted, AWS) suddenly stops working for cases where the .gitpod.yml contains an image: section. The error is Request startWorkspace failed with message: 13 INTERNAL: cannot resolve workspace image: Error response from daemon: denied: Your authorization token has expired. Reauthenticate and try again. Please file an issue if you think this is a bug.
I have been able to reproduce this multiple times by terraform destroy and then reinstalling.

Steps to reproduce

1. docker run --rm -it -v "$PWD/awsinstall":"/workspace" eu.gcr.io/gitpod-io/self-hosted/installer:latest aws
2. Github provider setup
3. Verified public repos
4. Verified private repos
5. Added an image section to a .gitpod.yaml in a public repo - verified that the image builds successfully, observed it in AWS ECR.
6. Go to bed :)
7. Start the workspace again, Fails with the given error.

Expected behavior

The workspace should continue to build and load up.

Additional information

Example repository

https://github.com/arleif-dfactory/django-locallibrary-tutorial/blob/ab-gitpod/.gitpod.yml

[AlexTugarev]

@wulfthimm, do you know what needed for the image builder to work nicely on AWS?

[arleif-dfactory]

Interestingly, I found this in the terraform.tfstate file. Note the expiry time. Is something failing to acquire a new token once that one expires?

{
      "module": "module.registry",
      "mode": "data",
      "type": "aws_ecr_authorization_token",
      "name": "gitpod_registry",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "authorization_token": "xxxxxxxxxxxxxxxxxxxxxxx",
            "expires_at": "2020-09-24T23:34:41Z",
            "id": "2020-09-24 11:34:41.314343087 +0000 UTC",
            "password": "xxxxxxx",
            "proxy_endpoint": "https://xxxxx.dkr.ecr.us-east-1.amazonaws.com",
            "registry_id": "xxxxxx",
            "user_name": "AWS"
          }
        }
{
      "module": "module.registry",
      "mode": "data",
      "type": "aws_ecr_authorization_token",
      "name": "gitpod_registry",
      "provider": "provider[\"registry.terraform.io/hashicorp/aws\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "authorization_token": "xxxxxxxxxxxxxx",
            "expires_at": "2020-09-24T23:34:41Z",
            "id": "2020-09-24 11:34:41.314343087 +0000 UTC",
            "password": "xxxxxxxxxxxxxxxxxxxxxxxxx",
            "proxy_endpoint": "https://xxxx.dkr.ecr.us-east-1.amazonaws.com",
            "registry_id": "xxxxxx",
            "user_name": "AWS"
          }
        }

[arleif-dfactory]

Update:
I worked around this by generating a new token from ECR, updating the gitpod-registry Secret and restarting the image-builder pod.

This confirms that the cause is that whatever is supposed to refresh the token, is not doing it. Any help is appreciated. Till then I will keep refreshing the token myself.

[melks]

I'm also experiencing this with AWS self-hosted, but struggled to find the correct location/format to update the Secret into.

@arleif-dfactory , here's what I did, but it didn't seem to work:

1. Get my new ECR token:
   aws ecr get-login-password --region <my-region>

2. Print yaml for old secret, store it, and replace the token with the one from (1):
   kubectl get secret gitpod-registry -o yaml

3. Delete the old secret:
   kubectl delete secret gitpod-registry

4. Create the new secret from the yaml file:
   kubectl create -f gitpod-registry.yaml

5. Restart the image-builder:
   kubectl rollout restart deployment/image-builder

What am I missing here?

[arleif-dfactory]

@melks What you used is the username and password encoded, not the correct token. There has to be a more elegant way, but this is what I did:

TOKEN=`aws ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken'`
echo "{\"auths\": {\"xxxxxxxx.us-east-1.amazonaws.com/workspace-images\": {\"auth\": \"${TOKEN}\"}}}" > /tmp/conf.json
kubectl delete secret generic gitpod-registry
kubectl create secret generic gitpod-registry --from-file=.dockerconfigjson=/tmp/conf.json --type=kubernetes.io/dockerconfigjson
#restart the image-builder pod

[arleif-dfactory]

Hi,

I'm hoping to stop using the workaround and adopt a proper solution. Which leads to the question: Is this a bug, bad setup, or just a feature (ECR token refresh) not yet supported?

Can someone confirm that the token refresh is something Gitpod doesn't handle yet? Any thoughts on this capability should be added? Perhaps I can contribute towards adding the capability.

[arleif-dfactory]

@wulfthimm Any thoughts on this? It is something that will be fixed inside the image-builder? Or are you looking at an external process to manage the token?

[meysholdt]

What you used is the username and password encoded, not the correct token

sounds like something we can fix.

[supra08]

From what I found looking at some existing projects are that token-regeneration should be a way to go. There is a constant expiration time imposed on the ECR tokens that is of 12 hours and it cannot be disabled (neither should it be).
So it seems a good place to implement a token refresh mechanism that would use the AWS Access Key and the Secret, which is already in the system after the installation.

I would like to work on this.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.