Skip to content

Implement automatic regeneration of AWS ECR tokens #2982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed

Implement automatic regeneration of AWS ECR tokens #2982

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 53 additions & 0 deletions install/aws-full-terraform/modules/registry/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,3 +106,56 @@ data "template_file" "gitpod_registry_values" {
secret_name = local.secret_name
}
}

resource "aws_iam_user" "gitpod_registry" {
name = "${var.project.name}-registry"

tags = {
project = var.project.name
}
}

resource "aws_iam_access_key" "gitpod_registry" {
user = aws_iam_user.gitpod_registry.name
}

data "template_file" "ecr_regeneration_script" {
template = file("${path.module}/template/regenerate-ecr.tpl")
vars = {
host = "${aws_ecr_repository.gitpod_registry.registry_id}.dkr.ecr.${var.region}.amazonaws.com"
secret_name = local.secret_name
region = var.region
access_key = aws_iam_access_key.gitpod_registry.id
secret_key = aws_iam_access_key.gitpod_registry.secret
}
}

resource "kubernetes_cron_job" "ecr_regeneration_cron" {
metadata {
name = "ecr_regeneration_cron"
}
spec {
concurrency_policy = "Allow"
failed_jobs_history_limit = 1
schedule = "0 */6 * * *"
starting_deadline_seconds = 10
successful_jobs_history_limit = 3
job_template {
metadata {}
spec {
backoff_limit = 2
ttl_seconds_after_finished = 10
template {
metadata {}
spec {
container {
name = "ecr-cred-helper"
image = "odaniait/aws-kubectl:latest"
command = ["/bin/sh", "-c", data.template_file.ecr_regeneration_script.rendered]
}
}
}
}
}
}
}
27 changes: 27 additions & 0 deletions install/aws-full-terraform/modules/registry/templates/regenerate-ecr.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/bin/bash

# Set the AWS auth environment variables
AWS_DEFAULT_REGION=${region}
AWS_SECRET_ACCESS_KEY=${secret_key}
AWS_ACCESS_KEY_ID=${access_key}

# Generate the auth token from the aws account
TOKEN=`aws ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken'`

# Delete the original secret
kubectl delete secret --ignore-not-found ${secret_name}

# Generate the new docker registry auth config
CONFIGJSON='{"auths": {"%s": {"auth": "%s"}}}\n'
UPDATEDCONFIG=$(printf "$CONFIGJSON" "${host}" "$TOKEN")
echo $UPDATEDCONFIG > /tmp/config.json

# Create a new kubernetes secret with the updated auth token
kubectl create secret generic ${secret_name} \
--from-file=.dockerconfigjson=/tmp/config.json \
--type=kubernetes.io/dockerconfigjson

# Update the default service account
kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"'${secret_name}'"}]}'

echo "Token regeneration complete"
54 changes: 54 additions & 0 deletions install/aws-terraform/modules/registry/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,3 +106,57 @@ data "template_file" "gitpod_registry_values" {
secret_name = local.secret_name
}
}

esource "aws_iam_user" "gitpod_registry" {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typo in esource

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are creating the IAM user here but you are not giving it any permissions so naturally it will not be able to refresh the token. what you want is the same permissions from resource "aws_iam_role_policy" "dns_manager" { or the very least "ecr:GetAuthorizationToken"

name = "${var.project.name}-registry"

tags = {
project = var.project.name
}
}

resource "aws_iam_access_key" "gitpod_registry" {
user = aws_iam_user.gitpod_registry.name
}

data "template_file" "ecr_regeneration_script" {
template = file("${path.module}/template/regenerate-ecr.tpl")
Copy link
Contributor

@komapa komapa Mar 10, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
template = file("${path.module}/template/regenerate-ecr.tpl")
template = file("${path.module}/templates/regenerate-ecr.tpl")

vars = {
host = "${aws_ecr_repository.gitpod_registry.registry_id}.dkr.ecr.${var.region}.amazonaws.com"
secret_name = local.secret_name
region = var.region
access_key = aws_iam_access_key.gitpod_registry.id
secret_key = aws_iam_access_key.gitpod_registry.secret
}
}

resource "kubernetes_cron_job" "ecr_regeneration_cron" {
metadata {
name = "ecr_regeneration_cron"
Copy link
Contributor

@komapa komapa Mar 10, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name = "ecr_regeneration_cron"
name = "ecr-regeneration-cron"

Error: metadata.0.name a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is 'a-z0-9?(.a-z0-9?)*')

}
spec {
concurrency_policy = "Allow"
failed_jobs_history_limit = 1
schedule = "0 */6 * * *"
starting_deadline_seconds = 10
successful_jobs_history_limit = 3
job_template {
metadata {}
spec {
backoff_limit = 2
ttl_seconds_after_finished = 10
template {
metadata {}
spec {
container {
name = "ecr-cred-helper"
image = "odaniait/aws-kubectl:latest"
command = ["/bin/sh", "-c", data.template_file.ecr_regeneration_script.rendered]
}
}
}
}
}
}
}

27 changes: 27 additions & 0 deletions install/aws-terraform/modules/registry/templates/regenerate-ecr.tpl
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
#!/bin/bash

# Set the AWS auth environment variables
AWS_DEFAULT_REGION=${region}
AWS_SECRET_ACCESS_KEY=${secret_key}
AWS_ACCESS_KEY_ID=${access_key}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These needs some export in order to be available in the TOKEN subshell below. With that said though, I believe it is much better if those ENV are set on the cronjob at runtime and are proper Kubernetes secrets instead. Embedding them like this in the command makes them readable to most people with access to Kubernetes as you would like to allow people to see cronjobs but NOT secrets.


# Generate the auth token from the aws account
TOKEN=`aws ecr get-authorization-token --output text --query 'authorizationData[].authorizationToken'`

# Delete the original secret
kubectl delete secret --ignore-not-found ${secret_name}

# Generate the new docker registry auth config
CONFIGJSON='{"auths": {"%s": {"auth": "%s"}}}\n'
UPDATEDCONFIG=$(printf "$CONFIGJSON" "${host}" "$TOKEN")
echo $UPDATEDCONFIG > /tmp/config.json

# Create a new kubernetes secret with the updated auth token
kubectl create secret generic ${secret_name} \
--from-file=.dockerconfigjson=/tmp/config.json \
--type=kubernetes.io/dockerconfigjson

# Update the default service account
kubectl patch serviceaccount default -p '{"imagePullSecrets":[{"name":"'${secret_name}'"}]}'

echo "Token regeneration complete"