Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(eks): kubectl resources fail before fargate profiles are created #8855

Merged
merged 1 commit into from
Jul 2, 2020

Conversation

eladb
Copy link
Contributor

@eladb eladb commented Jul 2, 2020

When a Fargate profile is being created, the Kubernetes API server in EKS sometimes rejects requests. This means that kubectl-related resources such as KubernetesResources and Helm charts may fail during deployment.

To address this, we add a "barrier resource" (in the form of an SSM parameter) which waits for all fargate profiles to be created before allowing kubectl resources to continue. This is done by the barrier taking a dependency on all FargateProfile resources and all kubectl resources taking a dependency on the barrier.

Fixes #8854

This commit also fixes #8574 by adding iam:ListAttachedRolePolicies to the cluster's creation role IAM policy.



By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

When a Fargate profile is being created, the Kubernetes API server in EKS sometimes rejects requests. This means that kubectl-related resources such as KubernetesResources Helm charts may fail during deployment.

To address this, we add a "barrier resource" (in the form of an SSM parameter) which waits for all fargate profiles to be created before allowing kubectl resources to continue. This is done by the barrier taking a dependency on all FargateProfile resources and all kubectl resources taking a dependency on the barrier.

Fixes #8854


This commit also fixes #8574 by adding `iam:ListAttachedRolePolicies` to the cluster's creation role IAM policy.
@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Jul 2, 2020
@eladb eladb requested a review from iliapolo July 2, 2020 06:36
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-qxepHUsryhcu
  • Commit ID: f9193c3
  • Result: FAILED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

provider = new KubectlProvider(this.stack, uid);

// allow the kubectl provider to assume the cluster creation role.
this._clusterResource.addTrustedRole(provider.role);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You moved this inside the singleton creation, but won't this mean that if two clusters are defined on the same stack, one of them wont have the trusted role?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, nice catch! Let me add a test and fix

@eladb eladb merged commit f9193c3 into benisrae/improve-cr-errors Jul 2, 2020
@eladb eladb deleted the benisrae/eks-wait-for-fargate branch July 2, 2020 10:51
@eladb
Copy link
Contributor Author

eladb commented Jul 2, 2020

Oops... Accidentally merged this into benisrae/improve-cr-errors.

@eladb
Copy link
Contributor Author

eladb commented Jul 2, 2020

New PR #8859

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution/core This is a PR that came from AWS.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants