Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[aws-eks] Kubernetes resources fail to create if fargate profiles are not stable #8854

Closed
eladb opened this issue Jul 2, 2020 · 0 comments · Fixed by #8859
Closed

[aws-eks] Kubernetes resources fail to create if fargate profiles are not stable #8854

eladb opened this issue Jul 2, 2020 · 0 comments · Fixed by #8859
Assignees
@eladb @iliapolo
Labels
@aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service bug This issue is a bug.
Milestone
[DevPreview] @aws...

Comments

@eladb
Copy link
Contributor

eladb commented Jul 2, 2020

When trying to deploy kubectl resources (such as KubernetesResource or HelmChart) against an EKS cluster with Fargate Profiles that are still being created, the API server may not be reachable. This causes these resources to to fail, the deployment to roll back and worst of all, as the rollback happens, CFN tries to delete the FargateProfile resources, but they still being creating. Basically leading to unrecoverable situation.

Environment

  • CLI Version :
  • Framework Version: 1.48.0
  • Node.js Version:
  • OS :
  • Language (Version):

This is 🐛 Bug Report
@eladb eladb added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Jul 2, 2020
@eladb eladb self-assigned this Jul 2, 2020
@github-actions github-actions bot added the @aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service label Jul 2, 2020
eladb pushed a commit that referenced this issue Jul 2, 2020
fix(eks): kubectl resources fail before fargate profiles are created
f9193c3 
When a Fargate profile is being created, the Kubernetes API server in EKS sometimes rejects requests. This means that kubectl-related resources such as KubernetesResources Helm charts may fail during deployment.

To address this, we add a "barrier resource" (in the form of an SSM parameter) which waits for all fargate profiles to be created before allowing kubectl resources to continue. This is done by the barrier taking a dependency on all FargateProfile resources and all kubectl resources taking a dependency on the barrier.

Fixes #8854


This commit also fixes #8574 by adding `iam:ListAttachedRolePolicies` to the cluster's creation role IAM policy.
@eladb eladb mentioned this issue Jul 2, 2020
Merged
1 task
eladb pushed a commit that referenced this issue Jul 2, 2020
fix(eks): kubectl resources fail before fargate profiles are created
28f2717 
When a Fargate profile is being created, the Kubernetes API server in EKS sometimes rejects requests. This means that kubectl-related resources such as KubernetesResources Helm charts may fail during deployment.

To address this, we add a "barrier resource" (in the form of an SSM parameter) which waits for all fargate profiles to be created before allowing kubectl resources to continue. This is done by the barrier taking a dependency on all FargateProfile resources and all kubectl resources taking a dependency on the barrier.

Fixes #8854


This commit also fixes #8574 by adding `iam:ListAttachedRolePolicies` to the cluster's creation role IAM policy.
@eladb eladb mentioned this issue Jul 2, 2020
Merged
@mergify mergify bot closed this as completed in #8859 Jul 2, 2020
mergify bot pushed a commit that referenced this issue Jul 2, 2020
fix(eks): kubectl resources fail before fargate profiles are created (#…
4fad9bc 
…8859)

When a Fargate profile is being created, the Kubernetes API server in EKS sometimes rejects requests. This means that kubectl-related resources such as KubernetesResources Helm charts may fail during deployment.

To address this, we add a "barrier resource" (in the form of an SSM parameter) which waits for all fargate profiles to be created before allowing kubectl resources to continue. This is done by the barrier taking a dependency on all FargateProfile resources and all kubectl resources taking a dependency on the barrier.

Fixes #8854


This commit also fixes #8574 by adding `iam:ListAttachedRolePolicies` to the cluster's creation role IAM policy.


----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@iliapolo iliapolo added this to the EKS Dev Preview milestone Aug 10, 2020
@iliapolo iliapolo removed the needs-triage This issue or PR still needs to be triaged. label Aug 16, 2020
@iliapolo iliapolo changed the title [eks] Kubernetes resources fail to create if fargate profiles are not stable [aws-eks] Kubernetes resources fail to create if fargate profiles are not stable Aug 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@eladb eladb

@iliapolo iliapolo

Labels
@aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service bug This issue is a bug.
Projects
None yet
Milestone
[DevPreview] @aws-cdk/aws-eks
Development

Successfully merging a pull request may close this issue.

fix(eks): kubectl resources fail before fargate profiles are created aws/aws-cdk
2 participants
@eladb @iliapolo