Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

(eks): LoadBalancer type of Service failed to start in the EKS cluster created by CDK #12166

Closed
yjw113080 opened this issue Dec 20, 2020 · 1 comment · Fixed by #12269
Closed

(eks): LoadBalancer type of Service failed to start in the EKS cluster created by CDK #12166

yjw113080 opened this issue Dec 20, 2020 · 1 comment · Fixed by #12269
Assignees
@iliapolo
Labels
@aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service bug This issue is a bug. effort/medium Medium work item – several days of effort p1

Comments

@yjw113080
Copy link

yjw113080 commented Dec 20, 2020 •
edited
Loading

Right after I create an EKS cluster with CDK I tried to provision a Service Object with LoadBalancer Type. I got the following error when I describe the service:

  Warning  SyncLoadBalancerFailed  6s (x4 over 43s)  service-controller  Error syncing load balancer: failed to ensure load balancer: Multiple tagged security groups found for instance i-0f13b228f51aa3c4c; ensure only the k8s security group is tagged; the tagged groups were sg-01e539b21c944bd17(ClusterStack-us-west-2-demogoclusterspotgroupInstanceSecurityGroupA4E3F5E6-13FWAGRXOSPPM) sg-02e6abb7dd3641358(eks-cluster-sg-demogo-1335419381)

I added ASG capacity with Spot Instances and CDK assigned two Security Groups with kubernetes.io/cluster/CLUSTER_NAME=owned tags.
From one of the Security Groups, I delete the tag (ref: kubernetes/kubernetes#73906) then it worked.
Changing the tag to shared did not work for me.

Reproduction Steps

  1. Create cluster and CI/CD resources using this: https://github.com/yjw113080/aws-cdk-eks-multi-region
  2. I applied Kubernetes Resources including Deployment and Service (Type=LoadBalancer)

What did you expect to happen?

Create Service without any issue.

What actually happened?

The service went pending.

Environment

  • CDK CLI Version : 1.79.0 (build 8b3bf9c)
  • Framework Version: 1.79.0
  • Node.js Version: v14.7.0
  • OS : Mojave 10.14.6
  • Language (Version): TypeScript (4.0.2)

Other

This is 🐛 Bug Report
@yjw113080 yjw113080 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Dec 20, 2020
@github-actions github-actions bot added the @aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service label Dec 20, 2020
@yjw113080 yjw113080 changed the title (eks): short issue description (eks): LoadBalancer type of Service failed to start in the EKS cluster created by CDK Dec 20, 2020
@iliapolo iliapolo mentioned this issue Dec 29, 2020
Merged
@iliapolo iliapolo added effort/medium Medium work item – several days of effort p1 and removed needs-triage This issue or PR still needs to be triaged. labels Dec 29, 2020
@mergify mergify bot closed this as completed in #12269 Dec 31, 2020
mergify bot pushed a commit that referenced this issue Dec 31, 2020
@iliapolo
fix(eks): Self managed nodes cannot be added to LoadBalancers created…
470a881 
… via the `LoadBalancer` service type (#12269)

Following this [PR](#12042), self managed nodes are now attached with the cluster security group. This causes the self managed nodes to have multiple security groups with the "owned" tag. This in turn causes load balancers to reject these instances since its unable to determine which security groups should be added with ingress rules to allow the load balancer to connect to the instances.

The fix is to exclude tagging the dedicated ASG security group with this tag, it is no longer necessary since the cluster security group has that tag by default. 

Fixes #12166

This breaksge is unfortunate, but I can't see a way out of it. And it does actually fix a bug.

BREAKING CHANGE: Existing self managed nodes may loose the ability to host additional services of type `LoadBalancer` . See #12269 (comment) for possible mitigations.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

flochaz pushed a commit to flochaz/aws-cdk that referenced this issue Jan 5, 2021
@iliapolo @flochaz
fix(eks): Self managed nodes cannot be added to LoadBalancers created…
4b7e863 
… via the `LoadBalancer` service type (aws#12269)

Following this [PR](aws#12042), self managed nodes are now attached with the cluster security group. This causes the self managed nodes to have multiple security groups with the "owned" tag. This in turn causes load balancers to reject these instances since its unable to determine which security groups should be added with ingress rules to allow the load balancer to connect to the instances.

The fix is to exclude tagging the dedicated ASG security group with this tag, it is no longer necessary since the cluster security group has that tag by default. 

Fixes aws#12166

This breaksge is unfortunate, but I can't see a way out of it. And it does actually fix a bug.

BREAKING CHANGE: Existing self managed nodes may loose the ability to host additional services of type `LoadBalancer` . See aws#12269 (comment) for possible mitigations.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iliapolo iliapolo

Labels
@aws-cdk/aws-eks Related to Amazon Elastic Kubernetes Service bug This issue is a bug. effort/medium Medium work item – several days of effort p1
Projects
None yet
Milestone
No milestone
2 participants
@iliapolo @yjw113080