[EKS] [request]: EKS Managed Nodes should allow for custom security groups

[bhops]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
I would like to be able to add additional security groups to the managed EKS nodes.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
EKS Managed nodes do not support the ability to specify custom security groups to be added to the worker nodes. The only access controls we have are the ability to pass an existing security group, which will be given access to port 22, or to not specify security groups, which allows access to port 22 from 0.0.0.0/0. This is either too restrictive (we would like to whitelist IP ranges), or too open (0.0.0.0/0 is not secure). This also prevents our ability to access services running on the nodes directly (not in kubernetes), if we were to have access to modify the userdata and install software (assuming #596 gets implemented).

Additional context
While I haven't directly tested this, this might also prevent the use of NodePort type services in Kubernetes because external resources will not be able to access the chosen port.

[dnascimento]

The managed nodes feature also implied that EKS now needs to have IAM role to create security groups, which isn't allowed by many organisations. If a custom security group is selectable, then the permission to create SG sgould be removed

[pierresteiner]

We create the network load balancer with terraform instead of k8s; so that we can dynamically link it with CloudFront.

We can't do that anymore without the ability to add security group

[mikestef9]

We are working on adding launch template support to managed node groups which will allow you to specify additional security groups.

Looking for feedback on whether to add it to the managed nodes API as well. Will adding additional security groups directly to the API allow you to use managed node groups? Or are there other fields as part of launch templates that would also be required for use?

[yann-soubeyrand]

@mikestef9 being able to add additional security groups to managed node groups will definitely unlock one of the two blockers for us, the second being the ability to specify target groups which the node group's instances should be part of.

[mikestef9]

@yann-soubeyrand What is the use case for needing to specify target groups as opposed to using an ingress controller or specifying a load balancer in your service manifest? Could you open a separate feature request on this roadmap with more details there.

[yann-soubeyrand]

@mikestef9 Unless I'm mistaken, one can not use a single NLB for several K8s services of type load balancer. For example, we want to be able to point ports 80 and 443 to our ingress controller service, but we also want port 22 to the SSH service of our GitLab.

Also we want to be able to share our NLB between classic EC2 instances and EKS cluster to be able to do a zero downtime migration from the stateless application running on EC2 instances to the same application running on an EKS cluster.

And the last use case we have is sharing a NLB between two EKS clusters (blue and green) to be able to seamlessly switch from one to the other (in case we have big changes to bring to our cluster, we prefer spawning a new cluster and switching to it after having tested that it works as intended).

[alexgavrisco]

We're also quite interested in this feature at Adobe. It became a blocker for us in migration to managed nodes (due to the way ALB Ingress Controller worked), there's a workaround, but we would prefer a more reliable way to deal with security groups and managed nodes.

[manfredlift]

We're also quite interested in this feature at Adobe. It became a blocker for us in migration to managed nodes (due to the way ALB Ingress Controller worked), there's a workaround, but we would prefer a more reliable way to deal with security groups and managed nodes.

We are also interested in this feature:

1. We have a similar use-case as @Alexx-G, we prefer a Terraform managed ALB as ALB Ingress Controller has some limitations at the moment. (I feel we might have tried the same workaround as in attaching security group rules to the automatically created remote_access_security_group_id)
2. Another use-case of allowing managed workers access to AWS MSK from its security group. (I am sure there are many other AWS Services that have source_security_group_id based permissioning)

[yann-soubeyrand]

@mikestef9 What's your feedback on these use cases? Do they appear to be valid use cases for you? We'd like to have you opinion on these as we'll have to find another solution in case target groups attachment isn't going to be implemented as part of managed node groups ;-)

[mikestef9]

Hi @yann-soubeyrand definitely a valid use case. I closed the ticket you opened as a duplicate, will track the feature request under #709

[yann-soubeyrand]

@mikestef9 Thanks for your feedback and sorry for the duplicate!

[jwhiting]

I want to specify a security group shared between EKS managed node groups and RDS instances. To work around the lack of security group control, I've added ingress by CIDR block of the worker node subnets to the RDS security group. This is harder to manage and more prone to mistakes in access control than using a dedicated security group.

[vbem]

As one of the largest customers in AWS region cn-north-1, we also need this feature.

[jerry123je]

we are facing the same issue.

[whereisaaron]

We've been test Managed Node Groups and the inability to attach or associate pre-existing Security Groups to the nodes is a blocker for us. We can't even attach the Security Group required for nodes to get NAT access, so we're kind of dead in the water on Managed Nodes until the launch templates feature surfaces.

Only being able to add security groups via launch templates might be ok, especially if eksctl config file hides the complication. But it seems a basic enough requirement for a node group that it should be on this API.

It seems odd right now there is this generous facility on the API to associate a whole list of security groups, but only for ssh access, and nothing for any other use case.

(The other one missing for us is being able to associate Target Group ARNs with Managed Node Groups at creation and ideally update later. Though that limitation is easier to work around.)

[nahuel242]

+1 to this feature

[jayyanar]

+1 to this feature. This is great disadvantage if you already have a EKS with UnManaged Node and its attached Security Group (SG) attached to other SG or Cross Account.
If we not able to map the same SG as part of Managed Node group. This is another hurdle to migration to Managed Nodegroup and also EKS Migration.
Since EKS 1.13 migration is just around the corner, we need this issue to be fixed soon.

[odg0318]

+1 to this feature. I don't totally understand at all why custom security group on managed node group is not implemented before managed node group was released. Custom security group is very very basic option with EC2 as far as I know. I strongly feel that I am just a kind of beta user paying costs.

When will be the custom launch template released?

[Hostersi]

+1 We need this as well.

[jen-gineer]

Can we allow a pre-existing security group to be specified in the creation/upgrading of the EKS cluster for both the control plane and managed node groups (in-place of the ClusterSecurityGroup)? Therefore, the EKS IAM role will no longer need permissions to create/update SG’s. This would help us comply with our company's segregation of duties policies.

[cdenneen]

Can we allow a pre-existing security group to be specified in the creation/upgrading of the EKS cluster for both the control plane and managed node groups (in-place of the ClusterSecurityGroup)? Therefore, the EKS IAM role will no longer need permissions to create/update SG’s. This would help us comply with our company's segregation of duties policies.

if your source is a computer over vpn or direct connect which doesn't belong to a SG cannot use nested SG but if we can attach pre-existing SG right to node that would work but not for Control Plane I don't think

[mikestef9]

Closing as this feature request is addressed by launch template support. See #585 for details!

One important note for custom security groups with launch templates, if you specify custom security groups in a launch template, EKS will not add the cluster security group, and you are responsible for making sure the necessary rules are present that allow communication between control plane and data plane. See docs for more details.

[shinebayar-g]

It could be much better UX if EKS allowed attaching additional security groups directly when creating node groups rather than forcing people to create their own launch templates.

[rtripat]

It could be much better UX if EKS allowed attaching additional security groups directly when creating node groups rather than forcing people to create their own launch templates.

That's totally valid feedback but let me explain our reasoning behind it. An Ec2 instance (1 or many via ASG) have several configuration parameters which are best captured in an Ec2 launch template. EKS is taking a tenet to prevent such configuration parameters from "leaking" into our APIs. Without this, our API will soon need to mimic the CreateLaunchTemplate API and worse we will always play catch up with new features released in EC2.

As a result, we keep the configuration parameters for an ASG to an absolute minimum in EKS API and ask customers to give us a launch template instead.

[tomfotherby]

EKS is taking a tenet to prevent such configuration parameters from "leaking" into our APIs. Without this, our API will soon need to mimic the CreateLaunchTemplate API and worse we will always play catch up with new features released in EC2.

@rtripat , As a engineer, I 100% understand where you're coming from, and would propably say something similar. But as a EKS customer, it kind of rubs me up the wrong way. We want a good product with a good user experience and TBH, it doesn't matter if a useful feature makes more work for AWS, or makes implementation harder, that's what we're paying for. Managed worker groups are cool, not easily being able to assign existing security groups is not cool. We aren't asking to mimic the whole CreateLaunchTemplate API, we are asking for the majority use-case to be implemented. Whilst it isn't what I wanted to hear, it is great to hear the thinking behind decisions, and I really appreciate the transparency 👍 .

[lgg42]

EKS is taking a tenet to prevent such configuration parameters from "leaking" into our APIs. Without this, our API will soon need to mimic the CreateLaunchTemplate API and worse we will always play catch up with new features released in EC2.

@rtripat , As a engineer, I 100% understand where you're coming from, and would propably say something similar. But as a EKS customer, it kind of rubs me up the wrong way. We want a good product with a good user experience and TBH, it doesn't matter if a useful feature makes more work for AWS, or makes implementation harder, that's what we're paying for. Managed worker groups are cool, not easily being able to assign existing security groups is not cool. We aren't asking to mimic the whole CreateLaunchTemplate API, we are asking for the majority use-case to be implemented. Whilst it isn't what I wanted to hear, it is great to hear the thinking behind decisions, and I really appreciate the transparency +1 .

Couldn't agree more with this comment. In order for us to use custom security groups with Managed Node Groups (so we can use Security Groups for pods) I can't find any "managed" part in the whole thing. Could be better off just using normal self-managed nodes.

[Vadim-Zenin]

After 3 years it is still not sorted out...
I hope AWS attention to customer needs do not became as low as Azure and Google.