[EKS] [request]: Enable PodNodeSelector Admission Controller for EKS

[toran414]

Tell us about your request
It would be nice if it was possible to use PodNodeSelector in EKS.

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
In our EKS clusters, we are looking to leverage namespaces to represent our deployment environments. In order to do this safely, we want to label different auto-scaling groups of nodes as production and non-production. We then could use have the clusterDefaultNodeSelector select the non-production nodes, and the prod namespace target production nodes.

Are you currently working around this issue?
We are currently considering 2 workarounds.

1. Having our CD pipeline inject a nodeSelector into each deployment manifest based on the namespace being deployed to.
2. Writing a MutatingAdmissionWebhook to handle node selection, but this seems like we would just be re-writing the built-in PodNodeSelector.

[whereisaaron]

The PodNodeSelector is an old-skool, compiled-in Admission Controller. A MutatingAdmissionWebhook is the more contemporary way to implement this functionality and requires no messing with the control plane configuration. So you are doing the right thing there I think. Plus your out-of-tree Admission Controller can be more flexible to your needs. Implementing this replacement would be good thing. Node selectors and thus PodNodeSelector should eventually be deprecated and replaced by affinity (once more efficient).

But if it were easy to enable PodNodeSelector right now, it would be a quicker fix for your issue.

[kinghajj]

Another solution I've considered using: give each auto scaling group a different taint. Say the current production environment is called "release 2019-05-01", then one ASG would have the taint release=2019-05-01:NoSchedule, and pods for that release have a toleration {"key": "release", "operator": "Equals", "value": "2019-05-01", "effect": "NoSchedule"}. For new staging releases, you'd make another ASG with a unique taint, and have its pods tolerate it. Thus, pods from the different releases couldn't intermingle on the same node.

[omerfsen]

So right now even with the latest EKS 1.13 eks.2 we don't have "PodNodeSelector" feature right?

[omerfsen]

My requirement is that to run pods on nodes on default namespace which has label lifecycle=Ec2Spot and run dashboard,grafana,prometheus etc.. (management stuff) on ASG labeled lifecycle=OnDemand so to achieve that

https://stackoverflow.com/questions/52487333/how-to-assign-a-namespace-to-certain-nodes

[rosscdh]

+1

[excieve]

Now that there are managed node groups in 1.14 this would be super useful to automatically restrict pods to certain groups only.

[liangrog]

For those whom still want podnodesselector functionality but can't be bothered to write the admission webhook controllers: https://github.com/liangrog/admission-webhook-server

[pierluigilenoci]

@tabern is there any hope that it will be activated on EKS soon?

[AceHack]

Yes, any update?

[AceHack]

Any update, this is no longer alpha

[tabern]

We're taking a look at this.

Update 11/11
We took a deeper look at this. Because the namespaces that are declared for this controller are statically defined in config files that are on the api server, it is not something that we can easily enable today. We're keeping this on our backlog and will continue to evaluate the feasibility of properly enabling this controller in the future.

[derhally]

Is there any other solution to this problem? We have a 3rd party helm chart that doesn't cleanly handle setting tolerations and affinity. We would need everything to run on a specific node group and this would have solved the problem

[jogoldberg]

Any news on this?

[pierluigilenoci]

There really is a need for this feature.
It would be great to have it on EKS. ❤️

[iverberk]

@tabern AKS supports this admission controller and it is an easy solution to an otherwise (more) complicated setup with custom mutation webhooks (be that OPA or something else). It solves quite an important aspect of enforcing multi-tenancy on a single cluster. Can you maybe elaborate on why the admission controller can't be made to work within the EKS context?

[rmgpinto]

We would like to use this as well.

[roshan1988]

Any updates on this feature yet ? Would make life much easier if this is supported

[avarf]

This is a great feature, please add this to EKS.

[pierluigilenoci]

This would be a great feature but AWS makes feature requests based on how many 👍🏻 the issue receives. To date, there are 44 that have more. If you want this to have a chance the only way is to get more 👍🏻 on the issue.

[avarf]

I am sure there will be more requests for this when more people and organizations start to use EKS over AWS WaveLength since you want to deploy on a specific WL and it is much easier to use PodNodeSelector at the namespace level.

[SergeyMuha]

Need this for EKS!!!

[rotemsh15]

need this for EKS.
AKS has this enabled and readily available.

[c-datculescu]

We took a deeper look at this. Because the namespaces that are declared for this controller are statically defined in config files that are on the api server, it is not something that we can easily enable today. We're keeping this on our backlog and will continue to evaluate the feasibility of properly enabling this controller in the future.

Usually noone is interested in the file format configuration, most ppl are interested in the configuration annotation format. So i think the point is rather moot as that is not the primary or the only way of configuring this.

For multi tenant clusters this is a big issue, and i would prefer to not roll my own solution if i have another option that is currently builtin.

Edit: grammar.

[thgruiz]

+1

[richardmatthewsdev]

👍

[raghulkrishna]

+1

[gaurav-kumar-curefit]

This will be a great feature and help us to solve some problems we are facing if enabled in EKS.

[DodgeCamaro]

+1

[valdemarjr]

+1

[hahasheminejad]

As iverberk mentioned, this can be achieved using Gatekeeper mutation feature. It automatically injects tolerations and nodeaffinity to pods at the deployment time.
A great documentation with ready to use configuration can be found here.
https://medium.com/google-cloud/automating-tenant-pod-scheduling-using-policy-controller-mutations-b325f4616943

[ericwang-klarrio]

Need this for EKS!!!

[eechava66]

+1

[mikestef9]

Enabling this on API server is not a priority due to challenges outlined in comment above. Instead, we have added a dedicated section to our multi-tenancy best practices guide outlining alternative options for isolating tenant workloads to specific nodes.

[pierluigilenoci]

@mikestef9 so you're telling me that AWS prefers a system for separating workloads based on developer goodwill (adding affinity & taints) or third party software (Gatekeeper, Kyverno) instead of using a native Kubernetes feature?

A feature that Azure seems to have enabled.

It has been requested for almost three years, I would say it has been in the backlog for quite a while.

[mikestef9]

We always prefer a managed built-in solution, but wanted to at least document known alternatives in the EKS best practices guide in the interim. Using the in-tree PodNodeSelector is not the direction we are leaning, but rather providing EKS integration with a policy engine, which can do pod node selection along with many other capabilities. #1435 tracks this feature.

We also think an integrated policy engine is the better answer here, given the pod node selector admission controller has been an alpha feature since Kubernetes v1.5, and EKS generally does not enable alpha features.

[mvalenzisi]

@mikestef9 the problem is that the integration with a policy engine is not there at the moment, and the community is asking for enabling a native plugin to get the job done.

[pierluigilenoci]

@mikestef9 I can understand the AWS policy but for PodNodeSelector it is really a special case.
The transition to "stable" only ran aground three years ago not for technical reasons but for lack of drive.
Ref: kubernetes/kubernetes#58818 and kubernetes/kubernetes#60284

[cmsmith7]

Hi .. Is there any update on this request?

[afagundes]

Any updates?

[ichylinux]

i came to the same challenge.
waiting for any update.

[krishnadas-m]

+1
Kindly enable this feature

[gfrid]

+1 looking for this feature :)

[abcdefstar]

+1

[aaadipop]

+100 :)

[IanAtSalt]

+1

[jgawley-videoamp]

+42

[shehbazk]

+43

[pierluigilenoci]

Instead of commenting +1, 👍🏻 the issue.
It's the only way to be taken into consideration.