Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support Windows for Warm Pools #290

Merged
merged 4 commits into from
May 4, 2021
Merged

feat: Support Windows for Warm Pools #290

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
94ca55c
feat: Support Windows for Warm Pools
backjo May 4, 2021
7fcdb39
clarify requirements for windows instances using warm pools
backjo May 4, 2021
b0e7412
remove CRLF character
backjo May 4, 2021
6cf774b
only run user-data on first and second startup
backjo May 4, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions controllers/provisioners/eks/helpers.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -124,8 +124,16 @@ func (ctx *EksInstanceGroupContext) GetBasicUserData(clusterName, args string, k
[string]$EKSBinDir = "$env:ProgramFiles\Amazon\EKS"
[string]$EKSBootstrapScriptName = 'Start-EKSBootstrap.ps1'
[string]$EKSBootstrapScriptFile = "$EKSBinDir\$EKSBootstrapScriptName"
& $EKSBootstrapScriptFile -EKSClusterName {{ .ClusterName }} -KubeletExtraArgs '{{ .KubeletExtraArgs }}' 3>&1 4>&1 5>&1 6>&1
{{range $post := .PostBootstrap}}{{$post}}{{end}}
[string]$IMDSToken=(curl -UseBasicParsing -Method PUT "http://169.254.169.254/latest/api/token" -H @{ "X-aws-ec2-metadata-token-ttl-seconds" = "21600"} | % { Echo $_.Content})
[string]$InstanceID=(curl -UseBasicParsing -Method GET "http://169.254.169.254/latest/meta-data/instance-id" -H @{ "X-aws-ec2-metadata-token" = "$IMDSToken"} | % { Echo $_.Content})
[string]$Lifecycle = Get-ASAutoScalingInstance $InstanceID | % { Echo $_.LifecycleState}
if ($Lifecycle -like "*Warmed*") {
Echo "Not starting Kubelet due to warmed state."
backjo marked this conversation as resolved.
Show resolved Hide resolved
& C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 –Schedule
} else {
& $EKSBootstrapScriptFile -EKSClusterName {{ .ClusterName }} -KubeletExtraArgs '{{ .KubeletExtraArgs }}' 3>&1 4>&1 5>&1 6>&1
{{range $post := .PostBootstrap}}{{$post}}{{end}}
}
</powershell>`
case OsFamilyBottleRocket:
UserDataTemplate = `
Expand Down
5 changes: 3 additions & 2 deletions docs/EKS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -412,9 +412,10 @@ spec:
You can configure your scaling group to use [AWS Warm Pools for Auto Scaling](https://docs.aws.amazon.com/autoscaling/ec2/userguide/ec2-auto-scaling-warm-pools.html), which allows you to keep a capacity separate pool of stopped instances have already run any pre-bootstrap userdata - using warm pools can reduce the time it takes for nodes to join the cluster.

Warm Pools is not officially supported with EKS, hence the following requirements exist in order to use it with EKS:
- Your AMI must have `awscli` & `jq`, you can also install it in pre-bootstrap userdata.
- For Amazon Linux 2 instances, your AMI must have `awscli` & `jq`, you can also install it in pre-bootstrap userdata.
- For Windows instances, your AMI must have the `Get-ASAutoScalingInstance` cmdlet installed.
- If you use your own IAM role for the instance group, you must make sure it has access to `DescribeAutoScalingInstances`, this is required in order to figure out the current lifecycle state within userdata. If you are provisioning your IAM role through the controller, simply be aware that the controller will add the managed policy `AutoScalingReadOnlyAccess` to the role it creates.
- This is currently only supported for AmazonLinux2 based AMIs.
- This is currently only supported for AmazonLinux2 and Windows based AMIs.

We hope to get rid of these requirements in the future once AWS offers native support for Warm Pools in EKS, these are currently required in order to avoid warming instances to join the cluster briefly while they are provisioned, we are able to avoid this problem by checking if the instances is in the Warmed* lifecycle, and skipping bootstrapping in that case. Skipping on these requirements and using warm pools might mean having nodes join the cluster when the are being warmed, and cause unneeded scheduling of pods on nodes that are about to shutdown.

Expand Down