Schedule pods on a nodegroup on which no concurrent actions are executed #7834
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cPu1
approved these changes
Jun 14, 2024
TiberiuGC
added a commit
to TiberiuGC/eksctl
that referenced
this pull request
Oct 7, 2024
…ted (eksctl-io#7834) * Schedule pods on a nodegroup on which no concurrent actions are executed * patch test assertions
yehielnetapp
added a commit
to spotinst/weaveworks-eksctl
that referenced
this pull request
Nov 13, 2024
…671) * Fix typo in publish-release.yaml * Prepare for next development iteration * Upgrade with explicit version if release version is up-to-date * Test Bottlerocket node upgrade and verify version * Add release notes for v0.175.0 (eksctl-io#7669) * Add release notes for v0.175.0 * remove empty acknowledgements section --------- Co-authored-by: yuxiang-zhang <23327251+yuxiang-zhang@users.noreply.github.com> Co-authored-by: Tibi <110664232+TiberiuGC@users.noreply.github.com> * Prepare for next development iteration (eksctl-io#7671) * Bump dependencies (eksctl-io#7668) * bump dependencies * update mocks * fix lint * bump helm * Aim for namespace uniqueness across parallel specs (eksctl-io#7680) ensure namespace uniqueness across parallel specs * Include MixedInstancesPolicy LaunchTemplate for validation * Allow GPU instance types for Windows nodes (eksctl-io#7681) * allow GPU instance type for Windows nodes * update unit test for case gpus:0 * Display full draft release notes in PR description (eksctl-io#7686) Update release-drafter.yaml * Bump mkdocs version (eksctl-io#7696) bump mkdocs version * Add support for AMIs based on AmazonLinux2023 (eksctl-io#7684) * add support for AL2023 for EKS-managed and self-managed nodes * ensure AL2023 only supports containerd * add GPU related validations + small nits * add support for upgrades * add support for EFA * improve validations * fix lint and unit tests * update docs * add validation error for maxpods limitation * add integration tests for al2023 * improve validation message * [EKSCTL create cluster command] Authorise self-managed nodes via `aws-auth configmap` when EKS access entries are disabled (eksctl-io#7698) * Disable access entry creation for self-managed nodes on clusters with CONFIG_MAP only * fix logic for updating aws-auth configmap * Enforce `authenticationMode:CONFIG_MAP` on Outposts (eksctl-io#7699) Make authenticationMode:CONFIG_MAP default on Outposts * Add release notes for v0.176.0 (eksctl-io#7672) Co-authored-by: TiberiuGC <110664232+TiberiuGC@users.noreply.github.com> * Prepare for next development iteration * Bump dependencies Closes eksctl-io#7694 eksctl-io#7693 eksctl-io#7692 eksctl-io#7691 eksctl-io#7690 eksctl-io#7689 eksctl-io#7688 eksctl-io#7687 eksctl-io#7679 eksctl-io#7678 eksctl-io#7676 eksctl-io#7673 eksctl-io#7581 eksctl-io#7579 eksctl-io#7577 eksctl-io#7576 * Update build image tag * Bump dependencies * Fix arn build logic to support different aws partitions * Fix reusing instanceRoleARN for nodegroups authorized with access entries This changelist changes the design of creating access entries for self-managed nodegroups that use a pre-existing instanceRoleARN by creating the access entry resource outside of the CloudFormation stack by making a separate call to the AWS API. When deleting such a nodegroup, it's the user's responsibility to also delete the corresponding access entry when no more nodegroups are associated with it. This is because eksctl cannot tell if an access entry resource is still in use by non-eksctl created self-managed nodegroups. Self-managed nodegroups not using a pre-existing instanceRoleARN will continue to have the access entry resource in the CloudFormation stack, making delete nodegroup an atomic operation for most use cases. Fixes eksctl-io#7502 * Add note about deleting nodegroups * Add integration tests * Fix cluster deletion in tests * Allow nodegroup creation after a cluster subnet is deleted (eksctl-io#7714) * Preserve eksctl commands correctness when user deletes subnets * update error when subnet availability validation fails * address PR comments * Handle K8s service account lifecycle on `eksctl create/delete podidentityassociation` commands (eksctl-io#7706) * Handle K8s service account lifecycle on eksctl create/delete podidentityassociations commands * correct typo Co-authored-by: Chetan Patwal <cPu1@users.noreply.github.com> --------- Co-authored-by: Chetan Patwal <cPu1@users.noreply.github.com> * Add support for Ubuntu Pro 22.04 based EKS images (eksctl-io#7711) * feat: Add support for Ubuntu Pro 22.04 based EKS images * update schema.json * test: Add nodegroup with Ubuntu Pro 22.04 * fix integration test --------- Co-authored-by: Tibi <110664232+TiberiuGC@users.noreply.github.com> * Disable IMDSv1 in unowned integration tests * include pre-releases as full releases when drafting release notes * Add utils command to migrate `iamidentitymappings` to EKS access entries (eksctl-io#7710) * Added migrate-to-access-entry cmd structure * Fix Target Authentication mode validation * Added logic to get accessEntries and cmEntries from cluster * Added logic to make unique list of configmap accessEntries, and stack creation logic * Added UpdateAuthentication mode and aeEntries filter logic * Add approve flag check * Added functionality to remove awsauth after switch to API only * Adds logic to fetch FullARN of path stripped IAMIdentityMappings * Updates some info log text * Adds test case and refactors code * Removes comments * Adds taskTree and address PR comments * Refactors code and Adds exception handling for NoSuchEntityException * Resolves go.mod and go.sum conflicts * Doc update for migrate-to-access-entry feature * Fixed minimum iam policies doc to add permission for iam:GetUser * Updated access-entries doc at migrate-to-access-entry section * Fixes failing Migrate To Access Entry Test & go.mod, go.sum * Amends migrate to access entry documentation * improve logs and simplify code logic * add unit tests * ensure target-auth-mode has a valid value --------- Co-authored-by: Pankaj Walke <advaitt@amazon.com> Co-authored-by: Venkat Penmetsa <vpenmets@amazon.com> Co-authored-by: Venkat Penmetsa <vpenmets@gmail.com> Co-authored-by: Tibi <110664232+TiberiuGC@users.noreply.github.com> * Revert "[Release drafter] Treat RCs as full releases when drafting notes" (eksctl-io#7725) * Fix creating pod identities Replaces usage of a per-loop variable with a per-iteration variable. * Fix deleting pod identities * Fix deleting clusters with a non-active status * Add release notes for v0.177.0 * update release notes for 0.177.0 * Prepare for next development iteration * Update aws-node from 1.12.6 to 1.18.1 (eksctl-io#7756) * Update aws-node from 1.12.6 to 1.18.1 1.18.1 is recommended for EKS clusters, where its documented that "For all Kubernetes releases, we recommend installing the latest VPC CNI release." as read at https://github.com/aws/amazon-vpc-cni-k8s?tab=readme-ov-file#recommended-version. The latest available addon for various k8s minor versions are listed at https://docs.aws.amazon.com/eks/latest/userguide/managing-vpc-cni.html#updating-vpc-cni-add-on, and it currently sais 1.18.1 for k8s 1.23 to 1.29. * Update tests for aws-node 1.18.1 * Reduce complexity of aws-node test * Fix kubeletExtraConfig support for AL2023 * Add release notes for 0.178 * Prepare for next development iteration * Support EKS 1.30 * Add release notes for v0.179.0 * Prepare for next development iteration * Add option to create service account for pod identities which defaults to `false` (eksctl-io#7784) * only create service account if explicitly instructed to do so * only delete SAs if they were created by eksctl * fix lint * Update json schema (eksctl-io#7788) upload schema * [Pod Identity Associations] Don't allow `--create-service-account` flag when `--config-file` is set (eksctl-io#7789) don't allow --create-service-account flag when --config-file is set * Add release notes for v0.180.0 (eksctl-io#7782) * Prepare for next development iteration (eksctl-io#7791) * add new addon fields required for pod identity support * ammend create addon command to create roles for pod identity associations * ammend delete addon command to delete roles for pod identity associations * small tweaks * Support updating podIdentityAssociations for addons * Show addon.podIdentityAssociations in `get addon` * Disallow updating podidentityassociations owned by addons * Show pod identities in `get addons`, use a pointer for addon.podIdentityAssociations * Update mocks * Fix deleting the specified addon instead of all addons * Disallow deletion of addon pod identities in `delete podidentityassociation` * Show ownerARN in `get podidentityassociations` * Fix `create cluster` when iam.podIdentityAssociations is unset * Delete IAM resources when addon.podIdentityAssociations is [] * take into account that not all EKS addons will support pod IDs at launch * add validations * Migrate EKS addons to pod identity using the Addons API * add unit tests and update generated files * Migrate: ignore pod identity associations that already exist Fixes eksctl-io#7753 * add docs && tweak validation * Delete old IRSA stack in `update addon` * Add integration test for addon.podIdentityAssociations * add integration tests for creating and deleting addons && bugfixes around validations and error checking * update describe addon config command to return pod identity config * add auto-create-pod-identity-associations CLI flag * update unit tests * update list of minimum IAM permissions * tech debt - unskip tests from PI suite * fix addons integration test * Allow updating addons with recommended IAM policies, disallow setting tags and wellKnownPolicies * Add more validation * Rename fields to addonsConfig.autoApplyPodIdentityAssociations and addon.useDefaultPodIdentityAssociations * Update AWS SDK * use service level endpoint resolver instead of global endpoint resolver which was deprecated * Update link to docs * Disallow IRSA config if addon has existing pod identity associations * Add release notes for v0.181.0 * Prepare for next development iteration * Fix formatting for notes in documentation * apply same formatting fix for addons.md file * G6 support * Subnets availability validation should use AZs resolved by `EC2::DescribeSubnets` call (eksctl-io#7816) Subnets availability validation should use AZs resolved by EC2::DescribeSubnets call * Update pkg/addons/assets/efa-device-plugin.yaml Co-authored-by: Chetan Patwal <cPu1@users.noreply.github.com> * Update pkg/addons/assets/efa-device-plugin.yaml Co-authored-by: Chetan Patwal <cPu1@users.noreply.github.com> * Fix upgrading AL2 ARM64 nodegroups * fix typo for iam policy * update aws-node to latest version * coredns script should exclude preview versions * Add release notes for v0.182.0 * Prepare for next development iteration * Make EKS 1.30 the default * Fix tests * Add release notes for v0.183.0 * Prepare for next development iteration * Stop using P2 instances which will be retired (eksctl-io#7826) stop using P2 instances which will be retired * Schedule pods on a nodegroup on which no concurrent actions are executed (eksctl-io#7834) * Schedule pods on a nodegroup on which no concurrent actions are executed * patch test assertions * use string in logging instead of wrapping error * Fix SDK paginator mocks The latest version of `aws-sdk-go-v2/service/eks` breaks unit tests. This [changelist](aws/aws-sdk-go-v2#2682) added SDK-specific feature tracking where all paginated operations now pass an additional argument (`addIsPaginatorUserAgent`) to add `UserAgentFeaturePaginator` to the user agent. The mocks, however, do not expect this variadic argument to be passed, resulting in failing assertions. Fixes eksctl-io#7845 * Allow cluster creation without default networking addons * Install default addons as EKS managed addons * Add integration tests and unit tests * Do not patch VPC CNI ServiceAccount to use IRSA if disableDefaultAddons is set * Honour the wait field when creating addons * Do not restart VPC CNI DaemonSet pods * Fix running kube-proxy on AL2023 nodes * Fix addon integration tests * Add documentation * Fix integration tests * Reorder addons task * Fix tests * Fix CRUD test * Add release notes for v0.184.0 * Prepare for next development iteration * fixed iam permissions for karpenter Signed-off-by: Sienna Satterwhite <sienna.satterwhite@jamf.com> * fix run as root efa device plugin bug The plugins were globally changed to have runAsNonRoot set to true. This breaks the efa plugin, which currently requires it. This PR was tested and confirmed to fix the bug in several cases. Signed-off-by: vsoch <vsoch@users.noreply.github.com> * add support for hpc7g arm images Signed-off-by: sochat1 <sochat1@llnl.gov> * update efa-device-plugin.yaml to one that workkks Signed-off-by: sochat1 <sochat1@llnl.gov> * add additional hpc7g instance types Signed-off-by: vsoch <vsoch@users.noreply.github.com> * Add auto-ssm ami resolution for ubuntu Issue eksctl-io#3224 * Avoid creating subnets in disallowed Availability Zone IDs * Add release notes for v0.185.0 * Prepare for next development iteration * Refactor: move bare cluster validation to NewCreateClusterLoader * Retry throttling errors, disable retry rate-limiting * Allow limiting the number of nodegroups created in parallel * Add release notes for v0.186.0 * Prepare for next development iteration * Restrict `VPC.SecurityGroup` egress rules validations to self-managed nodes (eksctl-io#7883) Restrict VPC.SecurityGroup egress rules validations to self-managed nodes * Add release notes for v0.187.0 (eksctl-io#7885) Co-authored-by: Tibi <110664232+TiberiuGC@users.noreply.github.com> * Prepare for next development iteration (eksctl-io#7890) * Add GH workflow for automatically updating nvidia device plugin static manifest (eksctl-io#7898) * Add GH workflow for automatically updating nvidia device plugin static manifest * update PR body * fix unit tests * updates userdocs * Add support for Kuala Lumpur region (ap-southeast-5) (eksctl-io#7910) * Update nvidia-device-plugin to v0.16.0 (eksctl-io#7900) update nvidia-device-plugin to v0.16.0 Co-authored-by: TiberiuGC <110664232+TiberiuGC@users.noreply.github.com> * Bump github.com/docker/docker from 24.0.9+incompatible to 26.1.4+incompatible (eksctl-io#7909) Bump github.com/docker/docker Co-authored-by: Tibi <110664232+TiberiuGC@users.noreply.github.com> * Add release notes for v0.188.0 (eksctl-io#7889) add release notes for v0.188.0 Co-authored-by: Tibi <110664232+TiberiuGC@users.noreply.github.com> * Prepare for next development iteration (eksctl-io#7917) * Fix SSM unit tests * fix: resolve segfault in validateBareCluster Signed-off-by: Mike Frisch <mikef17@gmail.com> * Skip creating OIDC manager for Outposts clusters * Add release notes for v0.189.0 * Prepare for next development iteration * Bump github.com/docker/docker Bumps [github.com/docker/docker](https://github.com/docker/docker) from 26.1.4+incompatible to 26.1.5+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](moby/moby@v26.1.4...v26.1.5) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com> * Bump jinja2 from 3.1.3 to 3.1.4 in /userdocs Bumps [jinja2](https://github.com/pallets/jinja) from 3.1.3 to 3.1.4. - [Release notes](https://github.com/pallets/jinja/releases) - [Changelog](https://github.com/pallets/jinja/blob/main/CHANGES.rst) - [Commits](pallets/jinja@3.1.3...3.1.4) --- updated-dependencies: - dependency-name: jinja2 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * Add release notes for v0.190.0 * Prepare for next development iteration * Prepare for next development iteration * Add support for EKS 1.31 (eksctl-io#7973) add support for eks 1.31 * Add release notes for v0.191.0 (eksctl-io#7965) Co-authored-by: tiberiugc <tiberiucopaciu@gmail.com> * Prepare for next development iteration * cleanup efa installer archive before install Currently, the UserData section that runs during cloud init happens before any root volumes are expanded with growpart. Although the best solution would be to ensure the filesystem resize happens before these scripts are run, a quick means to fix the current issue is simply to cleanup the efa installer tar.gz, which is very large. I have tested this with hpc7g for a size 2 and size 8 cluster (previously both not working) and can confirm the devices are functioning after. Signed-off-by: vsoch <vsoch@users.noreply.github.com> * efa-installer: remove archive in 2023 files Problem: the node consistently runs out of disk space when adding efa, resulting in an unusable cluster with scattered nodes where the installer failed. Solution: the installer archive itself is huge, and we can simply remove it and avoid this error. Signed-off-by: vsoch <vsoch@users.noreply.github.com> * Disallow `overrideBootstrapCommand` and `preBootstrapCommands` for MNG AL2023 (eksctl-io#7990) disallow overrideBootstrapCommand and preBootstrapCommands for MNG AL2023 * Add support for EKS accelerated AMIs based on AL2023 (eksctl-io#7996) add support for EKS accelerated AMIs based on AL2023 * Add release notes for v0.192.0 (eksctl-io#7974) Co-authored-by: TiberiuGC <110664232+TiberiuGC@users.noreply.github.com> * Prepare for next development iteration (eksctl-io#7997) * Add support for M8g instance types Signed-off-by: cpu1 <patwal.chetan@gmail.com> * Correct version drift in cluster-upgrade.md Correct the description of version drift during upgrade to match the current kubernetes documentation. Node version should not be newer than the cluster version. * Add release notes for v0.193.0 * Prepare for next development iteration * Fix missing ELB listener attribute actions required for AWS Load Balancer Controller v2.9.0 * Support EKS zonal shift config Signed-off-by: cpu1 <patwal.chetan@gmail.com> * Fix tests Signed-off-by: cpu1 <patwal.chetan@gmail.com> * Add release notes for v0.194.0 * after clean compile * fix merge .. cloudformation error while creating ocean nodegroup * actual work to add field in ocean cluster config * restore .goreleaser.yml to root --------- Signed-off-by: Sienna Satterwhite <sienna.satterwhite@jamf.com> Signed-off-by: vsoch <vsoch@users.noreply.github.com> Signed-off-by: sochat1 <sochat1@llnl.gov> Signed-off-by: Mike Frisch <mikef17@gmail.com> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: cpu1 <patwal.chetan@gmail.com> Co-authored-by: Yu Xiang Z <yxz.get@gmail.com> Co-authored-by: eksctl-bot <53547694+eksctl-bot@users.noreply.github.com> Co-authored-by: Yu Xiang Zhang <yuxz@amazon.com> Co-authored-by: yuxiang-zhang <23327251+yuxiang-zhang@users.noreply.github.com> Co-authored-by: Tibi <110664232+TiberiuGC@users.noreply.github.com> Co-authored-by: Weifeng Wang <qclaogui@gmail.com> Co-authored-by: Chetan Patwal <cPu1@users.noreply.github.com> Co-authored-by: cpu1 <patwal.chetan@gmail.com> Co-authored-by: TimAndy <xuchonglei@126.com> Co-authored-by: cPu1 <patwal.chetan@gmail.comm> Co-authored-by: Alberto Contreras <aciba90@gmail.com> Co-authored-by: punkwalker <126026317+punkwalker@users.noreply.github.com> Co-authored-by: Pankaj Walke <advaitt@amazon.com> Co-authored-by: Venkat Penmetsa <vpenmets@amazon.com> Co-authored-by: Venkat Penmetsa <vpenmets@gmail.com> Co-authored-by: Erik Sundell <erik.i.sundell@gmail.com> Co-authored-by: cPu1 <6298307+cPu1@users.noreply.github.com> Co-authored-by: tiberiugc <tiberiucopaciu@gmail.com> Co-authored-by: AI2Table <76231236+ai2table@users.noreply.github.com> Co-authored-by: Practicus AI <76231236+practicusai@users.noreply.github.com> Co-authored-by: Wei Zang <richzw@gmail.com> Co-authored-by: Andres More <moreandres@users.noreply.github.com> Co-authored-by: Sienna Satterwhite <sienna.satterwhite@jamf.com> Co-authored-by: vsoch <vsoch@users.noreply.github.com> Co-authored-by: sochat1 <sochat1@llnl.gov> Co-authored-by: Alberto Contreras <alberto.contreras@canonical.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Mike Frisch <mikef17@gmail.com> Co-authored-by: Martin Harriman <larvacea@mac.com> Co-authored-by: Jonathan Foster <jonathan@jonathanfoster.io>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Fixes #7792
CRUD integration test creates a bunch of nodegroups, on which, some of the specs, then execute different operations e.g. scale, delete, etc. Other parallel specs apply daemonsets, deployments, etc. Depending on timing, one spec might delete and/or drain a node onto which pods were scheduled by another spec, right when the namespace in which the pods are running is being terminated.
To fix ☝🏻, this PR ensures that all pods are scheduled on a separate nodegroup, onto which no other concurrent action is being executed.
Manual testing
https://github.com/eksctl-io/eksctl-ci/actions/runs/9510689795/job/26218234718
Checklist
README.md, or theuserdocsdirectory)area/nodegroup) and kind (e.g.kind/improvement)BONUS POINTS checklist: complete for good vibes and maybe prizes?! 🤯