Started getting this error subdir "modules/kubernetes-addons/helm-addon" not found

[mk2134226]

Everything was ok this morning but now started seeing this error

subdir "modules/kubernetes-addons/helm-addon" not found

[beezel]

I am having the same issue this morning. Tried to re-init and getting failures. Worked fine last Friday.

To reproduce, blow out your .terraform folder and try to terraform init.

[awarner-greshamtech]

Same. We pin the module version:

  module "eks_blueprints" {
    source = "github.com/aws-ia/terraform-aws-eks-blueprints?ref=v4.27.0"

[vale21]

Same for us.
We noticed that even if we pint a specific tag (v4.27.0) and sometimes it dowloads the correct version:

Downloading git::https://github.com/aws-ia/terraform-aws-eks-blueprints.git?ref=v4.27.0 for eks-cluster.eks_blueprints_kubernetes_addons...

there are a couple of internal modules that are downloading main:

Downloading git::https://github.com/aws-ia/terraform-aws-eks-blueprints.git for eks-cluster.eks_blueprints_kubernetes_addons.portworx.helm_addon...

Downloading git::https://github.com/aws-ia/terraform-aws-eks-blueprints.git for eks-cluster.eks_blueprints_kubernetes_addons.sysdig_agent.helm_addon...

[alexoughton]

I have found that this is due to calling external modules which do not pin to a specific release. For example:

terraform-aws-eks-blueprints/modules/kubernetes-addons/main.tf

Line 391 in 80bd3b9

module "portworx" {

which calls:

https://github.com/portworx/terraform-eksblueprints-portworx-addon/blob/bf6d1f68eee97052cac105563bebf19a5e6f4512/main.tf#L2

The problem is commit 19c7cfd to main which removes the referenced directory in the external module. It appears that a coordination is required here to fix this in external modules which are called, and perhaps also publish another 4.x release to call those fixes.

Edit: i.e. what @vale21 said :-)

[awarner-greshamtech]

Given the breaking fix was the fix to head of this repository about an hour ago (as part of the move from 4 to 5) and given that another release of 4 is going to be required to fully resolve this (probably - pointing to later releases of the troublesome modules that are not pinning versions) I would think a roll back of the last commit to head of this repository ASAP would be the thing to do to fix version 4.

[awarner-greshamtech]

@bryantbiggs

[DanielCalvo]

Same here on v4.30.0-- did an apply just fine this morning EU time and now its broken.

[yehoshoua]

Same here we use TAG 4.29

[yebolenko]

We have the same issue !!! Please fix. It also happens if we clone it locally (based on the tag).
If we use the local (cloned) repo, we still receive
Error: subdir "modules/kubernetes-addons/helm-addon" not found

[bryantbiggs]

we have temporarily re-added the two sub-modules that are un-pinned in the partner addons. Please let us know if this has temporarily resolved the issues

In the meantime, we will work with the partner addons (those not defined here, just referenced here) to get their implementations updated/pinned

[alexoughton]

@bryantbiggs Yes, this appears to have resolved the issue, at least for me. Thanks for the quick action here.

[f-palma]

@bryantbiggs Yeah, it solved the issue. thanks!

[DanielCalvo]

Solved here as well -- thank you @bryantbiggs !

[bryantbiggs]

awesome - thank you all for confirming and apologies for the issues. It looks like we have a nasty little dynamic link that needs to be resolved

[bryantbiggs]

FYI - please update your EKS Blueprints module(s) to use the just released v4.32.1

We have PRs out to the partner's addons to ensure they pin the version of EKS Blueprints that they are referring to, but we have also created a static copy of their current addons and this is what is being referenced in v4.32.1 to eliminate this cyclical error. #1634

(Read on if curious to understand what occurred today)

What happened

Today we merged in a large refactoring PR that removed all module definitions from the project as part of moving towards v5 of EKS Blueprints. Normally this would have been a no-op for users since pinning module versions is always recommended - only those that were sourcing directly from the main branch, in theory, would have been affected - this is something that is outside of our control. However, due to the fact that:

1. In v4 the partner addon implementations were embedded into EKS Blueprints which means that on each Terraform initialization, Terraform reaches out to that modules definition to download its configurations
2. The partner module definitions utilize the EKS Blueprints v4 helm-addon sub-module. However, this sub-module was not pinned to a version and therefore referred EKS Blueprints main branch.

In short, when the refactoring PR was merged, it removed the helm-addon sub-module from our main branch, which created the subdir "modules/kubernetes-addons/helm-addon" not found errors reported. The sub-modules were pointing at a moving target rather than a fixed point in time, and that moving target no longer had the helm-addon sub-module definition. This circular dependency is something that the team had identified awhile back and was one of the motivations for refactoring in v5 to externalize those utility/template modules (helm-addon and irsa) into a new, standalone module that lives on its own https://github.com/aws-ia/terraform-aws-eks-blueprints-addon. However, what we did not realize nor catch in any of the testing and validation - was the fact that partner addons pointing at main would break once that definition no longer exists on main. And for this, we apologize for the disruption caused today.

What steps are we taking to avoid this in the future

• First and foremost, we no longer allow modules embedded within modules without strict pinning. In addition, in v5, the modules have been refactor to remove 3rd party dependencies and offer better isolation from unintended changes outside of the module itself (flattened the hierarchy and reduced external dependencies outside of our control)
• We have temporarily re-added the sub-modules in question that were causing the issues reported when removed in fix: Temporarily re-add internal cyclical dependencies #1632
• We have opened PRs with the partner addons to apply a version pin to the helm-addon implementation
  • fix: Pin module version used for addon portworx/terraform-eksblueprints-portworx-addon#15
  • fix: Pin module version used for addon tetratelabs/terraform-eksblueprints-tetrate-istio-addon#13
  • fix: Pin module version used for addon sysdiglabs/terraform-eksblueprints-sysdig-addon#49
• In addition, we have copied the configurations from the partner addons into their respective local modules. This is what removes the dynamic, cyclical link between EKS Blueprints -> the partner addon that contains the helm-addon module implementation -> EKS Blueprints helm-module definition. This is a no-op change since the definitions are un-changed and changing module sources in Terraform do not create changes (i.e. - destroy and re-create). This is what is contained within the v4.32.1 release

With these changes in place, users are recommended to update to v4.32.1 to avoid any disruptions in the future when the helm-addon and irsa module definitions are re-removed from main. Please note - any version < 4.32.1 will be at risk of raising the errors reported in this issue when the helm-addon and irsa modules are re-removed from main. For example - version v4.12.0 is a static copy of configurations but it contains the 3rd party addons who's definitions are pointing back to EKS Blueprints main branch.

If you have any questions or comments, please feel free to let us know here. Thank you and again, apologies for any disruptions from today's changes

[bryantbiggs]

are you using version 4.32.1?

[uje-m]

Sorry, my mistake.
there's some project that still pinned to older version

[juliangovender]

Thanks @bryantbiggs , had this issue on v4.31.0, moving to v4.32.1 resolved it.

[mambon2]

Upgrading to v4.32.1 seems to come with a weird quirk. terraform plan always shows argocd_application["addons"] and argocd_application["workload"] have changed, even after apply. Does anyone else run into this issue?

[abhijith-centrifuge]

github.com/aws-ia/terraform-aws-eks-blueprints//modules/irsa.

I am looking for above module , code is breaking because of this . It was working fine before , is this moved to somewhere