Refactor testing to leverage core chart-testing capabilities and enable multiple root level charts

[marcofranssen]

This refactor leverages the chart-testing ci values feature (https://github.com/helm/chart-testing/blob/main/pkg/chart/chart.go#L571-L605).

Doing so we can also support multiple root level charts.

Some of the tests should in that case move into the specific root level chart once we start to break out those charts.

This also allows to run the tests locally on a kubecluster of your choice (active current-context). Could be Kind, Rancher, EKS, anything.

make test
make test-examples
make test-example-production
make test-example-tornjak
…

To ignore certain tests in the chart ci folder, you can rename them to anything that doesn't match the pattern *-values.yaml. e.g. oidc-values-ignore.yaml. That eases debugging single tests.

[kfox1111]

Discussion of something that affects this PR started here:
https://spiffe.slack.com/archives/C04BURZP1KK/p1688575164862469

[kfox1111]

If this change goes forward, I will lose the ability to migrate my customers onto this project. I've mentioned before, they see Helm as a package manager of an Application, and minimally they see the "Server, Agent, Container Manager / Registrar, and SPIFFE CSI Drivers" as part of the core application. They don't mind seeing Tornjak as a sub-chart, and they are considering possibly using sub-charts for their database deployments (which are currently heavily managed by terraform as they are Athena databases).

We should not make the server an Application, and the agent a different application and then require that they both be tied together by a sub-chart style deployment. I would ask that we reconsider moving the server and the agent, minimally, together into the root chart.

This style of application breakup within a chart is not uncommon. For example, gitlab on prem is often used by enterprise customers and is architected with subcharts:
https://gitlab.com/gitlab-org/charts/gitlab/-/tree/master/charts
Same with kube-prometheus-stack:
https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/Chart.yaml#L41-L57

Is the pushback related to a feeling that it shouldn't be done that way, or is there evidence that harm is done when structured that way? Would it help to get feedback from the helm dev team itself, speaking to the legitimacy or illegitimacy of using nested charts for this sort of application?

[faisal-memon]

If this change goes forward, I will lose the ability to migrate my customers onto this project. I've mentioned before, they see Helm as a package manager of an Application, and minimally they see the "Server, Agent, Container Manager / Registrar, and SPIFFE CSI Drivers" as part of the core application. They don't mind seeing Tornjak as a sub-chart, and they are considering possibly using sub-charts for their database deployments (which are currently heavily managed by terraform as they are Athena databases).

We should not make the server an Application, and the agent a different application and then require that they both be tied together by a sub-chart style deployment. I would ask that we reconsider moving the server and the agent, minimally, together into the root chart.

@edwbuck We will still support the aggregated chart, just provide options for people to deploy components separately for advanced deployments. Functionally it will still be the same, implementation details would be different. Anything specific that would prevent adoption by your customers?

[edwbuck]

@edwbuck We will still support the aggregated chart, just provide options for people to
deploy components separately for advanced deployments. Functionally it will still be the
same, implementation details would be different. Anything specific that would prevent
adoption by your customers?

Basically they see a cluster as an Application. The only exception to this is their root clusters (they run on nested spire) where the root cluster is also an Application, but the child clusters are again single Applications with an additional (optional) Agent that is part of the root cluster. For their standalone clusters, they use the child cluster with the "upstream" agent turned off.

They rely heavily on simple / easy to interpret templates, and they are not keen on having to dig into one Chart's file to ensure it coordinates with another Chart's file, but they are more inclined to grep across multiple .tpl files in the same Chart.

Part of that is because they also have monitoring and management through other tooling that would over-report each component as a different application, preventing them from getting the single view of the entire cluster. That also impacts their DataDog collection system, as the multiple deployments for a single conceptual cluster would affect their auto-labeling system such that the metrics would be more disjoint.

In short, my requests to at least have the core components of SPIRE be delivered in one Helm Chart (the extras can easily be sub-charted, they'll likely deploy them independently anyway) are not just interesting design ideas, it's what they want.

Currently, they are committed to maintaining their own charts, and their commentary on this project isn't worth repeating. Let's just say they're not fans.

[marcofranssen]

@edwbuck We will still support the aggregated chart, just provide options for people to
deploy components separately for advanced deployments. Functionally it will still be the
same, implementation details would be different. Anything specific that would prevent
adoption by your customers?

Basically they see a cluster as an Application. The only exception to this is their root clusters (they run on nested spire) where the root cluster is also an Application, but the child clusters are again single Applications with an additional (optional) Agent that is part of the root cluster. For their standalone clusters, they use the child cluster with the "upstream" agent turned off.

They rely heavily on simple / easy to interpret templates, and they are not keen on having to dig into one Chart's file to ensure it coordinates with another Chart's file, but they are more inclined to grep across multiple .tpl files in the same Chart.

Part of that is because they also have monitoring and management through other tooling that would over-report each component as a different application, preventing them from getting the single view of the entire cluster. That also impacts their DataDog collection system, as the multiple deployments for a single conceptual cluster would affect their auto-labeling system such that the metrics would be more disjoint.

In short, my requests to at least have the core components of SPIRE be delivered in one Helm Chart (the extras can easily be sub-charted, they'll likely deploy them independently anyway) are not just interesting design ideas, it's what they want.

Currently, they are committed to maintaining their own charts, and their commentary on this project isn't worth repeating. Let's just say they're not fans.

This PR is not even breaking out charts at this stage. This chart is all about using the industry standard tooling instead of all our custom scripting. Lets have that discussion of breaking out charts somewhere else. Even if we are not breaking out charts this refactor cleans up already +-140 lines of custom plumbing code.

[marcofranssen]

If this change goes forward, I will lose the ability to migrate my customers onto this project. I've mentioned before, they see Helm as a package manager of an Application, and minimally they see the "Server, Agent, Container Manager / Registrar, and SPIFFE CSI Drivers" as part of the core application. They don't mind seeing Tornjak as a sub-chart, and they are considering possibly using sub-charts for their database deployments (which are currently heavily managed by terraform as they are Athena databases).

We should not make the server an Application, and the agent a different application and then require that they both be tied together by a sub-chart style deployment. I would ask that we reconsider moving the server and the agent, minimally, together into the root chart.

So those same customers of yours, do they demand the same from any opensource project. Telling them on how to organize their Go packages, and if such a project to decide to move things into a separate library etc? Maybe we should ask the spire project to get rid of the spiffe module and put that in the same spire repo as well, because that would for me also be easier to find the code as opposed to pulling in a dependency managed in another repo 🤷‍♂️ .

What we are talking about here is internals. The public API for them will still be the same. They still install the spire Helm chart. They still provide the same values. We can't have others decide how we organize things internally for other usecases. So in terms of golang we are talking about things like. Put all in the same package vs having multiple packages vs having external dependencies.

[edwbuck]

If this change goes forward, I will lose the ability to migrate my customers onto this project. I've mentioned before, they see Helm as a package manager of an Application, and minimally they see the "Server, Agent, Container Manager / Registrar, and SPIFFE CSI Drivers" as part of the core application. They don't mind seeing Tornjak as a sub-chart, and they are considering possibly using sub-charts for their database deployments (which are currently heavily managed by terraform as they are Athena databases).
We should not make the server an Application, and the agent a different application and then require that they both be tied together by a sub-chart style deployment. I would ask that we reconsider moving the server and the agent, minimally, together into the root chart.

So those same customers of yours, do they demand the same from any opensource project. Telling them on how to organize their Go packages, and if such a project to decide to move things into a separate library etc? Maybe we should ask the spire project to get rid of the spiffe module and put that in the same spire repo as well, because that would for me also be easier to find the code as opposed to pulling in a dependency managed in another repo 🤷‍♂️ .

What we are talking about here is internals. The public API for them will still be the same. They still install the spire Helm chart. They still provide the same values. We can't have others decide how we organize things internally for other usecases. So in terms of golang we are talking about things like. Put all in the same package vs having multiple packages vs having external dependencies.

I will reach back to the customer with this messaging. Thank you. While the two charts appear very different, which is stalling them from adopting, if the differences are truly internal, this will help with future migration efforts.

[kfox1111]

I still have concerns its taking so long to do runs, but since we think we can get it reduced in follow on patches, I'll go along with that.

The rest looks ok.

[kfox1111]

Fixed the merge conflict