Test and release Filebeat and Metricbeat elastic/helm-charts for 8.x

[joshdover]

We need to provide an updated helm-chart in https://github.com/elastic/helm-charts that supports the most recent 8.x release. At this time, we're only planning to release a single version that supports the most recent 8.x release. Updating it for future releases will be discussed separately.

An initial analysis by @cmacknz shows that there should be little to no required changes in configuration for Filebeat and Metricbeat. The helm charts on main already use 8.1.0 version. I think the only required work here is:

• Update the image tag to a more recent version, like 8.2.0 or 8.3.0 (after release)
• Test to verify that the charts work
• Release the updated chart

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[joshdover]

@nimarezainia @jlind23 This should be a very small amount of work. Should we schedule for 8.5?

[jlind23]

@joshdover i will scope label it as 8.5 for now.

[jw-maynard]

@joshdover I was directed to this issue from elastic/helm-charts#1594. It seems like the team at Elasic is phasing out helm charts in favor of a k8s operator. That makes perfect sense to me for folks who are trying to run an entire ELK stack in k8s.

In our environment though, we use Elastic Cloud and only need to deploy various beats daemon sets to our k8s clusters to export logs. Using an operator to do this seems like bringing in a backhoe when you need a hole for a fence post. Does the beats team have any plans to continue maintaining helm charts for deploying beats to k8s going forward?

[joshdover]

In our environment though, we use Elastic Cloud and only need to deploy various beats daemon sets to our k8s clusters to export logs. Using an operator to do this seems like bringing in a backhoe when you need a hole for a fence post. Does the beats team have any plans to continue maintaining helm charts for deploying beats to k8s going forward?

tl;dr this 8.x helm chart for Beats should work for all 8.x releases of Beats going forward. We're focus our efforts into deploying Agent on k8s much easier, stay tuned for updates.

Long-winded version:

Yes, definitely agree that installing an operator in a k8s cluster that you're only intending to observe / collect data from is definitely overkill. We've had similar discussions around separating the solutions for deploying Elasticsearch and Kibana from deploying data collectors, like Beats and Elastic Agent.

At this point, we've decided to release the Beats charts for 8.x which we expect not to break during the 8.x lifecycle. After this chart release, we won't be investing time into improving the helm chart itself, but will take contributions to fix any issues with it should we find any. The helm chart for Beats is quite simple and I expect minimal to no changes will be required for customers to use that chart for all 8.x Beats releases.

Going forward, we will be focus our efforts in improving the deployment experience for Elastic Agent in k8s. We don't currently have a helm chart for Agent, but the Fleet UI can generate a k8s manifest for quick set up. We know we can do more to improve this and are excited to share more as we get a bit further in closing the gap between Beats and Elastic Agent functionality.

[leehinman]

@jmlrt updated with elastic/helm-charts#1688 while I was testing my changes :-)

Closing since his work does the update and fixes some 8.x issues.