[e2e-tests] feat: add k8s integration tests

[pkoutsovasilis]

What does this PR do?

Introduces mage integration:kubernetes target that performs integration testing of elastic-agent on Kubernetes. Currently it supports provisioning k8s cluster only through kind.

Inside the integration:kubernetes the following sequence happens:

1. Provision of the corresponding stack and k8s clusters based on the defined K8S_VERSION (this can specify a single or multiple k8s versions separated with a , character)
2. Invocation of the Kubernetes-related tests that require the following Group and OS

    info := define.Require(t, define.Requirements{
    Stack: &define.Stack{},
    Local: false,
    Sudo:  false,
    OS: []define.OS{
        {Type: define.Kubernetes},
      },
      Group: define.Kubernetes,
    })
   

Following the above paradigm, this PR also includes a k8s integration test that renders the corresponding kustomize template of a standalone elastic-agent and checks that respective pods are getting deployed successfully and waits until the get to a Ready state with all containers Running.

PS: check the K8s integration tests step in the buildkite CI

Why is it important?

Because it will allow properly testing elastic-agent, including the several deployment manifests atm, on Kubernetes

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

N/A

How to test this PR locally

Prerequisites:

• kind installed
• docker installed
• authenticated in GCP and ESS by mage integration:auth

Steps:

• mage integration:auth
• DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES=docker mage -v package
• K8S_VERSION="v1.30.2,v1.29.4,v1.28.9" INSTANCE_PROVISIONER=kind STACK_PROVISIONER=stateful SNAPSHOT=true mage integration:kubernetes

Related issues

• Closes Kubernetes e2e tests #4984

[pkoutsovasilis]

@cmacknz @blakerouse this is a PR that follows the proposed design details in the comments of this issue. Could you take a quick look around and see if you spot anything that is way-off?

PS1: The logic of dynamically provisioning stacks and instances through the framework from inside a CI runner, even after coding it, still feels kinda awkward to me. Maybe this is because in my prior-experience so far this provisioning was the responsibility of an buildkite equivalent layer and the internal testing framework was just checking that the required resources were actually provisioned and then it was invoking the respective tests. Either way, here we go 🙂

PS2: thanks @alexsapran for showing me the whereabouts of buildkite and helping me understand how to introduce these new kubernetes integration steps

[alexsapran]

PS1: The logic of dynamically provisioning stacks and instances through the framework from inside a CI runner, even after coding it, still feels kinda awkward to me. Maybe this is because in my prior-experience so far this provisioning was the responsibility of an buildkite equivalent layer and the internal testing framework was just checking that the required resources were actually provisioned and then it was invoking the respective tests. Either way, here we go 🙂

From my Point of View, it should be decoupled steps. The testing framework should only trigger the tests, and the provisioning and setup of the prerequisites should live in dedicated targets; then, CI should pull together all of those into an execution step. Now, for developer simplicity, we could have a dedicated target that could wrap around those steps, but within CI, there should be distinct targets.

[blakerouse]

I think this is an excellent start and I really like how this is shaping up. I really only have one major comment about the additions in pkg/component, other than that this is exactly what I was wanting for extending the framework to support kubernetes.

[blakerouse]

Changes look good, so far.

I guess an actual real test is next?

[pkoutsovasilis]

Changes look good, so far.

I guess an actual real test is next?

yep it is this time!!! an actual k8s deployment test that waits at least for all agent pods to become Running and Ready

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[blakerouse]

This is fantastic!

It even tests multiple versions of the kind cluster. Such a great starting point for testing on k8s.

[blakerouse]

I think as more tests are written some of the code in this test could be split out into helpers that other tests will use. Helpers would really help with reading this code as well.

Not a blocker, would like to see this merged. I see changes like helpers evolving over more tests.

[pkoutsovasilis]

thanks for the proposal @blakerouse , I was thinking something similar; defining a custom "k8sclient" struct with the helper functions attached so within a test you can get them from info.KubeClient(). However this felt like another PR to follow 🙂

[cmacknz]

@elastic/ingest-eng-prod FYI