Add support for multi node kubernetes cluster

[MichaelKatsoulis]

After the addition of a proposed kubernetes manifest for fleet managed elastic (manifest) and the update of elastic-package to fetch this manifest from beats repo(#452, #461), elastic package deploys agent as Daemonset to test the integrations.

It would be nice the daemonset to be deployed in a multi node kubernetes cluster.
This way we can have multiple agents running (one in each node). Kubernetes integrations features such as leader election(elastic/integrations#1321) and conditions for properly getting metrics from controller manager and scheduler (elastic/integrations#1324) can be tested.

The code currently does support that.
Things to consider:

1. All nodes of kind cluster must be connected to elastic-package network
2. After agents are deployed, elastic package tries to get the readiness of the pods by filtering with kind-control-plane as the node as agent was supposed to run only that node. (https://github.com/elastic/elastic-package/blob/master/internal/testrunner/runners/system/servicedeployer/kubernetes.go#L107)
3. The policies must be applied in all running agents

[mtojek]

Must haves:

• elastic-package should check if all agents using the same policy picked up the revision (transition from rev. 1 to rev. 2)
• elastic-package should wait until all agent's k8s resources are installed (see log below - elastic-agent waits until the API lists any k8s agent, bu doesn't wait on resource installation, outcome of recent changes)

[2021-08-05T04:08:00.881Z] 2021/08/05 04:08:00 DEBUG Apply Kubernetes stdin
[2021-08-05T04:08:00.881Z] 2021/08/05 04:08:00 DEBUG run command: /var/lib/jenkins/workspace/gest-manager_integrations_master/bin/kubectl apply -f - -o yaml
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG Handle "apply" command output
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG Extract resources from command output
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG Wait for ready resources
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG Sync resource info: elastic-agent (kind: DaemonSet, namespace: kube-system)
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG Sync resource info: elastic-agent (kind: ClusterRoleBinding, namespace: )
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG Sync resource info: elastic-agent (kind: RoleBinding, namespace: kube-system)
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG Sync resource info: elastic-agent-kubeadm-config (kind: RoleBinding, namespace: kube-system)
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG Sync resource info: elastic-agent (kind: ClusterRole, namespace: )
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG Sync resource info: elastic-agent (kind: Role, namespace: kube-system)
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG Sync resource info: elastic-agent-kubeadm-config (kind: Role, namespace: kube-system)
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG Sync resource info: elastic-agent (kind: ServiceAccount, namespace: kube-system)
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG beginning wait for 8 resources with timeout of 10m0s
[2021-08-05T04:08:01.456Z] 2021/08/05 04:08:01 DEBUG install custom Kubernetes definitions (directory: /var/lib/jenkins/workspace/gest-manager_integrations_master/src/github.com/elastic/integrations/packages/kubernetes/data_stream/apiserver/_dev/deploy/k8s)
[2021-08-05T04:08:01.457Z] 2021/08/05 04:08:01 DEBUG no custom definitions found (directory: /var/lib/jenkins/workspace/gest-manager_integrations_master/src/github.com/elastic/integrations/packages/kubernetes/data_stream/apiserver/_dev/deploy/k8s). Nothing else will be installed.
[2021-08-05T04:08:01.457Z] 2021/08/05 04:08:01 DEBUG GET http://127.0.0.1:5601/api/fleet/agents
[2021-08-05T04:08:02.027Z] 2021/08/05 04:08:01 DEBUG filter agents using criteria: NamePrefix=kind-control-plane
[2021-08-05T04:08:02.027Z] 2021/08/05 04:08:01 DEBUG found 0 enrolled agent(s)
[2021-08-05T04:08:02.972Z] 2021/08/05 04:08:02 DEBUG GET http://127.0.0.1:5601/api/fleet/agents
[2021-08-05T04:08:02.972Z] 2021/08/05 04:08:02 DEBUG filter agents using criteria: NamePrefix=kind-control-plane
[2021-08-05T04:08:02.972Z] 2021/08/05 04:08:02 DEBUG found 0 enrolled agent(s)
[2021-08-05T04:08:03.911Z] 2021/08/05 04:08:03 DEBUG GET http://127.0.0.1:5601/api/fleet/agents
[2021-08-05T04:08:03.911Z] 2021/08/05 04:08:03 DEBUG filter agents using criteria: NamePrefix=kind-control-plane
[2021-08-05T04:08:03.911Z] 2021/08/05 04:08:03 DEBUG found 0 enrolled agent(s)
[2021-08-05T04:08:05.302Z] 2021/08/05 04:08:04 DEBUG GET http://127.0.0.1:5601/api/fleet/agents
[2021-08-05T04:08:05.302Z] 2021/08/05 04:08:04 DEBUG filter agents using criteria: NamePrefix=kind-control-plane
[2021-08-05T04:08:05.302Z] 2021/08/05 04:08:04 DEBUG found 0 enrolled agent(s)
[2021-08-05T04:08:06.244Z] 2021/08/05 04:08:05 DEBUG GET http://127.0.0.1:5601/api/fleet/agents
[2021-08-05T04:08:06.244Z] 2021/08/05 04:08:05 DEBUG filter agents using criteria: NamePrefix=kind-control-plane
[2021-08-05T04:08:06.244Z] 2021/08/05 04:08:05 DEBUG found 0 enrolled agent(s)
[2021-08-05T04:08:07.185Z] 2021/08/05 04:08:06 DEBUG GET http://127.0.0.1:5601/api/fleet/agents
[2021-08-05T04:08:07.185Z] 2021/08/05 04:08:06 DEBUG filter agents using criteria: NamePrefix=kind-control-plane
[2021-08-05T04:08:07.185Z] 2021/08/05 04:08:06 DEBUG found 0 enrolled agent(s)
[2021-08-05T04:08:08.128Z] 2021/08/05 04:08:07 DEBUG GET http://127.0.0.1:5601/api/fleet/agents
[2021-08-05T04:08:08.128Z] 2021/08/05 04:08:07 DEBUG filter agents using criteria: NamePrefix=kind-control-plane
[2021-08-05T04:08:08.129Z] 2021/08/05 04:08:07 DEBUG found 0 enrolled agent(s)
[2021-08-05T04:08:09.073Z] 2021/08/05 04:08:08 DEBUG GET http://127.0.0.1:5601/api/fleet/agents
[2021-08-05T04:08:09.073Z] 2021/08/05 04:08:09 DEBUG filter agents using criteria: NamePrefix=kind-control-plane
[2021-08-05T04:08:09.073Z] 2021/08/05 04:08:09 DEBUG found 0 enrolled agent(s)
[2021-08-05T04:08:10.015Z] 2021/08/05 04:08:10 DEBUG GET http://127.0.0.1:5601/api/fleet/agents
[2021-08-05T04:08:10.275Z] 2021/08/05 04:08:10 DEBUG filter agents using criteria: NamePrefix=kind-control-plane
[2021-08-05T04:08:10.275Z] 2021/08/05 04:08:10 DEBUG found 0 enrolled agent(s)

(source)

[mtojek]

Hey @MichaelKatsoulis ! Do you have any update around this issue?

PS Feel free to unassign this if you don't plan to work on this now.

[MichaelKatsoulis]

Hey @MichaelKatsoulis ! Do you have any update around this issue?

PS Feel free to unassign this if you don't plan to work on this now.

Hey @mtojek! Unfortunately I won't have time to work on this until 8.0 Release.
If you or anyone else have time and want to work on that before that time it is fine by me.
If not I plan to work on that in November.