[wip] Decouple CAPM3 from metal3-dev-env

[adilGhaffarDev]

User Story

Right now CAPM3 CI is heavily dependent on metal3-dev-env, from environment setup to BMHs creation all is done by metal3-dev-env. We should move it to CAPM3 and also simplify it. Use Kustomize to deploy ironic and BMO instead of cloning repos. Similar work is already done in capm3 clusterctl upgrade tests:

Detailed Description
The following tasks need to be done in order to accomplish this:
(Let's do them in order so it does not break the CI)

• Launch management cluster:
  NOTE: Right now we are launching minikube cluster for Centos and kind for Ubuntu. In the case of kind cluster we deploy ironic in the container instead of inside the cluster. We should move away from kind and only use Minikube for both Ubuntu and Centos.

  • Define supported versions for BMO and Ironic in test/e2e/config/e2e_conf.yaml. Please check dev-env for right versions
  • Create Kustomize files for BMO and ironic in test/e2e/data folder if they are missing.
  • Launch management minikube cluster, if we want to implement it go check this kind provider. Here is how we are doing it in dev-env
  • Create metal3 namespace
  • Deploy Ironic with Kustomize, example
  • Deploy BMO with Kustomize, example

• Create BMHs
  Right now dev-env is creating BMHs in the metal3 namespace which are used by all tests. We should create BMHs at the start of each test and destroy them in the cleanup of each test.

  • Change dev-env to skip bmh deploy in case of capm3 e2e tests. PR: (Add option to skip apply BMHs metal3-dev-env#1463)
  • Change CAPM3 tests so they deploy their own bmhs and not rely on dev-env
    • clusterctl upgrade tests PR: 🌱 delete the process of removing bmhs from default namespace #2039
    • basic integration test
    • integration test
    • ip-reuse test
    • live-iso test( lets remove this test for capm3, its more related to BMO and we are testing it there)PR: 🌱 Removing live-iso test #2041
    • CAPI remediation test
    • pivoting test
    • node remediation test
    • k8s upgrade test

[adilGhaffarDev]

@mquhuy please add tasks related to creating VMs with golang library that we have added in BMO.

[adilGhaffarDev]

cc @Rozzii @kashifest @tuminoid please check

[Rozzii]

/triage accepted