e2e multicluster tests

[nrfox]

There needs to be e2e tests to ensure you can use the Sail operator and APIs to setup a multicluster istio deployment. The tests need to cover setting up:

1. multi-primary
2. primary-remote
3. external controlplane

These e2e tests should:

• Create two separate kind clusters
• Install the sail operator/crds into each
• Follow the upstream documentation for setting up the above multicluster deployments but use the Sail CRs rather than istioctl install ...: https://istio.io/latest/docs/setup/install/multicluster/multi-primary/
  • A shared trust will need to be created for both clusters: https://istio.io/latest/docs/setup/install/multicluster/before-you-begin/
• Ensure that istio is setup correctly after install

[nrfox]

Here are the steps to setup multi-primary in kind with Sail which can be automated if they aren't already in the e2e tests:

1. Install and start kind load balancer: https://kind.sigs.k8s.io/docs/user/loadbalancer/ and https://github.com/kubernetes-sigs/cloud-provider-kind?tab=readme-ov-file#how-to-use-it

2. Download istioctl

3. Create shared trust

mkdir -p certs
pushd certs
make -f istioctl_dir/tools/certs/Makefile.selfsigned.mk root-ca
make -f istioctl_dir/tools/certs/Makefile.selfsigned.mk cluster1-cacerts
make -f istioctl_dir/tools/certs/Makefile.selfsigned.mk cluster2-cacerts
popd

4. Create kind clusters

kind create cluster --name cluster1
kind create cluster --name cluster2

5. Install sail operator on each cluster:

kubectl config use-context kind-cluster1
make deploy
kubectl config use-context kind-cluster2
make deploy

6. Create istio-system namespace on each cluster and add intermediate CAs to each.

kubectl create namespace istio-system --context kind-cluster1
kubectl --context=kind-cluster1 label namespace istio-system topology.istio.io/network=network1
kubectl create secret generic cacerts -n istio-system --context kind-cluster1 \
      --from-file=certs/cluster1/ca-cert.pem \
      --from-file=certs/cluster1/ca-key.pem \
      --from-file=certs/cluster1/root-cert.pem \
      --from-file=certs/cluster1/cert-chain.pem

kubectl create namespace istio-system --context kind-cluster2
kubectl --context=kind-cluster2 label namespace istio-system topology.istio.io/network=network2
kubectl create secret generic cacerts -n istio-system --context kind-cluster2 \
      --from-file=certs/cluster2/ca-cert.pem \
      --from-file=certs/cluster2/ca-key.pem \
      --from-file=certs/cluster2/root-cert.pem \
      --from-file=certs/cluster2/cert-chain.pem

7. Create Sail CR on cluster1

kubectl apply --context kind-cluster1 -f - <<EOF
apiVersion: operator.istio.io/v1alpha1
kind: Istio
metadata:
  name: default
spec:
  version: v1.21.0
  namespace: istio-system
  updateStrategy:
    type: InPlace
    inactiveRevisionDeletionGracePeriodSeconds: 30
  values:
    pilot:
      resources:
        requests:
          cpu: 100m
          memory: 1024Mi
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster1
      network: network1
EOF

8. Create east-west gateway on cluster1

istioctl_dir/samples/multicluster/gen-eastwest-gateway.sh \
    --mesh mesh1 --cluster cluster1 --network network1 | \
    istioctl --context="kind-cluster1" install -y -f -

9. Expose services on cluster1

kubectl --context="kind-cluster1" apply -n istio-system -f \
    istioctl_dir/samples/multicluster/expose-services.yaml

10. Create Sail CR on cluster2

kubectl apply --context kind-cluster2 -f - <<EOF
apiVersion: operator.istio.io/v1alpha1
kind: Istio
metadata:
  name: default
spec:
  version: v1.21.0
  namespace: istio-system
  updateStrategy:
    type: InPlace
    inactiveRevisionDeletionGracePeriodSeconds: 30
  values:
    pilot:
      resources:
        requests:
          cpu: 100m
          memory: 1024Mi
    global:
      meshID: mesh1
      multiCluster:
        clusterName: cluster2
      network: network2
EOF

11. Create east-west gateway on cluster2

istioctl_dir/samples/multicluster/gen-eastwest-gateway.sh \
    --mesh mesh1 --cluster cluster2 --network network2 | \
    istioctl --context="kind-cluster2" install -y -f -

12. Expose services on cluster1

kubectl --context="kind-cluster2" apply -n istio-system -f \
    istioctl_dir/samples/multicluster/expose-services.yaml

13. Install a remote secret in cluster2 that provides access to cluster1’s API server.

CLUSTER2_CONTAINER_IP=$(kubectl get nodes cluster1-control-plane --context "kind-cluster1" -o jsonpath='{.status.addresses[?(@.type == "InternalIP")].address}')
istioctl create-remote-secret \
  --context="kind-cluster1" \
  --name=cluster1 \
  --server="https://${CLUSTER2_CONTAINER_IP}:6443" | \
  kubectl apply -f - --context="kind-cluster2"

14. Install a remote secret in cluster1 that provides access to cluster2’s API server.

CLUSTER1_CONTAINER_IP=$(kubectl get nodes cluster2-control-plane --context "kind-cluster2" -o jsonpath='{.status.addresses[?(@.type == "InternalIP")].address}')
istioctl create-remote-secret \
  --context="kind-cluster2" \
  --name=cluster2 \
  --server="https://${CLUSTER1_CONTAINER_IP}:6443" | \
  kubectl apply -f - --context="kind-cluster1"

15. Verify

kubectl create --context="kind-cluster1" namespace sample
kubectl create --context="kind-cluster2" namespace sample
kubectl label --context="kind-cluster1" namespace sample istio-injection=enabled
kubectl label --context="kind-cluster2" namespace sample istio-injection=enabled
kubectl apply --context="kind-cluster1" \
    -f samples/helloworld/helloworld.yaml \
    -l service=helloworld -n sample
kubectl apply --context="kind-cluster2" \
    -f samples/helloworld/helloworld.yaml \
    -l service=helloworld -n sample
kubectl apply --context="kind-cluster1" \
    -f istioctl_dir/samples/helloworld/helloworld.yaml \
    -l version=v1 -n sample
kubectl apply --context="kind-cluster2" \
    -f istioctl_dir/samples/helloworld/helloworld.yaml \
    -l version=v2 -n sample
kubectl apply --context="kind-cluster1" \
    -f istioctl_dir/samples/sleep/sleep.yaml -n sample
kubectl apply --context="kind-cluster2" \
    -f istioctl_dir/samples/sleep/sleep.yaml -n sample

kubectl exec --context="kind-cluster1" -n sample -c sleep \
    "$(kubectl get pod --context="kind-cluster1" -n sample -l \
    app=sleep -o jsonpath='{.items[0].metadata.name}')" \
    -- curl -sS helloworld.sample:5000/hello

kubectl exec --context="kind-cluster2" -n sample -c sleep \
    "$(kubectl get pod --context="kind-cluster2" -n sample -l \
    app=sleep -o jsonpath='{.items[0].metadata.name}')" \
    -- curl -sS helloworld.sample:5000/hello

[fjglira]

I can add this test if @nrfox is ok

[nrfox]

@fjglira sure!

[fjglira]

Already created a Multi-Primary - Multi-Network test in this PR. Working on: refactor kubectl util and add Primary-Remote - Multi-Network test