In multiple clusters on a host, how do we connect from one to other cluster?

[pentago]

I'm trying to setup 3 clusters on my laptop to replicate my company's infrastructure.

One of the clusters has ArgoCD installed and when I try adding one of other 2 clusters to it, it fails because I'm not sure which IP/DNS name to use in --api-port flag to be able to connect from one cluster to another.

I tried using host.k3d.internal, host.docker.internal, hosts IP address, none worked.

Any suggestions?

For example, this is how I create ArgoCD cluster:

k3d cluster create
--servers 1
--agents 1
--api-port gitops-cluster.dev.lan:64431
--image rancher/k3s:v1.21.6-k3s1
--token superSecretToken
--timeout 60s
--port 8083:80@loadbalancer:0
--port 4433:443@loadbalancer:0
--k3s-arg "--kube-apiserver-arg=feature-gates=EphemeralContainers=true@server:0"
--k3s-arg "--kube-scheduler-arg=feature-gates=EphemeralContainers=true@server:0"
--k3s-arg "--kubelet-arg=feature-gates=EphemeralContainers=true@agent:0"
--k3s-arg "--cluster-cidr=10.1.0.0/16@server:0"
--k3s-arg "--service-cidr=10.0.0.0/16@server:0"
--k3s-arg "--etcd-disable-snapshots@server:0"
--k3s-arg "--tls-san=gitops-cluster.dev.lan@server:0"
--k3s-arg "--disable=traefik@server:0"
--k3s-arg "--disable=network-policy@server:0"
--k3s-arg "--disable=cloud-controller@server:0"
--k3s-arg="--cluster-domain=gitops-cluster.dev.lan@server:0"
--k3s-node-label tier=server@server:0
--k3s-node-label tier=worker@agent:0
--k3s-node-label worker=a@agent:0
--wait gitops

[iwilltry42]

Hi @pentago , thanks for starting this discussion!
As per our Slack discussion, I'm going to try and set this up locally for testing.
From your comment on Slack: https://gist.github.com/pentago/785dbe6251fba12bd37505daad583a57 (setup)

EDIT: How are you setting up ArgoCD and try to add clusters to it?

[iwilltry42]

@pentago , what about the ArgoCD part? I see Terraform in your Makefile and assume it deploys it, but I don't know about any configuration or how you deploy it and how you use the CLI or anything to add clusters to it.
I'll need that to fully replicate your setup 👍

[pentago]

Basically, I use Makefile to create 3 separate clusters all connected in one network.
After clusters are up and running, I use Terraform and Helm provider to install ArgoCD and several other charts in GitOps cluster.

This all works fine, the problem starts when I need to add other clusters (dev, prod) to ArgoCD via its CLI.

Not sure why but ArgoCD times out when connecting to other clusters. I thought the network is the issue but I'm able to connect from GitOps cluster to others with curl from any running pods.

I tried using various API port and changing kubeconfig server key to k3d-serverlb containers but to no avail.

I'd appreciate literally any help 👍

[pentago]

Hey @iwilltry42, any luck with the setup?

[iwilltry42]

Worked for me, see screenshot below.

How

1. Add --k3s-arg "--tls-san=host.k3d.internal@server:0" for development and production clusters in Makefile
2. Run make multi
3. (gitops) Install argocd as per https://argo-cd.readthedocs.io/en/stable/getting_started/
4. Edit the local kubeconfig: change the k3d-development cluster config to use host.k3d.internal
5. Add host.k3d.internal mapping to /etc/hosts, so that kubectl knows how to get there: echo "127.0.0.1 host.k3d.internal" >> /etc/hosts
6. Add cluster: argocd cluster add k3d-development
7. Profit

UPDATE 1:
Alternative: First do echo "0.0.0.0 host.k3d.internal" >> /etc/hosts, remove all extra --tls-san lines from the Makefile and update --api-port to include host.k3d.internal before the port, then install argocd, etc.
(This makes the API-Port to be listening on all interfaces - 0.0.0.0 - but also adds host.k3d.internal TLS SAN)

[pentago]

I finally tried this way and it worked fine, clusters are added real smooth :)
Thanks for helping @iwilltry42, you rock!

If there's a way for me to send you a beer and/or pizza, let me know, this means a lot to me! 💪

[iwilltry42]

That's great! Happy to help :)

Heh, you can certainly support me and/or my work in some ways (pizza and beer are a great way to do so 😁).

I just recently set something up:

• https://www.buymeacoffee.com/thklein
• https://github.com/sponsors/iwilltry42

Also, there's IssueHunt if you urgently need to get an issue resolved or anything that you would really love to see soon here 🙂

Thank you very much for your feedback 🙂

[pentago]

5 coffees inbound <3

[iwilltry42]

Oh, thank you very much! ☕🙏

[tahuy]

Hi @iwilltry42,
I want to setup 3 kubernetes clusters with k3d and want each cluster can communicate to other.
As your answer, I'm not clear enough to make it work (look like your answer need to build k3d source?).
Can you detail more?