[HELP] Unable to connect between 2 k3d clusters with ArgoCD

[vvijayv]

What did you do

I have 2 K3D clusters in my Mac, created using the k3d command. On one of the cluster, I have installed the ArgoCD, and left the other for running applications.

• How was the cluster created?

  • k3d cluster create argocd --network my-network # cluster argocd is deployed (kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml)
  • k3d cluster create dev-cluster --network my-network # cluster to deploy my applications

• What did you do afterwards?
  I now run the argocd command to add dev-cluster using the below command (port-forward & argocd login steps are skipped for brevity)
  $ argocd cluster add k3d-qa-cluster

I got the below error

INFO[0000] ServiceAccount "argocd-manager" already exists in namespace "kube-system"
INFO[0000] ClusterRole "argocd-manager-role" updated
INFO[0000] ClusterRoleBinding "argocd-manager-role-binding" updated
FATA[0110] rpc error: code = Unavailable desc = transport is closing

What did you expect to happen

The Argocd should connect to the k3d-dev-cluster, create serviceaccount, clusterrole, clusterrolebindings add the cluster to the argocd.

Screenshots or terminal output

Which OS & Architecture

MacOS, x86

Which version of k3d

k3d version v4.4.2
k3s version v1.20.6-k3s1 (default)

Which version of docker

Client: Docker Engine - Community
 Cloud integration: 1.0.12
 Version:           20.10.5
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        55c4c88
 Built:             Tue Mar  2 20:13:00 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.5
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       363e9a8
  Built:            Tue Mar  2 20:15:47 2021
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.6.0)

Server:
 Containers: 36
  Running: 4
  Paused: 0
  Stopped: 32
 Images: 22
 Server Version: 20.10.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc version: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.25-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 1.941GiB
 Name: docker-desktop
 ID: WJ4B:GDYI:7265:JM2I:EJOK:MMUU:SJX4:54IL:MJV7:E2CD:UDMG:PXTJ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[iwilltry42]

Hi @vvijayv , thanks for starting this discussion (transferred over from issue, as this is not in k3d's scope).
I'm not exactly sure how I can help here, as I don't know, what cluster details you put here.
What I can tell you though is that due to using the same docker network, the k3d nodes (k3s containers) should be able to reach one another, as they're living in the same docker subnet.
And it seems you also have connection to the cluster from at least the CLI.
I don't know what's the step there where the Argo CLI is failing. Maybe someone from the agroproj folks can help here?

[MushiTheMoshi]

@vvijayv did you happen to figure out how to fix this?

[ErikEngerd]

I have just managed to add one k3d cluster to an existing k3d cluster that is running argocd. In my setup the cluster k3d-xyz is the cluster that is running argo and k3d-abc is the cluster I want to add. I use the utility kubectx to easily switch between contexts.

As the very first step, I used argocd cluster add k3d-abc, which fails at the end but does create the argocd-manager service account on k3d-abc.

The idea is to configure the cluster connection completely by hand using a secret. As step 1, the k3d-abc cluster must be added to the network of the k3d-xyz cluster:

docker network connect k3d-xyz k3d-abc-server-0

This allows the k3d-xyz cluster to access the api-server of k3d-abc on k3d-abc-server-0:6443. You can verify this by exec-ing into the server container of the k3d-xyz cluster and using telnet.

Next up is to obtain the bearer token of the argocd service account on the k3d-abc cluster:

kubectx k3d-abc
kubectl get sa -n kube-system argocd-manager
TOKEN="$( k get secret -n kube-system argocd-manager-token-cww95  -o json  | jq  -r .data.token | base64 -d )"

Note that above the last part of the secret name can/will differ in your case (just use autocomplete).

Next step is to define the cluster secret:

apiVersion: v1
kind: Secret
metadata:
  namespace: argocd
  name: k3d-abc-cluster-secret
  labels:
    argocd.argoproj.io/secret-type: cluster
type: Opaque
stringData:
  name: k3d-abc
  server: "https://k3d-abc-server-0:6443"
  config: | 
    {
      "bearerToken": "TOKEN",
      "tlsClientConfig": { 
        "insecure": false, 
        "caData": "CADATA"
      } 
    }

Here, TOKEN is the value of the TOKEN variable above. CADATA is the CA data obtained from the .kube/config file for the k3d-abc cluster.

After this, you might need to stop and start the k3d-xyz cluster. This will refresh the DNS entries for the coredns server inside the k3d-xyz cluster so it can resolve k3d-abc-server-0.

With this approach, I can add an application on k3d-xyz to deploy it on k3d-abc:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata: 
  name: directory-app
  namespace: argocd
spec: 
  destination: 
    namespace: directory-app
    server: "https://k3d-abc-server-0:6443"
  project: default
  source: 
    path: guestbook-with-sub-directories
    repoURL: "https://github.com/mabusaa/argocd-example-apps.git"
    targetRevision: master
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

[ErikEngerd]

An alternative solution is probably to use '--network host' when creating the argocd k3d cluster. The cause for the problems is that the URL from the .kube/config is used by 'argocd cluster add', but that same URL is also used by argo CD running inside the cluster to connect to the other cluster. If you use host networking then this should work, but without it, the .kube/config URL for the other cluster is not usable. Disadvantage of this approach is of course that you can only have one argocd cluster running at the same time.

Edit: just tried this and this also gives issues. Therefore, I would just stick with the approach above.