Managing Your Clusters (including public, private, hybrid, edge, etc) as easily as Visiting the Internet.
Clusternet (Cluster Internet) is an open source add-on that helps you manage thousands of millions of Kubernetes clusters as easily as visiting the Internet. No matter the clusters are running on public cloud, private cloud, hybrid cloud, or at the edge, Clusternet lets you manage/visit them all as if they were running locally. This also help eliminate the need to juggle different management tools for each cluster.
Clusternet can also help deploy and coordinate applications to multiple clusters from a single set of APIs in a hosting cluster.
Clusternet will help setup network tunnels in a configurable way, when your clusters are running in a VPC network, at the edge, or behind a firewall.
Clusternet also provides a Kubernetes-styled API, where you can continue using the Kubernetes way, such as KubeConfig, to visit a certain Managed Kubernetes cluster, or a Kubernetes service.
Clusternet is multiple platforms supported now, including
darwin/amd64
anddarwin/arm64
;linux/amd64
,linux/arm64
,linux/ppc64le
,linux/s390x
,linux/386
andlinux/arm
;
Clusternet is light-weighted that consists of two components, clusternet-agent
and clusternet-hub
.
clusternet-agent
is responsible for
- auto-registering current cluster to a parent cluster as a child cluster, which is also been called
ManagedCluster
; - reporting heartbeats of current cluster, including Kubernetes version, running platform,
healthz
/readyz
/livez
status, etc; - setting up a websocket connection that provides full-duplex communication channels over a single TCP connection to parent cluster;
clusternet-hub
is responsible for
- approving cluster registration requests and creating dedicated resources, such as namespaces, serviceaccounts and RBAC rules, for each child cluster;
- serving as an aggregated apiserver (AA), which is used to serve as a websocket server that maintain multiple active websocket connections from child clusters;
- providing Kubernstes-styled API to redirect/proxy/upgrade requests to each child cluster;
- coordinating and deploying applications to multiple clusters from a single set of APIs;
π π Note:
Since
clusternet-hub
is running as an AA, please make sure that parent apiserver could visit theclusternet-hub
service.
For every Kubernetes cluster that wants to be managed, we call it child cluster. The cluster where child clusters are registerring to, we call it parent cluster.
clusternet-agent
runs in child cluster, while clusternet-hub
runs in parent cluster.
ClusterRegistrationRequest
is an object thatclusternet-agent
creates in parent cluster for child cluster registration.ManagedCluster
is an object thatclusternet-hub
creates in parent cluster after approvingClusterRegistrationRequest
.HelmChart
is an object contains a helm chart configuration.Subscription
defines the resources that subscribers want to install into clusters. For every matched cluster, a correspondingBase
object will be created in its dedicated namespace.Localization
andGlobalization
will define the overrides with priority, where lower numbers are considered lower priority.Localization
is namespace-scoped resource, whileGlobalization
is cluster-scoped.Base
objects will be rendered toDescription
objects withGlobalization
andLocalization
settings applied.Descritpion
is the final resources to be deployed into target child clusters.
If you want to get participated and become a contributor to Clusternet, please don't hesitate to refer to our CONTRIBUTING document for details.
A developer guide is ready to help you
- build binaries for all platforms, such as
darwin/amd64
,linux/amd64
,linux/arm64
, etc; - build docker images for multiple platforms, such as
linux/amd64
,linux/arm64
, etc;
You need to deploy clusternet-agent
and clusternet-hub
in child cluster and parent cluster respectively.
$ kubectl apply -f deploy/hub
And then create a bootstrap token for clusternet-agent
,
$ # this will create a bootstrap token 07401b.f395accd246ae52d
$ kubectl apply -f manifests/samples/cluster_bootstrap_token.yaml
clusternet-agent
runs in child cluster and helps register self-cluster to parent cluster.
clusternet-agent
could be configured with below three kinds of SyncMode
(configured by flag --cluster-sync-mode
),
Push
means that all the resource changes in the parent cluster will be synchronized, pushed and applied to child clusters byclusternet-hub
automatically.Pull
meansclusternet-agent
will watch, synchronize and apply all the resource changes from the parent cluster to child cluster.Dual
combines bothPush
andPull
mode. This mode is strongly recommended, which is usually used together with feature gateAppPusher
.
Feature gate AppPusher
works on agent side, which is introduced mainly for below two reasons,
-
SyncMode
is not suggested getting changed after registration, which may bring in inconsistent settings and behaviors. That's whyDual
mode is strong recommended. WhenDual
mode is set, feature gateAppPusher
provides a way to help switchPush
mode toPull
mode without really changing flag--cluster-sync-mode
, and vice versa. -
For security concerns, such as child cluster security risks, etc.
When a child cluster has disabled feature gate
AppPusher
, the parent cluster won't deploy any applications to it, even if SyncModePush
orDual
is set. At this time, this child cluster is working likePull
mode.Resources to be deployed are represented as
Description
, you can run your own controllers as well to watch changes ofDescription
objects, then distribute and deploy resources.
Upon deploying clusternet-agent
, a secret that contains token for cluster registration should be created firstly.
$ # create namespace clusternet-system if not created
$ kubectl create ns clusternet-system
$ # here we use the token created above
$ PARENTURL=https://192.168.10.10 REGTOKEN=07401b.f395accd246ae52d envsubst < ./deploy/templates/clusternet_agent_secret.yaml | kubectl apply -f -
The PARENTURL
above is the apiserver address of the parent cluster that you want to register to, the https
scheme
must be specified and it is the only one supported at the moment. If the apiserver is not listening on the standard
https port (:443), please specify the port number in the URL to ensure the agent connects to the right endpoint, for
instance, https://192.168.10.10:6443
.
$ # before deploying, you could update the SyncMode if needed
$ kubectl apply -f deploy/agent
$ # clsrr is an alias for ClusterRegistrationRequest
$ kubectl get clsrr
NAME CLUSTER ID STATUS AGE
clusternet-dc91021d-2361-4f6d-a404-7c33b9e01118 dc91021d-2361-4f6d-a404-7c33b9e01118 Approved 3d6h
$ kubectl get clsrr clusternet-dc91021d-2361-4f6d-a404-7c33b9e01118 -o yaml
apiVersion: clusters.clusternet.io/v1beta1
kind: ClusterRegistrationRequest
metadata:
labels:
clusters.clusternet.io/cluster-id: dc91021d-2361-4f6d-a404-7c33b9e01118
clusters.clusternet.io/cluster-name: clusternet-cluster-dzqkw
clusters.clusternet.io/registered-by: clusternet-agent
name: clusternet-dc91021d-2361-4f6d-a404-7c33b9e01118
spec:
clusterId: dc91021d-2361-4f6d-a404-7c33b9e01118
clusterName: clusternet-cluster-dzqkw
clusterType: EdgeClusterSelfProvisioned
status:
caCertificate: REDACTED
dedicatedNamespace: clusternet-dhxfs
managedClusterName: clusternet-cluster-dzqkw
result: Approved
token: REDACTED
After ClusterRegistrationRequest
is approved, the status will be updated with corresponding credentials that can be
used to access parent cluster if needed. Those credentials have been set with scoped RBAC rules, see blow two rules for
details.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
annotations:
clusternet.io/autoupdate: "true"
labels:
clusters.clusternet.io/bootstrapping: rbac-defaults
clusters.clusternet.io/cluster-id: dc91021d-2361-4f6d-a404-7c33b9e01118
clusternet.io/created-by: clusternet-hub
name: clusternet-dc91021d-2361-4f6d-a404-7c33b9e01118
rules:
- apiGroups:
- clusters.clusternet.io
resources:
- clusterregistrationrequests
verbs:
- create
- get
- apiGroups:
- proxies.clusternet.io
resourceNames:
- dc91021d-2361-4f6d-a404-7c33b9e01118
resources:
- sockets
verbs:
- '*'
and
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
annotations:
clusternet.io/autoupdate: "true"
labels:
clusters.clusternet.io/bootstrapping: rbac-defaults
clusternet.io/created-by: clusternet-hub
name: clusternet-managedcluster-role
namespace: clusternet-dhxfs
rules:
- apiGroups:
- '*'
resources:
- '*'
verbs:
- '*'
$ # mcls is an alias for ManagedCluster
$ # kubectl get mcls -A
$ # or append "-o wide" to display extra columns
$ kubectl get mcls -A -o wide
NAMESPACE NAME CLUSTER ID CLUSTER TYPE SYNC MODE KUBERNETES READYZ AGE
clusternet-dhxfs clusternet-cluster-dzqkw dc91021d-2361-4f6d-a404-7c33b9e01118 EdgeClusterSelfProvisioned Dual v1.19.10 true 7d23h
$ kubectl get mcls -n clusternet-dhxfs clusternet-cluster-dzqkw -o yaml
apiVersion: clusters.clusternet.io/v1beta1
kind: ManagedCluster
metadata:
labels:
clusters.clusternet.io/cluster-id: dc91021d-2361-4f6d-a404-7c33b9e01118
clusters.clusternet.io/cluster-name: clusternet-cluster-dzqkw
clusternet.io/created-by: clusternet-agent
name: clusternet-cluster-dzqkw
namespace: clusternet-dhxfs
spec:
clusterId: dc91021d-2361-4f6d-a404-7c33b9e01118
clusterType: EdgeClusterSelfProvisioned
syncMode: Dual
status:
apiserverURL: http://10.0.0.10:8080
appPusher: true
healthz: true
k8sVersion: v1.19.10
lastObservedTime: "2021-06-30T08:55:14Z"
livez: true
platform: linux/amd64
readyz: true
The status of ManagedCluster
is updated by clusternet-agent
every 3 minutes for default, which can be configured by
flag --cluster-status-update-frequency
.
Clusternet supports visiting all your managed clusters with RBAC.
There is one prerequisite here, that is kube-apiserver
should allow anonymous requests. The
flag --anonymous-auth
is set to be true
by default. So you can just ignore this unless this flag is set to false
explicitly.
Actually what you need is to
-
Append
/apis/proxies.clusternet.io/v1alpha1/sockets/<CLUSTER-ID>/proxy/https/<SERVER-URL>
or/apis/proxies.clusternet.io/v1alpha1/sockets/<CLUSTER-ID>/proxy/direct
at the end of original parent cluster server address-
CLUSTER-ID
is a UUID for your child cluster, which is auto-populated byclusternet-agent
, such as dc91021d-2361-4f6d-a404-7c33b9e01118. You could get this UUID from objectsClusterRegistrationRequest
,ManagedCluster
, etc. Also this UUID is labeled with keyclusters.clusternet.io/cluster-id
. -
SERVER-URL
is the apiserver address of your child cluster, it could belocalhost
,127.0.0.1
and etc, only ifclusternet-agent
could access.
You can follow below commands to help modify above changes.
$ # suppose your parent cluster kubeconfig locates at /home/demo/.kube/config.parent $ kubectl config view --kubeconfig=/home/demo/.kube/config.parent --minify=true --raw=true > ./config-cluster-dc91021d-2361-4f6d-a404-7c33b9e01118 $ $ export KUBECONFIG=`pwd`/config-cluster-dc91021d-2361-4f6d-a404-7c33b9e01118 $ kubectl config view apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://10.0.0.10:6443 name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: REDACTED client-key-data: REDACTED $ $ # suppose your child cluster running at https://demo1.cluster.net $ kubectl config set-cluster `kubectl config get-clusters | grep -v NAME` \ --server=https://10.0.0.10:6443/apis/proxies.clusternet.io/v1alpha1/sockets/dc91021d-2361-4f6d-a404-7c33b9e01118/proxy/https/demo1.cluster.net $ # or just use the direct proxy path $ kubectl config set-cluster `kubectl config get-clusters | grep -v NAME` \ --server=https://10.0.0.10:6443/apis/proxies.clusternet.io/v1alpha1/sockets/dc91021d-2361-4f6d-a404-7c33b9e01118/proxy/direct
π π Note:
Clusternet supports both http and https scheme.
If you want to use scheme
http
to demonstrate how it works, i.e./apis/proxies.clusternet.io/v1alpha1/sockets/<CLUSTER-ID>/proxy/http/<SERVER-URL>
, you can simply run a local proxy in your child cluster, for example,$ kubectl proxy --address='10.212.0.7' --accept-hosts='^*$'
Please replace
10.212.0.7
with your real local IP address.Then you can visit child cluster with http scheme. The KubeConfig here would be quite simple,
apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://10.0.0.10:6443/apis/proxies.clusternet.io/v1alpha1/sockets/dc91021d-2361-4f6d-a404-7c33b9e01118/proxy/http/10.212.0.7 name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: username: system:anonymous
-
-
Then update user entry with credentials from child clusters
π π Note:
Clusternet-hub
does not care about those credentials at all, passing them directly to child clusters.-
If you're using tokens, such as bootstrap tokens, ServiceAccount tokens , etc, please follow below modifications.
$ export KUBECONFIG=`pwd`/config-cluster-dc91021d-2361-4f6d-a404-7c33b9e01118 $ # below is what we modified in above step 1 $ kubectl config view apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://10.0.0.10:6443/apis/proxies.clusternet.io/v1alpha1/sockets/dc91021d-2361-4f6d-a404-7c33b9e01118/proxy/direct name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: REDACTED client-key-data: REDACTED $ $ # modify user part to below $ vim config-cluster-dc91021d-2361-4f6d-a404-7c33b9e01118 ... user: username: system:anonymous as: clusternet as-user-extra: clusternet-token: - BASE64-DECODED-PLEASE-CHANGE-ME
Please replace
BASE64-DECODED-PLEASE-CHANGE-ME
to a token that valid from child cluster. Please notice the tokens replaced here should be base64 decoded. -
If you're using TLS certificates, please follow below modifications.
$ export KUBECONFIG=`pwd`/config-cluster-dc91021d-2361-4f6d-a404-7c33b9e01118 $ # below is what we modified in above step 1 $ kubectl config view apiVersion: v1 clusters: - cluster: certificate-authority-data: DATA+OMITTED server: https://10.0.0.10:6443/apis/proxies.clusternet.io/v1alpha1/sockets/dc91021d-2361-4f6d-a404-7c33b9e01118/proxy/direct name: kubernetes contexts: - context: cluster: kubernetes user: kubernetes-admin name: kubernetes-admin@kubernetes current-context: kubernetes-admin@kubernetes kind: Config preferences: {} users: - name: kubernetes-admin user: client-certificate-data: REDACTED client-key-data: REDACTED $ $ # modify user part to below $ vim config-cluster-dc91021d-2361-4f6d-a404-7c33b9e01118 ... user: username: system:anonymous as: clusternet as-user-extra: clusternet-certificate: - CLIENT-CERTIFICATE-DATE-BASE64-ENCODED-PLEASE-CHANGE-ME clusternet-privatekey: - CLIENT-KEY-DATE-PLEASE-BASE64-ENCODED-CHANGE-ME
Please replace
CLIENT-CERTIFICATE-DATE-BASE64-ENCODED-PLEASE-CHANGE-ME
andCLIENT-KEY-DATE-PLEASE-BASE64-ENCODED-CHANGE-ME
with certficate and private key from child cluster. Please notice the tokens replaced here should be base64 encoded.
-
Clusternet has provided two ways to help interact with Clusternet.
- kubectl plugin kubectl-clusternet
- use client-go to interact with Clusternet
Clusternet supports deploying applications to multiple clusters from a single set of APIs in a hosting cluster.
π π Note:
Feature gate
Deployer
should be enabled byclusternet-hub
.
First, let's see an exmaple application. Below Subscription
"app-demo" defines the target child clusters to be
distributed to, and the resources to be deployed with.
# examples/applications/subscription.yaml
apiVersion: apps.clusternet.io/v1alpha1
kind: Subscription
metadata:
name: app-demo
namespace: default
spec:
subscribers: # defines the clusters to be distributed to
- clusterAffinity:
matchLabels:
clusters.clusternet.io/cluster-id: dc91021d-2361-4f6d-a404-7c33b9e01118 # PLEASE UPDATE THIS CLUSTER-ID TO YOURS!!!
feeds: # defines all the resources to be deployed with
- apiVersion: apps.clusternet.io/v1alpha1
kind: HelmChart
name: mysql
namespace: default
- apiVersion: v1
kind: Namespace
name: foo
- apiVersion: apps/v1
kind: Service
name: my-nginx-svc
namespace: foo
- apiVersion: apps/v1
kind: Deployment
name: my-nginx
namespace: foo
Before applying this Subscription
, please
modify examples/applications/subscription.yaml
with your clusterID.
After installing kubectl plugin kubectl-clusternet, you could run below commands to distribute this application to child clusters.
$ kubectl clusternet apply -f examples/applications/
helmchart.apps.clusternet.io/mysql created
namespace/foo created
deployment.apps/my-nginx created
service/my-nginx-svc created
subscription.apps.clusternet.io/app-demo created
$ # or
$ # kubectl-clusternet apply -f examples/applications/
Then you can view the resources just created,
$ # list Subscription
$ kubectl clusternet get subs -A
NAMESPACE NAME AGE
default app-demo 6m4s
$ kubectl clusternet get chart
NAME CHART VERSION REPO STATUS AGE
mysql mysql 8.6.2 https://charts.bitnami.com/bitnami Found 71s
$ kubectl clusternet get ns
NAME CREATED AT
foo 2021-08-07T08:50:55Z
$ kubectl clusternet get svc -n foo
NAME CREATED AT
my-nginx-svc 2021-08-07T08:50:57Z
$ kubectl clusternet get deploy -n foo
NAME CREATED AT
my-nginx 2021-08-07T08:50:56Z
Clusternet
will help deploy and coordinate applications to multiple clusters. You can check the status by following
commands,
$ kubectl clusternet get mcls -A
NAMESPACE NAME CLUSTER ID SYNC MODE KUBERNETES READYZ AGE
clusternet-5l82l clusternet-cluster-hx455 dc91021d-2361-4f6d-a404-7c33b9e01118 Dual v1.21.0 true 5d22h
$ # list Descriptions
$ kubectl clusternet get desc -A
NAMESPACE NAME DEPLOYER STATUS AGE
clusternet-5l82l app-demo-generic Generic Success 2m55s
clusternet-5l82l app-demo-helm Helm Success 2m55s
$ kubectl describe desc -n clusternet-5l82l app-demo-generic
...
Status:
Phase: Success
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfullyDeployed 2m55s clusternet-hub Description clusternet-5l82l/app-demo-generic is deployed successfully
$ # list Helm Release
$ # hr is an alias for HelmRelease
$ kubectl clusternet get hr -n clusternet-5l82l
NAME CHART VERSION REPO STATUS AGE
helm-demo-mysql mysql 8.6.2 https://charts.bitnami.com/bitnami deployed 2m55s
You can also verify the installation with Helm command line in your child cluster,
$ helm ls -n abc
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
helm-demo-mysql abc 1 2021-07-06 14:34:44.188938 +0800 CST deployed mysql-8.6.2 8.0.25