Implement Cluster for zedkube related functions #4408
Conversation
- check kubernetes pod status and publish ENClusterAppStatus for multi-node cluster - subscribe the EdgeNodeClusterConfig and DeviceNetworkStatus for publishing the EdgeNodeClusterStatus for cluster-init.sh to use - implement cluster server status HTTP service for others to join the cluster - collect stats from kubernetes cluster for node/pod/vmi and publish clusterInfo - implement cluster leader election for cluster reporter - documented the functionalities of zedkube in pillar/doc/zedkube.md Signed-off-by: Naiming Shen <naiming@zededa.com>
naiming-zededa
force-pushed
the
naiming-cluster-zedkube
branch
from
90e527b to
15e7b28
Compare
| return | ||
| } | ||
| mux := http.NewServeMux() | ||
| mux.HandleFunc("/status", func(w http.ResponseWriter, r *http.Request) { |
There was a problem hiding this comment.
Are you intending to only support the GET operation? You should check the operation(s) in the handler.
There was a problem hiding this comment.
Yes, updated with the check for GET
| // Start the server in a goroutine | ||
| go func() { | ||
| defer ctx.statusServerWG.Done() | ||
| if err := ctx.statusServer.ListenAndServe(); err != nil && err != http.ErrServerClosed { |
There was a problem hiding this comment.
This listens on all IP addresses. What's the security approach for this open port? (And which port number does it run on? Some documentation seems to be missing ;-)
There was a problem hiding this comment.
This actually only listens on the clusterprefix IP address with the cluster status port.
netstat showing this service:
tcp 0 0 10.244.244.2:12346 0.0.0.0:* LISTEN 4193/zedkube
this is documented in pkg/pillar/doc/zedkube.md in this PR. Updated the above with comments to that.
There was a problem hiding this comment.
Is this clusterprefix IP address reachable from the outside of the cluster?
There was a problem hiding this comment.
It should not be reachable from outside of the local network, the router will not know this private ip prefix.
There was a problem hiding this comment.
@eriknordmark Would you be OK with this approach? When this is merged I can open a followup PR with networking changes and the ACL rule allowing access to cluster-ip:12346 added.
naiming-zededa
force-pushed
the
naiming-cluster-zedkube
branch
from
15e7b28 to
e1ba3b7
Compare
pkg/pillar/docs/zedkube.md
Outdated
|
|
||
| ### App Ethernet Passthrough | ||
|
|
||
| When the application uses passthrough on ethernet ports, zedkube creates a special NAD, Network Attachment Definition, for the direct connection, uses the name 'host-eth1' for example. It creates the NAD to the kubernetes cluster, and the domainmgr will use this NAD when setup the application configure to kubernetes. |
There was a problem hiding this comment.
Is this passthrough all the way to the application VM or just to the launcher Pod and then bridged by kubevirt?
There was a problem hiding this comment.
this is actually only for the native container, not for VMIs. I'll add this specific thing to the doc here.
There was a problem hiding this comment.
For VM and container-inside-VM we use this for passthrough: https://kubevirt.io/user-guide/compute/host-devices/
Correct?
Should we document it here?
There was a problem hiding this comment.
(can be done also in a future PR)
|
|
||
| ### Cluster Status Server | ||
|
|
||
| In the Kubernetes cluster mode with multiple HA servers, when it starts up, it needs to join the cluster by specifying the 'bootstrap' server IP address. Even if the IP address is there, sometimes the 'bootstrap' node is still in single-node mode or it has not been converted into the cluster server yet. This will create problem for the joining server, and will later have conflicts with the status and certificates or tokens. To handle this server joining, zedkube is responsible for reporting it's cluster status through HTTP service. Each of the cluster servers will have a HTTP service on the cluster interface with port number '12346' using URL /status. It will report status of 'cluster' if the node has the property of 'master' and 'etcd'. The new joining server node or agent node will not move forward for starting the kubernetes node unless the http query returns 'cluster' status over the cluster network. The 'ClusterStatus' port for the HTTP is explicitly allowed on EVE firewall. |
There was a problem hiding this comment.
Are we restricting this to use the cluster network interface and IP addresses using some ACLs? Can we use https using some certificates which k3s might have?
I don't think we can have TCP port number 12346 accessible from the Internet.
There was a problem hiding this comment.
Currently this is just HTTP, using certs only going to verify the server side which does not help. Yes, this is only listening on the cluster IP which is a private prefix setup locally.
There was a problem hiding this comment.
the firewall check is specific to the cluster interface and cluster IP w/ this port:
clusterStatusRule := iptables.Rule{
RuleLabel: "Allow access to Cluster Status",
MatchOpts: []string{"-p", "tcp", "-i", clusterPort.IfName,
"-d", clusterStatus.ClusterIPPrefix.IP.String(), "--dport", "12346"},
Target: "ACCEPT",
Description: "Allow access to Cluster Status via cluster interface",
}
- Add vendor files for Implement Cluster for zedkube related functions Signed-off-by: Naiming Shen <naiming@zededa.com>
naiming-zededa
force-pushed
the
naiming-cluster-zedkube
branch
from
e1ba3b7 to
54ba466
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #4408 +/- ##
=======================================
Coverage 20.93% 20.93%
=======================================
Files 13 13
Lines 2895 2895
=======================================
Hits 606 606
Misses 2163 2163
Partials 126 126 ☔ View full report in Codecov by Sentry. |
|
@eriknordmark If you are OK with the port 12346 being open for (only) cluster-IP, I can fix the conflict in zedkube.md, merge the PR and we could proceed with the followup PR containing network changes. |
|
@milan-zededa I wonder where we should document this somewhere. But we can figure that out later. Is there a device running this PR where we can try a port scan just to make sure it isn't exposed outside of the cluster IP? |
@zedi-pramodh @andrewd-zededa Do you have some cluster deployed running EVE with all your latest changes? |
|
@milan-zededa check either alpha or local please. Both have a cluster running recent code. |
The overview section was accidentally removed.
|
I took the liberty of resolving the zedkube.md merge conflicts to help move this forward so that I can submit my last outstanding commit for kubevirt-EVE. |
|
Closing since this is now superseded by #4454 |
for multi-node cluster
publishing the EdgeNodeClusterStatus for cluster-init.sh to use
the cluster
publish clusterInfo