Enable heal tests

Signed-off-by: Artem Glazychev <artem.glazychev@xored.com>
networkservicemesh · Nov 11, 2021 · fbf3391 · fbf3391
1 parent a16ba0d
commit fbf3391
Show file tree

Hide file tree

Showing 7 changed files with 500 additions and 16 deletions.
diff --git a/examples/heal/README.md b/examples/heal/README.md
@@ -8,13 +8,14 @@ To run any heal example follow steps for [Basic NSM setup](../basic)
 
 ## Includes
 
-# Temporary disable heal tests
-# - [Local Forwarder death](./local-forwarder-death)
-# - [Remote Forwarder death](./remote-forwarder-death)
-# - [Local NSMgr restart](./local-nsmgr-restart)
-# - [Remote NSMgr death](./remote-nsmgr-death)
-# - [Local NSE death](./local-nse-death)
-# - [Remote NSE death](./remote-nse-death)
-# - [Registry restart](./registry-restart)
-# https://github.com/networkservicemesh/sdk/issues/1079
-# - [Remote NSMgr restart](./remote-nsmgr-restart)
+ - [Local Forwarder death](./local-forwarder-death)
+ - [Remote Forwarder death](./remote-forwarder-death)
+ - [Local NSMgr restart](./local-nsmgr-restart)
+ - [Remote NSMgr death](./remote-nsmgr-death)
+ - [Local NSE death](./local-nse-death)
+ - [Remote NSE death](./remote-nse-death)
+ - [Registry restart](./registry-restart)
+ - [Remote NSMgr restart](./remote-nsmgr-restart)
+ - [Remote NSMgr restart ip](./remote-nsmgr-restart-ip)
+ - [Remote Forwarder death ip](./remote-forwarder-death-ip)
+ - [Remote NSE death ip](./remote-nse-death-ip)
diff --git a/examples/heal/local-forwarder-death/README.md b/examples/heal/local-forwarder-death/README.md
@@ -130,12 +130,12 @@ kubectl wait --for=condition=ready --timeout=1m pod -l app=forwarder-vpp --field
 
 Ping from NSC to NSE:
 ```bash
-kubectl exec ${NSC} -n ${NAMESPACE} -- ping -c 4 172.16.1.102
+kubectl exec ${NSC} -n ${NAMESPACE} -- ping -c 4 172.16.1.100
 ```
 
 Ping from NSE to NSC:
 ```bash
-kubectl exec ${NSE} -n ${NAMESPACE} -- ping -c 4 172.16.1.103
+kubectl exec ${NSE} -n ${NAMESPACE} -- ping -c 4 172.16.1.101
 ```
 
 ## Cleanup

diff --git a/examples/heal/remote-forwarder-death-ip/README.md b/examples/heal/remote-forwarder-death-ip/README.md
@@ -0,0 +1,150 @@
+# Test remote Forwarder death
+
+This example shows that NSM keeps working after the remote Forwarder death.
+
+NSC and NSE are using the `kernel` mechanism to connect to its local forwarder.
+Forwarders are using the `vxlan` mechanism to connect with each other.
+
+## Requires
+
+Make sure that you have completed steps from [basic](../../basic) or [memory](../../memory) setup.
+
+## Run
+
+Create test namespace:
+```bash
+NAMESPACE=($(kubectl create -f https://raw.githubusercontent.com/networkservicemesh/deployments-k8s/7113942326f9001fa67b7a9effdf38d4eba2dbdd/examples/heal/namespace.yaml)[0])
+NAMESPACE=${NAMESPACE:10}
+```
+
+Get nodes exclude control-plane:
+```bash
+NODES=($(kubectl get nodes -o go-template='{{range .items}}{{ if not .spec.taints  }}{{index .metadata.labels "kubernetes.io/hostname"}} {{end}}{{end}}'))
+```
+
+Create customization file:
+```bash
+cat > kustomization.yaml <<EOF
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: ${NAMESPACE}
+
+bases:
+- https://github.com/networkservicemesh/deployments-k8s/apps/nsc-kernel?ref=7113942326f9001fa67b7a9effdf38d4eba2dbdd
+- https://github.com/networkservicemesh/deployments-k8s/apps/nse-kernel?ref=7113942326f9001fa67b7a9effdf38d4eba2dbdd
+
+patchesStrategicMerge:
+- patch-nsc.yaml
+- patch-nse.yaml
+EOF
+```
+
+Create NSC patch:
+```bash
+cat > patch-nsc.yaml <<EOF
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nsc-kernel
+spec:
+  template:
+    spec:
+      containers:
+        - name: nsc
+          env:
+            - name: NSM_REQUEST_TIMEOUT
+              value: 45s
+            - name: NSM_NETWORK_SERVICES
+              value: kernel://icmp-responder-ip/nsm-1
+      nodeSelector:
+        kubernetes.io/hostname: ${NODES[0]}
+EOF
+```
+Create NSE patch:
+```bash
+cat > patch-nse.yaml <<EOF
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nse-kernel
+spec:
+  template:
+    spec:
+      containers:
+        - name: nse
+          env:
+            - name: NSM_CIDR_PREFIX
+              value: 172.16.1.100/30
+            - name: NSM_PAYLOAD
+              value: IP
+            - name: NSM_SERVICE_NAMES
+              value: icmp-responder-ip
+      nodeSelector:
+        kubernetes.io/hostname: ${NODES[1]}
+EOF
+```
+
+Deploy NSC and NSE:
+```bash
+kubectl apply -k .
+```
+
+Wait for applications ready:
+```bash
+kubectl wait --for=condition=ready --timeout=1m pod -l app=nsc-kernel -n ${NAMESPACE}
+```
+```bash
+kubectl wait --for=condition=ready --timeout=1m pod -l app=nse-kernel -n ${NAMESPACE}
+```
+
+Find NSC and NSE pods by labels:
+```bash
+NSC=$(kubectl get pods -l app=nsc-kernel -n ${NAMESPACE} --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
+```
+```bash
+NSE=$(kubectl get pods -l app=nse-kernel -n ${NAMESPACE} --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
+```
+
+Ping from NSC to NSE:
+```bash
+kubectl exec ${NSC} -n ${NAMESPACE} -- ping -c 4 172.16.1.100
+```
+
+Ping from NSE to NSC:
+```bash
+kubectl exec ${NSE} -n ${NAMESPACE} -- ping -c 4 172.16.1.101
+```
+
+Find remote Forwarder:
+```bash
+FORWARDER=$(kubectl get pods -l app=forwarder-vpp --field-selector spec.nodeName==${NODES[1]} -n nsm-system --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
+```
+
+Remove remote Forwarder and wait for a new one to start:
+```bash
+kubectl delete pod -n nsm-system ${FORWARDER}
+```
+```bash
+kubectl wait --for=condition=ready --timeout=1m pod -l app=forwarder-vpp --field-selector spec.nodeName==${NODES[1]} -n nsm-system
+```
+
+Ping from NSC to NSE:
+```bash
+kubectl exec ${NSC} -n ${NAMESPACE} -- ping -c 4 172.16.1.100
+```
+
+Ping from NSE to NSC:
+```bash
+kubectl exec ${NSE} -n ${NAMESPACE} -- ping -c 4 172.16.1.101
+```
+
+## Cleanup
+
+Delete ns:
+```bash
+kubectl delete ns ${NAMESPACE}
+```
diff --git a/examples/heal/remote-forwarder-death/README.md b/examples/heal/remote-forwarder-death/README.md
@@ -130,12 +130,12 @@ kubectl wait --for=condition=ready --timeout=1m pod -l app=forwarder-vpp --field
 
 Ping from NSC to NSE:
 ```bash
-kubectl exec ${NSC} -n ${NAMESPACE} -- ping -c 4 172.16.1.102
+kubectl exec ${NSC} -n ${NAMESPACE} -- ping -c 4 172.16.1.100
 ```
 
 Ping from NSE to NSC:
 ```bash
-kubectl exec ${NSE} -n ${NAMESPACE} -- ping -c 4 172.16.1.103
+kubectl exec ${NSE} -n ${NAMESPACE} -- ping -c 4 172.16.1.101
 ```
 
 ## Cleanup

diff --git a/examples/heal/remote-nse-death-ip/README.md b/examples/heal/remote-nse-death-ip/README.md
@@ -0,0 +1,182 @@
+# Local NSE death
+
+This example shows that NSM keeps working after the remote NSE death.
+
+NSC and NSE are using the `kernel` mechanism to connect to its local forwarder.
+Forwarders are using the `vxlan` mechanism to connect with each other.
+
+## Requires
+
+Make sure that you have completed steps from [basic](../../basic) or [memory](../../memory) setup.
+
+## Run
+
+Create test namespace:
+```bash
+NAMESPACE=($(kubectl create -f https://raw.githubusercontent.com/networkservicemesh/deployments-k8s/7113942326f9001fa67b7a9effdf38d4eba2dbdd/examples/heal/namespace.yaml)[0])
+NAMESPACE=${NAMESPACE:10}
+```
+
+Get nodes exclude control-plane:
+```bash
+NODES=($(kubectl get nodes -o go-template='{{range .items}}{{ if not .spec.taints  }}{{index .metadata.labels "kubernetes.io/hostname"}} {{end}}{{end}}'))
+```
+
+Create customization file:
+```bash
+cat > kustomization.yaml <<EOF
+---
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+
+namespace: ${NAMESPACE}
+
+bases:
+- https://github.com/networkservicemesh/deployments-k8s/apps/nsc-kernel?ref=7113942326f9001fa67b7a9effdf38d4eba2dbdd
+- https://github.com/networkservicemesh/deployments-k8s/apps/nse-kernel?ref=7113942326f9001fa67b7a9effdf38d4eba2dbdd
+
+patchesStrategicMerge:
+- patch-nsc.yaml
+- patch-nse.yaml
+EOF
+```
+
+Create NSC patch:
+```bash
+cat > patch-nsc.yaml <<EOF
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nsc-kernel
+spec:
+  template:
+    spec:
+      containers:
+        - name: nsc
+          env:
+            - name: NSM_REQUEST_TIMEOUT
+              value: 45s
+            - name: NSM_NETWORK_SERVICES
+              value: kernel://icmp-responder-ip/nsm-1
+
+      nodeSelector:
+        kubernetes.io/hostname: ${NODES[0]}
+EOF
+
+```
+Create NSE patch:
+```bash
+cat > patch-nse.yaml <<EOF
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nse-kernel
+spec:
+  template:
+    spec:
+      containers:
+        - name: nse
+          env:
+            - name: NSM_CIDR_PREFIX
+              value: 172.16.1.100/31
+            - name: NSM_PAYLOAD
+              value: IP
+            - name: NSM_SERVICE_NAMES
+              value: icmp-responder-ip
+      nodeSelector:
+        kubernetes.io/hostname: ${NODES[1]}
+EOF
+```
+
+Deploy NSC and NSE:
+```bash
+kubectl apply -k .
+```
+
+Wait for applications ready:
+```bash
+kubectl wait --for=condition=ready --timeout=1m pod -l app=nsc-kernel -n ${NAMESPACE}
+```
+```bash
+kubectl wait --for=condition=ready --timeout=1m pod -l app=nse-kernel -n ${NAMESPACE}
+```
+
+Find NSC and NSE pods by labels:
+```bash
+NSC=$(kubectl get pods -l app=nsc-kernel -n ${NAMESPACE} --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
+```
+```bash
+NSE=$(kubectl get pods -l app=nse-kernel -n ${NAMESPACE} --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
+```
+
+Ping from NSC to NSE:
+```bash
+kubectl exec ${NSC} -n ${NAMESPACE} -- ping -c 4 172.16.1.100
+```
+
+Ping from NSE to NSC:
+```bash
+kubectl exec ${NSE} -n ${NAMESPACE} -- ping -c 4 172.16.1.101
+```
+
+Create a new NSE patch:
+```bash
+cat > patch-nse.yaml <<EOF
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: nse-kernel
+spec:
+  template:
+    metadata:
+      labels:
+        version: new
+    spec:
+      containers:
+        - name: nse
+          env:
+            - name: NSM_CIDR_PREFIX
+              value: 172.16.1.102/31
+            - name: NSM_PAYLOAD
+              value: IP
+            - name: NSM_SERVICE_NAMES
+              value: icmp-responder-ip
+      nodeSelector:
+        kubernetes.io/hostname: ${NODES[1]}
+EOF
+```
+
+Apply patch:
+```bash
+kubectl apply -k .
+```
+
+Wait for new NSE to start:
+```bash
+kubectl wait --for=condition=ready --timeout=1m pod -l app=nse-kernel -l version=new -n ${NAMESPACE}
+```
+
+Find new NSE pod:
+```bash
+NEW_NSE=$(kubectl get pods -l app=nse-kernel -l version=new -n ${NAMESPACE} --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
+```
+
+Ping from NSC to new NSE:
+```bash
+kubectl exec ${NSC} -n ${NAMESPACE} -- ping -c 4 172.16.1.102
+```
+
+Ping from new NSE to NSC:
+```bash
+kubectl exec ${NEW_NSE} -n ${NAMESPACE} -- ping -c 4 172.16.1.103
+```
+
+## Cleanup
+
+Delete ns:
+```bash
+kubectl delete ns ${NAMESPACE}
+```