Add perf testing

1. Scripts 2. Manifest 3. Document Signed-off-by: Hao, Ruomeng <ruomeng.hao@intel.com>
cc-api · Jul 12, 2024 · cd28626 · cd28626
1 parent 36d01b4
commit cd28626
Show file tree

Hide file tree

Showing 8 changed files with 553 additions and 79 deletions.
diff --git a/container/ccnp-example/Dockerfile b/container/ccnp-example/Dockerfile
@@ -36,6 +36,7 @@ RUN chown $USER:$GROUP /run/ccnp
 
 COPY test ./
 COPY sdk/python3/example/py_sdk_example.py ./
+COPY test/perf/py_perf.py ./
 COPY --from=python-builder cc-trusted-api/common/python/dist/cctrusted_base*.whl ./
 COPY --from=python-builder ccnp-sdk/dist/ccnp*.whl ./
 

diff --git a/deployment/kubernetes/manifests/ccnp-example-perf-deployment.yaml b/deployment/kubernetes/manifests/ccnp-example-perf-deployment.yaml
@@ -0,0 +1,26 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+ name: ccnp-perf-example-PLACEHOLDER
+ namespace: perf
+spec:
+ selector:
+ matchLabels:
+ app: ccnp-perf-example-PLACEHOLDER
+ template:
+ metadata:
+ labels:
+ app: ccnp-perf-example-PLACEHOLDER
+ annotations:
+ "ccnp.cc-api/require": "true"
+ spec:
+ containers:
+ - name: ccnp-perf-example-PLACEHOLDER
+ image: "docker.io/library/ccnp-example:latest"
+ imagePullPolicy: Always
+ resources:
+ limits:
+ memory: "128Mi"
+ cpu: "100m"
+ nodeSelector:
+ "feature.node.kubernetes.io/cpu-security.tdx.protected": "true"
diff --git a/test/perf/README.md b/test/perf/README.md
@@ -1,102 +1,49 @@
-# Performance Test
+# CCNP Performance Test
 
-We have these KPIs for performance test.
-
-| KPI | HIB/LIB | Unit | Comment |
-| ------------------------------------------- | -------- | ----- | ----------------------------------------------------- |
-| CCNP service get measurement throughput | HIB | ops | Service Throughput |
-| CCNP service get measurement response time | LIB | ms | Service Response time |
-| CCNP service get eventlog throughput | HIB | ops | Service Throughput |
-| CCNP service get eventlog response time | LIB | ms | Service Response time |
-| CCNP service get quote throughput | HIB | ops | Service Throughput |
-| CCNP service get quote response time | LIB | ms | Service Response time |
-| CCNP initialization time | LIB | s | CCNP device plugin, DaemonSet and service readiness. |
-
-*Note: we use the CCNP SDK to access the CCNP service because it's convenient to prepare the request data (e.g. container ID, etc.)
+CCNP performance tests focus on the latency of calling CCNP SDK key APIs: `get_cc_eventlog`, `get_cc_report` and `get_cc_measurement`.
+It will simulate requests from multiple pods in parallel and calculate average time of all the requests.
 
 Below are the steps for you to build and run the performance test.
 
 ## Prerequisites
 
-To run the test, you need a K8S cluster with CCNP enabled (CCNP Device Plugin and CCNP Service deployed and ready).
-
-## Build
-
-```bash
-# Make sure you are on the repo's top dir
-cd <the-dir-of-confidential-cloud-native-primitives>
+Please make sure you have CCNP deployed in a K8S cluster, and ccnp-example image has been built.
+Please refer to [here](../../deployment/README.md) for image building and CCNP deployment.
 
-# Run doker build
-docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --build-arg no_proxy=$no_proxy -t ccnp-perf:latest -f container/ccnp-perf/Dockerfile .
+## Run Tests
 
-# View build result
-docker image ls | grep ccnp-perf
+### Deploy pods for performance testing
 
-# Save the docker image for later use
-docker save ccnp-perf:latest > ccnp-perf_latest.tar
+```bash
+# Deploy ccnp-example pods
+$ sudo ./deploy-perf.sh -r <ccnp-example image registry> -g <ccnp-example image tag> -n <number of pods>
 ```
 
-## Deploy
+### Run Tests
+
+The script will run tests in parallel. The log will be saved in files with prefix `perf_output` under current directory.
 
 ```bash
-# Load the docker image for K8S using containerd.
-# You need to run this on the node where you want to deploy the ccnp-perf test
-ctr -n=k8s.io image import ccnp-perf_latest.tar
+# Test for get event log
+$ sudo ./perf-para.sh -n <number of pods> -e
 
-# Make sure you are on the repo's top dir
-cd <the-dir-of-confidential-cloud-native-primitives>
+# Test for get measurement
+$ sudo ./perf-para.sh -n <number of pods> -m
 
-# Deploy ccnp-perf test
-kubectl apply -f deployment/kubernetes/manifests/ccnp-perf-deployment.yaml
+# Test for get quote
+$ sudo ./perf-para.sh -n <number of pods> -r
 ```
 
-## Test
+Run below script to calculate average time of a request.
 
 ```bash
-# Get the pod name of ccnp-perf
-kubectl get pod | grep ccnp-perf
-
-# Run all perf test on the specified pod name got from above command
-kubectl exec -ti <ccnp-perf-pod-name> -- python3 -m pytest --log-cli-level=INFO --verbose ccnp_perf.py
+$ sudo ./average.sh -f perf_output_quote
 ```
 
-Sample test output looks like this:
+### Clear
+
+Run below command to delete the pods for performance testing.
 
 ```bash
-root@ccnp-perf-0:~/ccnp/confidential-cloud-native-primitives# kubectl exec -ti ccnp-perf-7f8798bf85-8s6zg -- python3 -m pytest --log-cli-level=INFO --verbose
- ccnp_perf.py
-==================================================================== test session starts ====================================================================
-platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0 -- /usr/local/bin/python3
-cachedir: .pytest_cache
-rootdir: /run/ccnp
-collected 7 items
-
-ccnp_perf.py::test_svc_get_cc_measurement_throughput
------------------------------------------------------------------------ live log call -----------------------------------------------------------------------
-INFO ccnp_perf:ccnp_perf.py:191 Perf test average throughput is: 70.75 ops (operations per second)
-PASSED [ 14%]
-ccnp_perf.py::test_svc_get_cc_measurement_response
------------------------------------------------------------------------ live log call -----------------------------------------------------------------------
-INFO ccnp_perf:ccnp_perf.py:213 Perf test average response time is: 25.89662575 ms (milliseconds)
-PASSED [ 28%]
-ccnp_perf.py::test_svc_get_cc_eventlog_throughput
------------------------------------------------------------------------ live log call -----------------------------------------------------------------------
-INFO ccnp_perf:ccnp_perf.py:191 Perf test average throughput is: 57.8 ops (operations per second)
-PASSED [ 42%]
-ccnp_perf.py::test_svc_get_cc_eventlog_response
------------------------------------------------------------------------ live log call -----------------------------------------------------------------------
-INFO ccnp_perf:ccnp_perf.py:213 Perf test average response time is: 76.130223 ms (milliseconds)
-PASSED [ 57%]
-ccnp_perf.py::test_svc_get_cc_report_throughput
------------------------------------------------------------------------ live log call -----------------------------------------------------------------------
-INFO ccnp_perf:ccnp_perf.py:191 Perf test average throughput is: 54.9 ops (operations per second)
-PASSED [ 71%]
-ccnp_perf.py::test_svc_get_cc_report_response
------------------------------------------------------------------------ live log call -----------------------------------------------------------------------
-INFO ccnp_perf:ccnp_perf.py:213 Perf test average response time is: 29.38618825 ms (milliseconds)
-PASSED [ 85%]
-ccnp_perf.py::test_ccnp_init PASSED [100%]
-
-=============================================================== 7 passed in 66.95s (0:01:06) ================================================================
-root@ccnp-perf-0:~/ccnp/confidential-cloud-native-primitives#
-```
+$ sudo ./deploy-per.sh -n <number of pods> -d
+```
diff --git a/test/perf/average.sh b/test/perf/average.sh
@@ -0,0 +1,73 @@
+#!/bin/bash
+# Input filename keyword and keyword for grep. Find the files in current dir, search the numbers needed, calculate sum and average
+
+grep_keyword="total command "
+
+usage() {
+ cat << EOM
+usage: $(basename "$0") [OPTION]...
+ -f Input file name keyword
+ -k Keyword to be searched in the input files
+EOM
+ exit 1
+}
+
+process_args() {
+ while getopts ":f:k:h" option; do
+ case "${option}" in
+ f) filename_keyword=${OPTARG};;
+ k) grep_keyword=${OPTARG};;
+ h) usage;;
+ *) echo "Invalid option: -${OPTARG}" >&2
+ usage
+ ;;
+ esac
+ done
+}
+
+
+# Find all files containing the filename keyword
+files=$(find . -type f -name "*${filename_keyword}*"| cut -d/ -f2)
+
+total_sum=0
+total_count=0
+total_aver=0
+
+# Loop through the found files
+for file in $files; do
+ sum=0
+ count=0
+
+ tag=$(echo "$file" | cut -d- -f2)
+ tmpfile="tmp_output_"$tag
+ touch "$tmpfile"
+
+ # Grep the keyword and extract the number. The number is the 2nd last word in the line.
+ grep -i "${grep_keyword}" "$file" | awk '{print $(NF-1)}' > "$tmpfile"
+
+ while IFS= read -r line; do
+ # Add the number to sum
+ sum=$(echo "$sum + $line" | bc)
+ # Increment the count
+ count=$((count + 1))
+ done < "$tmpfile"
+
+ # Calculate the average
+ if [ $count -ne 0 ]; then
+ average=$(echo "scale=6; $sum / $count" | bc)
+ echo "Output file: $file"
+ echo "Number file: $tmpfile"
+ echo "Count: $count"
+ echo "Sum: $sum"
+ echo "Average: $average"
+ else
+ echo "The file is empty"
+ fi
+ total_sum=$(echo "$sum + $total_sum" | bc)
+ total_count=$(echo "$count + $total_count" | bc)
+done
+
+echo "Total Sum: $total_sum"
+echo "Total count: $total_count"
+total_aver=$(echo "scale=6; $total_sum / $total_count" | bc)
+echo "Total average: $total_aver"
diff --git a/test/perf/ccnp-pytest.md b/test/perf/ccnp-pytest.md
@@ -0,0 +1,102 @@
+# Performance Test
+
+We have these KPIs for performance test.
+
+| KPI | HIB/LIB | Unit | Comment |
+| ------------------------------------------- | -------- | ----- | ----------------------------------------------------- |
+| CCNP service get measurement throughput | HIB | ops | Service Throughput |
+| CCNP service get measurement response time | LIB | ms | Service Response time |
+| CCNP service get eventlog throughput | HIB | ops | Service Throughput |
+| CCNP service get eventlog response time | LIB | ms | Service Response time |
+| CCNP service get quote throughput | HIB | ops | Service Throughput |
+| CCNP service get quote response time | LIB | ms | Service Response time |
+| CCNP initialization time | LIB | s | CCNP device plugin, DaemonSet and service readiness. |
+
+*Note: we use the CCNP SDK to access the CCNP service because it's convenient to prepare the request data (e.g. container ID, etc.)
+
+Below are the steps for you to build and run the performance test.
+
+## Prerequisites
+
+To run the test, you need a K8S cluster with CCNP enabled (CCNP Device Plugin and CCNP Service deployed and ready).
+
+## Build
+
+```bash
+# Make sure you are on the repo's top dir
+cd <the-dir-of-confidential-cloud-native-primitives>
+
+# Run doker build
+docker build --build-arg http_proxy=$http_proxy --build-arg https_proxy=$https_proxy --build-arg no_proxy=$no_proxy -t ccnp-perf:latest -f container/ccnp-perf/Dockerfile .
+
+# View build result
+docker image ls | grep ccnp-perf
+
+# Save the docker image for later use
+docker save ccnp-perf:latest > ccnp-perf_latest.tar
+```
+
+## Deploy
+
+```bash
+# Load the docker image for K8S using containerd.
+# You need to run this on the node where you want to deploy the ccnp-perf test
+ctr -n=k8s.io image import ccnp-perf_latest.tar
+
+# Make sure you are on the repo's top dir
+cd <the-dir-of-confidential-cloud-native-primitives>
+
+# Deploy ccnp-perf test
+kubectl apply -f deployment/kubernetes/manifests/ccnp-perf-deployment.yaml
+```
+
+## Test
+
+```bash
+# Get the pod name of ccnp-perf
+kubectl get pod | grep ccnp-perf
+
+# Run all perf test on the specified pod name got from above command
+kubectl exec -ti <ccnp-perf-pod-name> -- python3 -m pytest --log-cli-level=INFO --verbose ccnp_perf.py
+```
+
+Sample test output looks like this:
+
+```bash
+root@ccnp-perf-0:~/ccnp/confidential-cloud-native-primitives# kubectl exec -ti ccnp-perf-7f8798bf85-8s6zg -- python3 -m pytest --log-cli-level=INFO --verbose
+ ccnp_perf.py
+==================================================================== test session starts ====================================================================
+platform linux -- Python 3.12.2, pytest-8.1.1, pluggy-1.4.0 -- /usr/local/bin/python3
+cachedir: .pytest_cache
+rootdir: /run/ccnp
+collected 7 items
+
+ccnp_perf.py::test_svc_get_cc_measurement_throughput
+----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
+INFO ccnp_perf:ccnp_perf.py:191 Perf test average throughput is: 70.75 ops (operations per second)
+PASSED [ 14%]
+ccnp_perf.py::test_svc_get_cc_measurement_response
+----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
+INFO ccnp_perf:ccnp_perf.py:213 Perf test average response time is: 25.89662575 ms (milliseconds)
+PASSED [ 28%]
+ccnp_perf.py::test_svc_get_cc_eventlog_throughput
+----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
+INFO ccnp_perf:ccnp_perf.py:191 Perf test average throughput is: 57.8 ops (operations per second)
+PASSED [ 42%]
+ccnp_perf.py::test_svc_get_cc_eventlog_response
+----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
+INFO ccnp_perf:ccnp_perf.py:213 Perf test average response time is: 76.130223 ms (milliseconds)
+PASSED [ 57%]
+ccnp_perf.py::test_svc_get_cc_report_throughput
+----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
+INFO ccnp_perf:ccnp_perf.py:191 Perf test average throughput is: 54.9 ops (operations per second)
+PASSED [ 71%]
+ccnp_perf.py::test_svc_get_cc_report_response
+----------------------------------------------------------------------- live log call -----------------------------------------------------------------------
+INFO ccnp_perf:ccnp_perf.py:213 Perf test average response time is: 29.38618825 ms (milliseconds)
+PASSED [ 85%]
+ccnp_perf.py::test_ccnp_init PASSED [100%]
+
+=============================================================== 7 passed in 66.95s (0:01:06) ================================================================
+root@ccnp-perf-0:~/ccnp/confidential-cloud-native-primitives#
+```