Update benchmarking scripts

.github/workflows/gha.yml

@@ -25,7 +25,10 @@ jobs:
         run: echo ::set-output name=ansible_files::$(grep -lr 'ft=ansible' ansible-playbook)
 
       - name: Run ansible linter
-        uses: ansible/ansible-lint-action@master
+        uses: ansible/ansible-lint-action@v4-eol
         with:
           args: "-p -c .ansible-lint"
+          override-deps: |
+            ansible==2.9.17
+            ansible-lint==4.2.0
           targets: ${{ steps.vars.outputs.ansible_files }}

README.md

@@ -1,21 +1,21 @@
 [//]: # (SPDX-License-Identifier: BSD-3-Clause)
-[//]: # (Copyright 2021, Intel Corporation)
+[//]: # (Copyright 2021-2022, Intel Corporation)
 
 # redis-cluster on k8s testcase
 
-In this repository you can find scripts to run a memtier benchmark on a cluster of k8s-hosted Redis & TieredMemDB servers. 
+In this repository you can find scripts to run a memtier_benchmark on a cluster of k8s-hosted [Redis](https://redis.io/) & [TieredMemDB](https://tieredmemdb.io/) servers. 
 
 ## Repository structure
 The repository is divided into subdirectories as follows:
-* [`ansible-playbook`](./ansible-playbook) - contains ansible playbooks to run a memtier benchmark on a k8s-hosted redis-cluster(s).
+* [`ansible-playbook`](./ansible-playbook) - contains Ansible playbooks to run a memtier_benchmark on a k8s-hosted redis-cluster(s).
 * [`docker_bitnami`](./docker_bitnami) - contains TieredMemDB Docker image. Dockerfile is aligned with [Bitnami redis-cluster](https://github.com/bitnami/bitnami-docker-redis-cluster) image.
 * [`systemd_unit`](./systemd_unit) - contains systemd unit files for exposing PMem devices as system-ram.
 
 To get more information about usage of scripts in the subdirectories, follow their respective READMEs.
 
 ## Prerequisites
 Before you start working with the repository, make sure you have installed:
-* **Ansible 2.9**
+* **Python 3.6+**
 * **Helm**
-* **k8s**
+* **Kubernetes**
 * **Docker** (required to build TieredMemDB image)

ansible-playbook/README.md

@@ -1,10 +1,10 @@
 [//]: # (SPDX-License-Identifier: BSD-3-Clause)
-[//]: # (Copyright 2021, Intel Corporation)
+[//]: # (Copyright 2021-2022, Intel Corporation)
 
 # redis-cluster and memtier_benchmark on k8s playbooks
 
 ## General info
-The redis-cluster testing can be run for **Redis** or for **TieredMemDB**, to choose between the databases set the `test_mode` variable value to `redis` or to `tmdb`.
+The redis-cluster testing can be run for [**Redis**](https://redis.io/) or for [**TieredMemDB**](https://tieredmemdb.io/), to choose between the databases set the `test_mode` variable value to `redis` or to `tmdb`.
 
 The test can be run in three variants:
 * **single cluster** - when only one redis-cluster is deployed and benchmarked;
@@ -17,9 +17,11 @@ The test can be run in three variants:
 #### Inventory file
 To create an `inventory` file, copy the content of `inventory.example` file and replace the example hostnames for `[computes]`, `[utility]` and `[clients]` groups with own hostnames or IPs.
 * The `[computes]` group specifies k8s nodes, where redis-cluster(s) instances are deployed.
-* The `[utility]` group specifies a node, which runs the playbooks (ansible controller). The node can reside outside of the k8s cluster, but it needs to be able to communicate with the cluster.
+* The `[utility]` group specifies a node, which runs the playbooks (Ansible controller). The node can reside outside of the k8s cluster, but it needs to be able to communicate with the cluster.
 * The `[clients]` group specifies k8s nodes, which run memtier_benchmark instances.
 
+Then, to enable passwordless connection, generate ssh keys for all the hosts specified in `inventory` file and copy them to `[utility]` node.
+
 #### Changing default values of environment variables
 There are two ways to change default values of environment variables:
 * by passing new values as command-line arguments when invoking playbooks:
@@ -37,6 +39,9 @@ To export benchmark results to S3 bucket right after testcase run, specify [S3 b
 clusters_count=<optional> run_id=<run_identifier> ansible-playbook -i inventory run_benchmark.yml --extra-vars "@vars/s3_vars.yml"
 ```
 
+#### Preparing for running playbooks behind proxy server
+To run playbooks behind proxy, specify default values of [Proxy specific](#proxy-specific) variables in `vars/proxy_vars.yml`. Then, invoke playbooks as usual (as described in [How to run](#how-to-run) section).
+
 ### Testcase execution
 
 #### Running single cluster testcase
@@ -49,6 +54,9 @@ To run multiple clusters testcase, the `clusters_count` variable needs to be set
 For a single cluster case, cluster objects are deployed to `redis-cluster1` k8s namespace.\
 For multiple clusters, each redis-cluster is created in separate namespace named according to the convention: `redis-cluster<cluster-number>`.
 
+#### Test run identifier
+By default the value of `run_id` variable is set according to the convention: `<four-character-random-hash>-th{{ threads }}-cln{{ clients }}-p{{ pipeline }}-req{{ requests }}`. It can be accessed after execution of `populate.yml` playbook in `group_vars/all` file.
+
 ### After execution
 
 #### Collecting results
@@ -57,6 +65,10 @@ Benchmark logs and run metadata are uploaded to `{{ logdir_default }}/{{ log_dir
 ## How to run
 
 ### Prework
+Prepare OS for running Ansible playbooks - install Ansible and other necessary Python packages using pip (needs to be run only once):
+```shell
+pip3 install -r utils/controller_requirements.txt
+```
 Prepare OS for running benchmarking playbooks - install necessary Python packages and Ansible collections (needs to be run only once):
 ```shell
 ansible-playbook -i inventory setup.yml
@@ -82,12 +94,12 @@ clusters_count=<optional> test_mode=<redis_or_tmdb> ansible-playbook -i inventor
 ```shell
 clusters_count=<optional> run_id=<run_identifier> ansible-playbook -i inventory populate.yml
 ```
-> For this step you can modify [memtier_benchmark specific](#memtier_benchmark-specific) variables.
+> For this step you can modify selected [memtier_benchmark specific](#memtier_benchmark-specific) variables.
 
 \
-**3. Run memtier_benchmark** (use the same `run_id` as in the population step):
+**3. Run memtier_benchmark**:
 ```shell
-clusters_count=<optional> run_id=<run_identifier> ansible-playbook -i inventory run_benchmark.yml [--extra-vars "@vars/s3_vars.yml"]
+clusters_count=<optional> ansible-playbook -i inventory run_benchmark.yml [--extra-vars "@vars/s3_vars.yml"]
 ```
 > For this step you can modify [memtier_benchmark specific](#memtier_benchmark-specific) variables.
 
@@ -105,11 +117,11 @@ clusters_count=<optional> test_mode=<redis_or_tmdb> run_id=<run_identifier> ansi
 ## Environment variables (selection)
 
 ### General
-* `run_id`
 * `test_mode`: Possible values - `redis` or `tmdb`;
 * `clusters_count`: Number of redis-clusters to deploy (default: `1`);
 * `collect_emon_data`: Specifies whether to run EMON during benchmark execution (default: `False`);
 * `multiple_clusters_balanced`: Specifies whether to run balanced testcase (default: `True`) (further description in [General info](#general-info));
+* `run_id`: Test run identifier (default: auto-generated value) (further description in [Test run identifier](#test-run-identifier));
 * `warm_up_benchmark_run`: Specifies whether to run one warm-up benchmark run before test execution (default: `False`);
 
 ### Redis & TieredMemDB specific
@@ -119,10 +131,15 @@ clusters_count=<optional> test_mode=<redis_or_tmdb> run_id=<run_identifier> ansi
 * `pmem_variant`: Specifies variant of Persistent Memory allocation (default: `multiple`)
 
 ### memtier_benchmark specific
-* `clients`
-* `datasize`
-* `pipeline`
-* `requests`
+* `clients`: Number of clients per thread (default: `3`);
+* `datasize`: Object data size in bytes (default: `1024`);
+* `key_pattern`: Set:Get requests pattern used for benchmark execution (default: `G:G`);
+* `pipeline`: Number of concurrent pipelined requests (default: `1`);
+* `ratio`: Set:Get requests ratio used for benchmark execution (default: `1:4`);
+* `requests`: Number of total requests per client (default: `150000000`);
+* `run_count`: Number of full-test iterations to perform during benchmark execution (default: `3`);
+* `test_time`: Number of seconds to run one full-test iteration during benchmark execution (default: `1800`);
+* `threads`: Number of threads (default: `8`);
 
 ### S3 bucket specific
 * `S3_ACCESS_KEY`
@@ -131,11 +148,18 @@ clusters_count=<optional> test_mode=<redis_or_tmdb> run_id=<run_identifier> ansi
 * `S3_SECRET_KEY`
 
 ### Registry specific - Redis
-* `redis_registry`
-* `redis_repository`
-* `redis_tag`
+* `redis_registry`: (default: `docker.io`);
+* `redis_repository`: (default: `bitnami/redis-cluster`);
+* `redis_tag`: (default: `6.2.6-debian-10-r196`);
 
 ### Registry specific - TieredMemDB
 * `tmdb_registry`
 * `tmdb_repository`
 * `tmdb_tag`
+
+### Proxy specific
+* `no_proxy`
+* `proxy`
+
+### PMem specific
+* `pmem_generation`: Specifies whether PMem dimms available in the system are Intel Optane Persistent Memory 100 series (Apache Pass/1st gen) or Intel Optane Persistent Memory 200 series (Barlow Pass/2nd gen). Possible values - `1` or `2` (default: `2`);

ansible-playbook/deploy.yml

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2021, Intel Corporation
+# Copyright 2021-2022, Intel Corporation
 #
 ---
 - hosts: utility[0]
@@ -12,6 +12,9 @@
     - name: "deploy | Load playbook variables"
       include_vars: "vars/vars.yml"
 
+    - name: "deploy | Load proxy specific playbook variables"
+      include_vars: "vars/proxy_vars.yml"
+
   tasks:
     - name: "deploy | Clear variables"
       set_fact:
@@ -33,8 +36,8 @@
 
     - name: "deploy | Set config files paths for Redis and TieredMemDB"
       set_fact:
-        redis_chart_values: "{{ playbook_dir }}/templates/helm_chart_values/redis-cluster_Redis_values.yml.j2"
-        tmdb_chart_values: "{{ playbook_dir }}/templates/helm_chart_values/redis-cluster_TieredMemDB_values.yml.j2"
+        redis_chart_values: "{{ playbook_dir }}/templates/helm_chart_values/redis-cluster_Redis_values_{{ chart_version }}.yml.j2"
+        tmdb_chart_values: "{{ playbook_dir }}/templates/helm_chart_values/redis-cluster_TieredMemDB_values_{{ chart_version }}.yml.j2"
         multiple_clusters_chart_configs: [ "first-group", "second-group" ]
 
     - name: "deploy | Use config for {{ 'Redis' if test_mode == 'redis' else 'TieredMemDB' }}"
@@ -77,6 +80,9 @@
           --post-renderer "{{ playbook_dir }}/templates/helm_chart_deployment/helm_post_renderer_hook.sh"
         environment:
           KUSTOMIZE_DIR: "{{ k8s_definitions_dir }}/helm_chart_deployment"
+          http_proxy: "{{ proxy }}"
+          https_proxy: "{{ proxy }}"
+          no_proxy: "{{ no_proxy }}"
         loop: "{{ namespaces }}"
         loop_control:
           loop_var: redis_cluster_namespace
@@ -106,6 +112,9 @@
           --post-renderer "{{ playbook_dir }}/templates/helm_chart_deployment/helm_post_renderer_hook.sh"
         environment:
           KUSTOMIZE_DIR: "{{ k8s_definitions_dir }}/helm_chart_deployment"
+          http_proxy: "{{ proxy }}"
+          https_proxy: "{{ proxy }}"
+          no_proxy: "{{ no_proxy }}"
         loop: "{{ namespaces }}"
         loop_control:
           loop_var: redis_cluster_namespace
@@ -132,6 +141,9 @@
           --post-renderer "{{ playbook_dir }}/templates/helm_chart_deployment/helm_post_renderer_hook.sh"
         environment:
           KUSTOMIZE_DIR: "{{ k8s_definitions_dir }}/helm_chart_deployment"
+          http_proxy: "{{ proxy }}"
+          https_proxy: "{{ proxy }}"
+          no_proxy: "{{ no_proxy }}"
         loop: "{{ namespaces }}"
         loop_control:
           loop_var: redis_cluster_namespace
@@ -152,4 +164,8 @@
       retries: 10
       delay: 30
 
+    - name: "deploy | Wait for redis-cluster deployments to be ready"
+      pause:
+        seconds: 120
+
 # vi:et:sw=2 ts=2 sts=2 ft=ansible

ansible-playbook/destroy.yml

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2021, Intel Corporation
+# Copyright 2021-2022, Intel Corporation
 #
 ---
 - hosts: utility[0]
@@ -32,4 +32,10 @@
       loop_control:
         loop_var: redis_cluster_namespace
 
+    - name: "destroy | Remove run_id from group_vars"
+      lineinfile:
+        path: "{{ playbook_dir }}/group_vars/all"
+        regexp: "^run_id:"
+        state: absent
+
 # vi:et:sw=2 ts=2 sts=2 ft=ansible

ansible-playbook/include/playbooks/process_emon.yml

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2021, Intel Corporation
+# Copyright 2021-2022, Intel Corporation
 #
 ---
 - hosts: utility[0]
@@ -18,8 +18,6 @@
       file:
         path: "{{ logdir }}/{{ log_dir_structure }}/{{ run_id }}"
         state: "directory"
-        owner: "solben"
-        group: "solben"
 
 - hosts: computes
   pre_tasks:
@@ -86,13 +84,17 @@
   become_user: root
 
   tasks:
+    - set_fact:
+        computes_hostnames: "{{ ' '.join(groups['computes']) }}"
+
     - name: "process_emon | Set processing script flags"
       set_fact:
-        processing_script_flags: "{{ '--single-cluster' if single_cluster_testcase else '' }}"
+        processing_script_flags: "{{ '' if single_cluster_testcase else '--multiple-clusters-hostnames ' + computes_hostnames }}"
 
     - name: "process_emon | Populate excel template with processed emon data"
-      command: |
-        {{ processing_script_path }}/emon_parser_redis-cluster.py {{ processing_script_flags }} \
+      shell: |
+        {{ playbook_dir }}/../../redis-cluster-venv/bin/python3 {{ processing_script_path }}/emon_parser_redis-cluster.py {{ processing_script_flags }} \
+        --pmem-generation {{ pmem_generation }} \
         --paths {{ logdir }}/{{ log_dir_structure }}/{{ run_id }} \
         --name {{ logdir }}/{{ log_dir_structure }}/{{ run_id }}/{{ run_id }}-results
 

ansible-playbook/include/playbooks/process_memtier_results.yml

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2021, Intel Corporation
+# Copyright 2021-2022, Intel Corporation
 #
 ---
 - hosts: utility[0]
@@ -18,8 +18,6 @@
       file:
         path: "{{ logdir }}/{{ log_dir_structure }}/{{ run_id }}"
         state: "directory"
-        owner: "solben"
-        group: "solben"
 
 - hosts: clients
   become: true

ansible-playbook/include/playbooks/run_emon.yml

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2021, Intel Corporation
+# Copyright 2021-2022, Intel Corporation
 #
 ---
 - hosts: computes
@@ -68,8 +68,8 @@
 
     - name: "run_emon | Get *-events.txt file"
       copy:
-        src: "{{ emon_events_file_dir }}/clx-2s-events.txt"
-        dest: "{{ edp_metrics_dir }}/clx-2s-events.txt"
+        src: "{{ emon_events_file_dir }}/{{ emon_events_file }}"
+        dest: "{{ edp_metrics_dir }}/{{ emon_events_file }}"
       become: yes
 
     - name: "run_emon | Check if SEP module is already loaded"
@@ -92,7 +92,7 @@
     - name: "run_emon | Run emon and write to emon.dat"
       shell: |
         /opt/intel/sep/bin64/emon -v -t1 -l0 -i \
-        {{ edp_metrics_dir }}/clx-2s-events.txt > /opt/{{ run_id }}/EMON/emon-{{ hostname }}-{{ run_id }}-{{ exec_stamp }}.dat &
+        {{ edp_metrics_dir }}/{{ emon_events_file }} > /opt/{{ run_id }}/EMON/emon-{{ hostname }}-{{ run_id }}-{{ exec_stamp }}.dat &
       async: 10
       poll: 0
       become: yes

ansible-playbook/include/tasks/benchmark_clusters.yml

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2021, Intel Corporation
+# Copyright 2021-2022, Intel Corporation
 #
 ---
 - set_fact:
@@ -9,7 +9,7 @@
 
 - name: "benchmark_clusters | Save kustomized {{ job_name }} jobs definitions to files"
   copy:
-    content: "{{ lookup('kubernetes.core.kustomize', dir=k8s_definitions_dir_path + '/' + redis_cluster_namespace + '/run_tests_timed', binary_path='/usr/local/bin/kustomize') }}"  # noqa 204
+    content: "{{ lookup('kubernetes.core.kustomize', dir=k8s_definitions_dir_path + '/' + redis_cluster_namespace + '/run_tests_timed') }}"
     dest: "{{ k8s_definitions_dir_path }}/{{ redis_cluster_namespace }}/run_tests_timed/kustomized_benchmark_run_tests_job.yml"
   loop: "{{ benchmark_namespaces }}"
   loop_control:

ansible-playbook/include/tasks/populate_clusters.yml

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2021, Intel Corporation
+# Copyright 2021-2022, Intel Corporation
 #
 ---
 - set_fact:
@@ -9,7 +9,7 @@
 
 - name: "populate_clusters | Deploy {{ job_name }} jobs inside namespaces"
   kubernetes.core.k8s:
-    resource_definition: "{{ lookup('kubernetes.core.kustomize', dir=k8s_definitions_dir + '/' + redis_cluster_namespace + '/populate_db', binary_path='/usr/local/bin/kustomize') }}"  # noqa 204
+    resource_definition: "{{ lookup('kubernetes.core.kustomize', dir=k8s_definitions_dir + '/' + redis_cluster_namespace + '/populate_db') }}"
     state: present
   ignore_errors: yes
   register: deployed_jobs_names_stdout

ansible-playbook/populate.yml

@@ -1,5 +1,5 @@
 # SPDX-License-Identifier: BSD-3-Clause
-# Copyright 2021, Intel Corporation
+# Copyright 2021-2022, Intel Corporation
 #
 ---
 - import_playbook: "include/playbooks/check_dimms.yml"
@@ -15,6 +15,23 @@
       include_vars: "vars/vars.yml"
 
   tasks:
+    - name: "populate | Generate run_id variable"
+      set_fact:
+        run_id: "{{ lookup('env', 'run_id') | default((99999999 | random | to_uuid | hash('md5'))[:4], true) }}\
+                  -th{{ threads }}-cln{{ clients }}-p{{ pipeline }}-req{{ requests }}"
+
+    - name: "populate | Write run_id to group_vars"
+      lineinfile:
+        create: true
+        insertafter: EOF
+        line: "run_id: {{ run_id }}"
+        path: "{{ playbook_dir }}/group_vars/all"
+        state: present
+
+    - name: "populate | Print run_id"
+      debug:
+        var: run_id
+
     - name: "populate | Clear variables"
       set_fact:
         namespaces_counter: []