Adapt logs format to visualize dist-tests results in Kibana and Grafana

[veaceslavdoina]

Intro

We can easily run dist-tests locally and it works fine. We can run them inside remote Kubernetes cluster as well and it also works.

A single point we miss right now is to get the tests execution status, inside the remote Kubernetes, in an suitable way to be able to

• Identify test execution status
• Identify exact commit of the codex which fails
• Identify exact commit of the tests which fails

We can implement it in the following way

1. Run tests inside remote Kubernetes
2. Ship runner logs to Elasticsearch
3. Visualize logs in Kibana to get the data about tests execution status
4. Search for logs in Kibana to get all tests debug data
5. Visualize tests execution status in Grafana and probably make it publicly available

To implement that we should add some additional data into the logs and also add additional log file.

Proposal

1. Add additional log file to the tests execution

   07-50-31Z_DownloadTests            - Existing - Folder with tests debug logs and metrics
   07-50-31Z_DownloadTests_FAILED.log - Existing - Failed logs
   07-50-31Z_DownloadTests_STATUS.log - Proposed - Test run status
   

2. Add additional fields to every log file

   runid    - Unique ID of the tests run
   category - Category of the tests
   name     - Test name
   

   By using these fields in the Kibana, we would be able to get data for

   • All logs for the specific run
   • All logs for Short or Long tests category
   • All logs for specific test by name

Status log fields

Field	Example	Default	Set mode	Description
time	Jul 17, 2023 @ 10:46:13.455	%time%	Auto by app	Time of the log entry
runid	20230717-104613	null	Via env var	Unique ID of the test run
category	short	%category%	Auto by app	Test category
name	DownloadTests	%name%	Auto by app	Test name
status	successful	%status%	Auto by app	Test execution status
testid	sha-3257e42	null	Via env var	cs-codex-dist-tests commit ID
codexid	codexstorage/nim-codex:sha-a899384	%docker image%	Auto by app	Codex Docker image ID

Status log example

Log format can be in different format, but in case of JSON, parsing should be easier

{"time": "2023-07-17T13:20:30+00:00", "runid": "20230717-132030", "status": "successful", "testid": "sha-3257e42", "codexid": "codexstorage/nim-codex:sha-a899384"}

Elasticsearch Demo index

We can use this example to create Demo index in Elasticsearch and put some data to visualize it in Kibana and Grafana

# Variables
elasticsearch_url=http://localhost:9200
date=$(date +%Y-%m-%d)
index_name=dist-tests-demo-$date
index_type=default
time=$(date +%Y-%m-%d'T'%H:%M:%S.%3N)
runid=$(date +%Y%m%d-%H%M%S)
category=short
# category=long
status=successful
# status=failed
name=DownloadTests
# name=LargeFileTests

# Send to Elasticsearch
curl -H "Content-Type: application/json" \
  -XPOST "$elasticsearch_url/$index_name/_doc" \
  -d '{"time": "'${time}'", "runid": "'${runid}'", "category": "'${category}'", "name": "'${name}'", "status": "'${status}'", "testid": "sha-3257e42", "codexid": "codexstorage/nim-codex:sha-a899384"}'

Kibana and Grafana visualization of the data from the Status log

Screenshots

Implementation

1. Add new log file for every tests using specification above
   • Add RUNID variable to app
   • Add CODEXID variable to app (reuse CODEXDOCKERIMAGE?)
   • Write log file in json format as specified above
2. Add new fields to the existing logs
   • Add RUNID variable to app and it should be added to every log line after the time
   • Add CATEGORY to every log line after the RUNID
   • Add NAME to every log line after the CATEGORY
time - runid - category - name - ...

Comments

1. All variable names and values provided just as an example and should be reviewed and changed in case of need

[veaceslavdoina]

We probably should consider to add a branch name to the table in case we will run tests from a non-default one. If so, app should handle one more variable like BRANCH.