Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration of K6 into API Github workflow #1309

Merged
merged 53 commits into from
Sep 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
70dfadd
Add workflow load-tests
MogageNicolae Aug 16, 2024
11016d3
Add initialization of the project
MogageNicolae Aug 16, 2024
b434f41
Fix port when waiting for API
MogageNicolae Aug 16, 2024
f8a972c
Add config for API
MogageNicolae Aug 16, 2024
82fbf5e
Add folder content for debug
MogageNicolae Aug 16, 2024
b769cbf
Add folder content for debug before api start
MogageNicolae Aug 16, 2024
9edab35
Add debug for config file
MogageNicolae Aug 16, 2024
fb582d5
Remove config.yaml from gitignore
MogageNicolae Aug 16, 2024
7ddb134
Add config.yaml file in dist
MogageNicolae Aug 16, 2024
c8ac3ae
Move config copy after build
MogageNicolae Aug 16, 2024
26666f6
Add docker services
MogageNicolae Aug 16, 2024
1fca5cd
Remove wrong added -
MogageNicolae Aug 16, 2024
87caacd
Change rabbitmq version
MogageNicolae Aug 16, 2024
559b3c1
Add background API start
MogageNicolae Aug 16, 2024
b83aa80
Add docker installation in workflow
MogageNicolae Aug 19, 2024
1366341
Changed docker compose version use to start services
MogageNicolae Aug 19, 2024
e9ed9b1
Changed docker compose version
MogageNicolae Aug 19, 2024
3f688d8
Add docker compose global
MogageNicolae Aug 19, 2024
49cdb9c
Change docker compose plugin version
MogageNicolae Aug 19, 2024
9d8b363
Change docker compose plugin version
MogageNicolae Aug 19, 2024
a401b46
Update trend names
MogageNicolae Aug 19, 2024
ef05bc7
Add cache preload
MogageNicolae Aug 19, 2024
3c9a66b
Add 1 minute duration
MogageNicolae Aug 19, 2024
9e43001
Merge pull request #1312 from multiversx/SERVICES-2549-test-integrati…
MogageNicolae Aug 20, 2024
1284d94
Add preload for all tokens/nodes
MogageNicolae Aug 20, 2024
33ba747
Add preload for all tokens/nodes
MogageNicolae Aug 20, 2024
18047b7
Merge branches
MogageNicolae Aug 20, 2024
c5e9b1c
Merge pull request #1313 from multiversx/SERVICES-2549-test-integrati…
MogageNicolae Aug 20, 2024
0b76266
test
MogageNicolae Aug 20, 2024
b0355a8
Change table
MogageNicolae Aug 20, 2024
1edf2de
Change wrong closing tag for table
MogageNicolae Aug 20, 2024
d2dfd66
Refactor generateTable function
MogageNicolae Aug 20, 2024
497ed24
Add more endpoints
MogageNicolae Aug 21, 2024
c6dd914
Fix wrong name for Trends
MogageNicolae Aug 21, 2024
31e0810
Merge pull request #1314 from multiversx/SERVICES-2549-test-integrate…
MogageNicolae Aug 21, 2024
53e3508
Add empty line at end of file
MogageNicolae Aug 21, 2024
9c7b8fe
Fix wrong trend calls
MogageNicolae Aug 21, 2024
92a4f15
Merge pull request #1315 from multiversx/SERVICES-2549-test-integrata…
MogageNicolae Aug 21, 2024
4c334eb
Merge branch 'development' into SERVICES-2549-integrate-k-6-s-in-api
tanghel Sep 10, 2024
147e2b6
Changed actions/upload-artifacts from v2 to v3
GuticaStefan Sep 10, 2024
6c16703
changed actions/download-artifact from v2 to v3
GuticaStefan Sep 10, 2024
a2d9167
Clear docker images caching in workflow
GuticaStefan Sep 10, 2024
833addf
removed unncessary clear docker images cache
GuticaStefan Sep 10, 2024
5d26ad3
Update load-tests.yml
cfaur09 Sep 10, 2024
bdc2c81
Update load-test.yml
GuticaStefan Sep 10, 2024
061a5c7
Merge branch 'development' into SERVICES-2549-integrate-k-6-s-in-api
cfaur09 Sep 10, 2024
f139cb7
updated preload.js path
GuticaStefan Sep 10, 2024
88cf4a9
update path for preload.js
GuticaStefan Sep 10, 2024
5b041bb
test
GuticaStefan Sep 10, 2024
a1deceb
disable preload cache
cfaur09 Sep 10, 2024
609e11d
revert ref base
cfaur09 Sep 10, 2024
10c56b8
run action for PRs to main or development only
GuticaStefan Sep 11, 2024
2b41e61
update load-tests.yml
GuticaStefan Sep 11, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
186 changes: 186 additions & 0 deletions .github/workflows/load-tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,186 @@
name: Load Tests

on:
push:
branches: [main, development]
pull_request:
branches: [main, development]

jobs:
test-base:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.base.sha }}

- name: Set up Node.js
uses: actions/setup-node@v2
with:
node-version: '20'

- name: Install dependencies
run: npm ci

- name: Initialize the project
run: npm run init

- name: Build
run: npm run build

- name: Copy devnet config file from src to dist
run: cp ./config/config.devnet.yaml ./dist/config/config.yaml

- name: Start docker services
run: docker compose up -d

- name: Start Node.js API
run: node ./dist/src/main.js &

- name: Install k6
run: |
sudo gpg -k
sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update
sudo apt-get install k6

- name: Wait for API to be ready
run: |
until curl --output /dev/null --silent --fail http://localhost:4001/hello; do
echo 'Waiting for API...'
sleep 1
done

- name: Preload cache
run: k6 run ./k6/preload.js

- name: Run k6 Load Test
run: k6 run ./k6/script.js

- name: Upload result file for base branch
uses: actions/upload-artifact@v3
with:
name: base-results
path: k6/output/summary.json

- name: Stop docker services
run: docker compose down

test-head:
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v2
with:
ref: ${{ github.event.pull_request.head.sha }}

- name: Set up Node.js
uses: actions/setup-node@v2
with:
node-version: '20'

- name: Install dependencies
run: npm ci

- name: Initialize the project
run: npm run init

- name: Build
run: npm run build

- name: Copy devnet config file from src to dist
run: cp ./config/config.devnet.yaml ./dist/config/config.yaml

- name: Start docker services
run: docker compose up -d

- name: Start Node.js API
run: node ./dist/src/main.js &

- name: Install k6
run: |
sudo gpg -k
sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
sudo apt-get update
sudo apt-get install k6

- name: Wait for API to be ready
run: |
until curl --output /dev/null --silent --fail http://localhost:4001/hello; do
echo 'Waiting for API...'
sleep 1
done

- name: Preload cache
run: k6 run ./k6/preload.js

- name: Run k6 Load Test
run: k6 run ./k6/script.js

- name: Upload result file for head branch
uses: actions/upload-artifact@v3
with:
name: head-results
path: k6/output/summary.json

- name: Stop docker services
run: docker compose down

compare-results:
runs-on: ubuntu-latest

needs: [test-base, test-head]
steps:
- uses: actions/checkout@v2

- name: Download all artifacts
uses: actions/download-artifact@v3
with:
path: artifacts

- name: Set up Node.js
uses: actions/setup-node@v2
with:
node-version: '20'

- name: Compare test results
run: |
node ./k6/compare-results.js ${{ github.event.pull_request.base.sha }} artifacts/base-results/summary.json ${{ github.event.pull_request.head.sha }} artifacts/head-results/summary.json report.md

- name: Render the report from the template
id: template
uses: chuhlomin/render-template@v1
if: github.event_name == 'pull_request'
with:
template: report.md
vars: |
base: ${{ github.event.pull_request.base.sha }}
head: ${{ github.event.pull_request.head.sha }}

- name: Upload the report markdown
uses: actions/upload-artifact@v3
if: github.event_name == 'pull_request'
with:
name: report-markdown
path: report.md

- name: Find the comment containing the report
id: fc
uses: peter-evans/find-comment@v2
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
with:
issue-number: ${{ github.event.pull_request.number }}
comment-author: 'github-actions[bot]'
body-includes: 'k6 load testing comparison'

- name: Create or update the report comment
uses: peter-evans/create-or-update-comment@v2
if: github.event_name == 'pull_request' && github.event.pull_request.head.repo.full_name == github.repository
with:
comment-id: ${{ steps.fc.outputs.comment-id }}
issue-number: ${{ github.event.pull_request.number }}
body: ${{ steps.template.outputs.result }}
edit-mode: replace
2 changes: 2 additions & 0 deletions k6/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
output/*
!output/.gitkeep
124 changes: 124 additions & 0 deletions k6/compare-results.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
const fs = require('fs');

function generateComparisonTable(baseCommitHash, baseMetricsPath, targetCommitHash, targetMetricsPath, outputPath) {
// Load JSON outputs from k6
const baseMetrics = JSON.parse(fs.readFileSync(baseMetricsPath, 'utf8'));
const targetMetrics = JSON.parse(fs.readFileSync(targetMetricsPath, 'utf8'));

const baseData = extractMetrics(baseMetrics);
const targetData = extractMetrics(targetMetrics);

const table = generateTable(baseCommitHash, baseData, targetCommitHash, targetData);

fs.writeFileSync(outputPath, table);
}

function extractMetrics(metrics) {
const extractedMetrics = {};
const metricKeys = Object.keys(metrics.metrics);

for (const key of metricKeys) {
if (key.endsWith('_http_req_duration')) {
const values = metrics.metrics[key].values;
const avgResponseTime = values.avg;
const maxResponseTime = values.max;
const p90 = values['p(90)'];
const p95 = values['p(95)'];

const name = key.split('_')[0].charAt(0).toUpperCase() + key.split('_')[0].slice(1);

if (!extractedMetrics[name]) {
extractedMetrics[name] = { avgResponseTime, maxResponseTime, p90, p95 };
} else {
extractedMetrics[name].avgResponseTime = avgResponseTime;
extractedMetrics[name].maxResponseTime = maxResponseTime;
extractedMetrics[name].p90 = p90;
extractedMetrics[name].p95 = p95;
}
}
}

extractedMetrics['Test Run Duration'] = metrics.state.testRunDurationMs;

return extractedMetrics;
}

function generateTable(baseCommitHash, baseData, targetCommitHash, targetData) {
const headers = ['Avg', 'Max', '90', '95'];
let table = `k6 load testing comparison.\nBase Commit Hash: ${baseCommitHash}\nTarget Commit Hash: ${targetCommitHash}\n\n`;
table += '<table><tr> \
<th rowspan="2">Metric</th> \
<th colspan="4">Base</th> \
<th colspan="4">Target</th> \
<th colspan="4">Diff</th> \
</tr><tr> '
for (let i = 0; i < 3; i++) {
headers.forEach(header => {
table += `<th>${header}</th>`;
});
}
table += '</tr>';

for (const key of Object.keys(baseData)) {
if (key === 'Test Run Duration') {
continue;
}
const baseAvg = baseData[key].avgResponseTime;
const targetAvg = targetData[key].avgResponseTime;
const baseMax = baseData[key].maxResponseTime;
const targetMax = targetData[key].maxResponseTime;
const baseP90 = baseData[key].p90;
const targetP90 = targetData[key].p90;
const baseP95 = baseData[key].p95;
const targetP95 = targetData[key].p95;

const avgDiff = getDifferencePercentage(baseAvg, targetAvg);
const maxDiff = getDifferencePercentage(baseMax, targetMax);
const p90Diff = getDifferencePercentage(baseP90, targetP90);
const p95Diff = getDifferencePercentage(baseP95, targetP95);

const avgColor = getColor(baseAvg, targetAvg);
const maxColor = getColor(baseMax, targetMax);
const p90Color = getColor(baseP90, targetP90);
const p95Color = getColor(baseP95, targetP95);

table += `<tr><td><b>${key}</b></td>`;
table += `<td>${baseAvg.toFixed(2)}</td><td>${baseMax.toFixed(2)}</td><td>${baseP90.toFixed(2)}</td><td>${baseP95.toFixed(2)}</td>`;
table += `<td>${targetAvg.toFixed(2)}</td><td>${targetMax.toFixed(2)}</td><td>${targetP90.toFixed(2)}</td><td>${targetP95.toFixed(2)}</td>`;
table += `<td>${avgDiff} ${avgColor}</td><td>${maxDiff} ${maxColor}</td><td>${p90Diff} ${p90Color}</td><td>${p95Diff} ${p95Color}</td></tr>`;
}

const baseDuration = baseData['Test Run Duration'].toFixed(2);
const targetDuration = targetData['Test Run Duration'].toFixed(2);
table += `<tr><td><b>Test Run Duration</b></td><td colspan="4">${baseDuration}</td><td colspan="4">${targetDuration}</td><td colspan="4"></td></tr></table>`;
table += '\n\nLegend: Avg - Average Response Time, Max - Maximum Response Time, 90 - 90th Percentile, 95 - 95th Percentile\nAll times are in milliseconds.\n';

return table;
}

function getColor(baseValue, targetValue) {
if (baseValue >= targetValue) {
return '✅'; // Green emoji for improvement or equivalence
} else {
return '🔴'; // Red emoji for degradation
}
}

function getDifferencePercentage(baseValue, targetValue) {
const difference = ((targetValue - baseValue) / baseValue) * 100;
const sign = difference >= 0 ? '+' : '';
return `${sign}${difference.toFixed(2)}%`;
}

if (process.argv.length !== 7) {
console.error('Usage: node compare-results.js baseCommitHash baseMetricsPath targetCommitHash targetMetricsPath outputFile');
process.exit(1);
}

const baseCommitHash = process.argv[2];
const baseMetricsPath = process.argv[3];
const targetCommitHash = process.argv[4];
const targetMetricsPath = process.argv[5];
const outputPath = process.argv[6];

generateComparisonTable(baseCommitHash, baseMetricsPath, targetCommitHash, targetMetricsPath, outputPath);
Empty file added k6/output/.gitkeep
Empty file.
10 changes: 10 additions & 0 deletions k6/preload.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
import http from 'k6/http';

const BASE_URL = 'http://localhost:3001';

export default function preloadCache() {
const numberofTokens = http.get(`${BASE_URL}/tokens/count`);
http.get(`${BASE_URL}/tokens?size=${numberofTokens}`);
const numberofNodes = http.get(`${BASE_URL}/nodes/count`);
http.get(`${BASE_URL}/nodes?size=${numberofNodes}`);
}
Loading
Loading