Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arrival rate executors review #19

Merged
merged 6 commits into from
Jan 18, 2023
Merged

Arrival rate executors review #19

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
4d56fb1
change arrival rate examples to avoid recommending the use of maxVUs …
immavalls Jan 8, 2023
2cfd29a
add initial example when we would use this executor
immavalls Jan 8, 2023
69ba0d2
Apply suggestions from Matt's code review
immavalls Jan 18, 2023
0d51da6
fix some typos
immavalls Jan 18, 2023
66ade0b
Apply suggestions from Paul's review
immavalls Jan 18, 2023
461576d
Update from Paul's review
immavalls Jan 18, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
163 changes: 92 additions & 71 deletions ...iate/08-Setting-load-profiles-with-executors/Constant-Arrival-Rate-Exercises.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,39 +2,41 @@

As noted in [Setting load profiles with executors](../08-Setting-load-profiles-with-executors.md#Constant-Arrival-Rate), the _Constant Arrival Rate_ executor has a primary focus on the _iteration rate_ being applied over a specified timeframe.

This is a scenario typically seen with [load testing an API](https://k6.io/docs/testing-guides/api-load-testing/) when there is a need to simulate a constant request rate for a particular API endpoint.

## Exercises

For our exercises, we're going to start by using a very basic script that simply performs an HTTP request and then waits one second before completing the test iteration. We're providing some console output as things change.
Our exercises start with a basic script. It runs an HTTP request on each test iteration and tries to achieve 50 requests per time unit (which defaults to 1 second). We provide some console output as things change.

### Creating our script

Let's begin by implementing our test script with the minimally required configuration. Create a file named _test.js_ with the following content:

```js
import http from 'k6/http';
import { sleep } from 'k6';

export const options = {
scenarios: {
k6_workshop: {
executor: 'constant-arrival-rate',
rate: 10,
rate: 50,
duration: '30s',
preAllocatedVUs: 5,
preAllocatedVUs: 2,
},
},
};

export default function () {
console.log(`[VU: ${__VU}, iteration: ${__ITER}] Starting iteration...`);
http.get('https://test.k6.io/contacts.php');
sleep(1);
}
```

We're starting with the bare minimum to use the executor. Compared to previous executors, this has a bit more required configuration beyond the usual `executor` itself.

Reviewing the additional options, we see that the `rate` and `duration` are now required. This makes sense given the focus of this executor is achieving and maintaining the specified _iteration rate_ over the provided timeframe. Let's defer the discussion of `preAllocatedVUs` for the moment until we get our initial test execution completed.
As noted in the [`constant-arrival-rate` documentation](https://k6.io/docs/using-k6/scenarios/executors/constant-arrival-rate/), `rate` and `duration` are now required. This makes sense given the focus of this executor is to achieve and maintain the specified _iteration rate_ over the provided timeframe.

Let's defer the discussion of `preAllocatedVUs` for the moment until our initial test finishes.

### Initial test run

Expand All @@ -47,92 +49,111 @@ k6 run test.js
Your test will now start printing results to your terminal...

```bash
INFO[0000] [VU: 4, iteration: 0] Starting iteration... source=console
INFO[0000] [VU: 5, iteration: 0] Starting iteration... source=console
WARN[0000] Insufficient VUs, reached 5 active VUs and cannot initialize more executor=constant-arrival-rate scenario=k6_workshop
INFO[0001] [VU: 2, iteration: 1] Starting iteration... source=console
INFO[0002] [VU: 1, iteration: 1] Starting iteration... source=console
...
INFO[0029] [VU: 4, iteration: 26] Starting iteration... source=console
INFO[0029] [VU: 5, iteration: 26] Starting iteration... source=console
execution: local
script: test.js
output: -

running (0m30.7s), 0/5 VUs, 135 complete and 0 interrupted iterations
k6_workshop ✓ [======================================] 0/5 VUs 30s 10 iters/s
scenarios: (100.00%) 1 scenario, 2 max VUs, 1m0s max duration (incl. graceful stop):
* k6_workshop: 50.00 iterations/s for 30s (maxVUs: 2, gracefulStop: 30s)

INFO[0000] [VU: 1, iteration: 0] Starting iteration... source=console
INFO[0000] [VU: 2, iteration: 0] Starting iteration... source=console
WARN[0000] Insufficient VUs, reached 2 active VUs and cannot initialize more executor=constant-arrival-rate scenario=k6_workshop
INFO[0000] [VU: 1, iteration: 1] Starting iteration... source=console
INFO[0000] [VU: 2, iteration: 1] Starting iteration... source=console
...
iterations.....................: 135 4.402609/s
vus............................: 5 min=5 max=5
INFO[0030] [VU: 2, iteration: 204] Starting iteration... source=console
INFO[0030] [VU: 1, iteration: 204] Starting iteration... source=console

running (0m30.1s), 0/2 VUs, 410 complete and 0 interrupted iterations
k6_workshop ✓ [======================================] 0/2 VUs 30s 50.00 iters/s

data_received..................: 325 kB 11 kB/s
data_sent......................: 47 kB 1.6 kB/s
dropped_iterations.............: 1091 36.235918/s
http_req_blocked...............: avg=3.48ms min=1µs med=6µs max=256.93ms p(90)=10µs p(95)=12µs
http_req_connecting............: avg=1.73ms min=0s med=0s max=136.79ms p(90)=0s p(95)=0s
http_req_duration..............: avg=132.35ms min=103.06ms med=115.38ms max=827.03ms p(90)=143.56ms p(95)=165.43ms
{ expected_response:true }...: avg=132.35ms min=103.06ms med=115.38ms max=827.03ms p(90)=143.56ms p(95)=165.43ms
http_req_failed................: 0.00% ✓ 0 ✗ 410
http_req_receiving.............: avg=86.77µs min=19µs med=80µs max=524µs p(90)=142µs p(95)=153.54µs
http_req_sending...............: avg=26.88µs min=7µs med=24.5µs max=194µs p(90)=39.1µs p(95)=43.54µs
http_req_tls_handshaking.......: avg=1.74ms min=0s med=0s max=126.36ms p(90)=0s p(95)=0s
http_req_waiting...............: avg=132.23ms min=102.98ms med=115.28ms max=826.93ms p(90)=143.43ms p(95)=165.28ms
http_reqs......................: 410 13.617531/s
iteration_duration.............: avg=136.06ms min=103.26ms med=115.77ms max=827.24ms p(90)=144.98ms p(95)=183ms
iterations.....................: 410 13.617531/s
vus............................: 2 min=2 max=2
vus_max........................: 2 min=2 max=2
```
Our test ran successfully, but closer inspection of the results shows that our actual results are not as intended.

Our test ran successfully. However, closer inspections show that our results are not as intended.

Looking at the output, we see the following:

```bash
WARN[0000] Insufficient VUs, reached 5 active VUs and cannot initialize more executor=constant-arrival-rate scenario=k6_workshop
WARN[0000] Insufficient VUs, reached 2 active VUs and cannot initialize more executor=constant-arrival-rate scenario=k6_workshop
```

What happened here? Remember the `preAllocatedVUs` setting we glossed over earlier? With this setting, we simply told k6 to start the test with 5 virtual users; after a few iterations, k6 was able to determine that it would **not** be able to achieve our desired iteration rate.
This fact is evident by looking at the `iterations` value in the test summary; our test was only able to attain a rate of 4.4 iterations per second---we wanted 10.
What happened here?

Remember the `preAllocatedVUs` setting we glossed over earlier? With this setting, we told k6 to start the test with 2 virtual users. With such few allocated VUs, k6 could **not** achieve our desired iteration rate (50 iterations/s).

You can confirm this fact in the `iterations` value in the test summary; our test attained a rate of only 13.61 iterations per second and we wanted 50.

We _could_ simply double the number of `preAllocatedVUs`, or better yet, we could allow k6 to automatically control the number of VUs to achieve our rate.
### Adjusting preallocated virtual users

### Autoscaling of virtual users
To restate, _Constant Arrival Rate_ focuses on the _iteration rate_. k6 aims to eliminate the need to be overly concerned about the actual number of users required to achieve such a rate. However, we still must give our script enough VUs to achieve that rate.

Restating once again, the _Constant Arrival Rate_ executor is focused on the _iteration rate_ over a period of time. k6 aims to eliminate the need to be overly concerned about the actual number of users required to achieve such a rate. As a user of the executor, our script can simply specify the minimum and maximum number of VUs allowed, letting k6 handle the actual details. This can be referred to as _autoscaling_.
In the preceding example, the request duration or latency is, on average, 132.35ms, and the 95 percentile is around 165.43ms. With just 2 VUs (`preAllocatedVUs`), in a very optimistic scenario, we cannot expect more than `2 VUs / 0.132 s = 15.15 iterations/s`.

With the `preAllocatedVUs`, we're basically providing the starting number of virtual users. Now we will provide the `maxVUs` to denote our upper bounds. Update the `options` as follows:
Playing a bit with the number of `preAllocatedVUs`, we can update your script to a higher value, e.g. 25.

```js
export const options = {
scenarios: {
k6_workshop: {
executor: 'constant-arrival-rate',
rate: 10,
duration: '30s',
preAllocatedVUs: 5,
maxVUs: 50,
scenarios: {
k6_workshop: {
executor: 'constant-arrival-rate',
rate: 50,
duration: '30s',
preAllocatedVUs: 25,
},
},
},
};
```
To show us when the autoscaling is taking place, let's include the following javascript before the `options` block.
```js
// This will be executed for each VU when initialized
console.log(`Hello, VU #${__VU} has entered the test!`);
```

Let's run our test once again:
```bash
k6 run test.js
```

```bash
INFO[0000] Hello, VU #1 has entered the test! source=console
INFO[0000] Hello, VU #0 has entered the test! source=console
INFO[0000] [VU: 2, iteration: 0] Starting iteration... source=console
INFO[0000] [VU: 5, iteration: 0] Starting iteration... source=console
INFO[0000] [VU: 3, iteration: 0] Starting iteration... source=console
INFO[0000] [VU: 1, iteration: 0] Starting iteration... source=console
INFO[0000] [VU: 4, iteration: 0] Starting iteration... source=console
INFO[0000] Hello, VU #6 has entered the test! source=console
INFO[0000] [VU: 6, iteration: 0] Starting iteration... source=console
INFO[0001] Hello, VU #7 has entered the test! source=console
...
INFO[0030] [VU: 6, iteration: 22] Starting iteration... source=console
INFO[0030] [VU: 8, iteration: 22] Starting iteration... source=console
INFO[0031] Hello, VU #0 has entered the test! source=console

running (0m31.1s), 00/13 VUs, 293 complete and 0 interrupted iterations
k6_workshop ✓ [======================================] 00/13 VUs 30s 10 iters/s
INFO[0031] Hello, VU #0 has entered the test! source=console
...
iterations.....................: 293 9.432192/s
vus............................: 13 min=8 max=13
running (0m30.5s), 00/25 VUs, 1501 complete and 0 interrupted iterations
k6_workshop ✓ [======================================] 00/25 VUs 30s 50.00 iters/s

data_received..................: 1.2 MB 40 kB/s
data_sent......................: 174 kB 5.7 kB/s
http_req_blocked...............: avg=4.26ms min=1µs med=4µs max=332.88ms p(90)=8µs p(95)=14µs
http_req_connecting............: avg=2.05ms min=0s med=0s max=167.98ms p(90)=0s p(95)=0s
http_req_duration..............: avg=136.87ms min=101.78ms med=114.56ms max=862.77ms p(90)=146.68ms p(95)=184.36ms
{ expected_response:true }...: avg=136.87ms min=101.78ms med=114.56ms max=862.77ms p(90)=146.68ms p(95)=184.36ms
http_req_failed................: 0.00% ✓ 0 ✗ 1501
http_req_receiving.............: avg=46.57µs min=11µs med=39µs max=503µs p(90)=82µs p(95)=96µs
http_req_sending...............: avg=17.21µs min=5µs med=15µs max=151µs p(90)=28µs p(95)=30µs
http_req_tls_handshaking.......: avg=2.19ms min=0s med=0s max=203.88ms p(90)=0s p(95)=0s
http_req_waiting...............: avg=136.81ms min=101.73ms med=114.5ms max=862.74ms p(90)=146.57ms p(95)=184.31ms
http_reqs......................: 1501 49.213783/s
iteration_duration.............: avg=141.27ms min=101.88ms med=114.83ms max=862.9ms p(90)=150.38ms p(95)=407.84ms
iterations.....................: 1501 49.213783/s
vus............................: 25 min=25 max=25
vus_max........................: 25 min=25 max=25
```
> :point_up: Feel free to ignore the entries about `VU #0`...it's a bit of a special case.

Reviewing the output now, we see that the desired rate was more closely achieved at 9.43 iterations per second (for the overall test), and that k6 ultimately scaled up to 13 VUs to achieve the desired rate. That's autoscaling!
Reviewing the output now, we see that the desired rate is more closely achieved at 49.21 iterations per second (for the overall test).

You might be tempted to set a lower value for [`preAllocatedVUs` and use `maxVUs` option](https://k6.io/docs/using-k6/scenarios/executors/constant-arrival-rate/) to autoscale the test. `maxVUs` is the maximum number of VUs to allow during the test run, which defaults to `preAllocatedVUs` when not set. However, it's usually best to leave the default value for `maxVUs` and increase the `preAllocatedVUs`. In this way, the VUs are allocated before the test starts and will be used when (and only if) necessary.

> :point_up: By **not** specifying `maxVUs` in our first test, we essentially disabled autoscaling of VUs.
Allocating VUs in the middle of the test can be costly in terms of CPU resources in the load generator instance, and can skew the tests. With preallocating VUs, the test starts with all the VUs available―no need to wait to allocate more when needed in the middle of the test run.

### Other rate options

Expand All @@ -149,22 +170,22 @@ export const options = {
timeUnit: '1h',
duration: '30s',
preAllocatedVUs: 5,
maxVUs: 50,
},
},
};
```

Running the script shows a pace of 3 iterations per second:
Running the script shows a pace of 2.78 iterations per second, and we can get there with 5 VUs.

```bash
INFO[0029] [VU: 1, iteration: 20] Starting iteration... source=console
INFO[0030] [VU: 2, iteration: 20] Starting iteration... source=console
INFO[0031] Hello, VU #0 has entered the test! source=console
scenarios: (100.00%) 1 scenario, 5 max VUs, 1m0s max duration (incl. graceful stop):
* k6_workshop: 2.78 iterations/s for 30s (maxVUs: 5, gracefulStop: 30s)
...
INFO[0029] [VU: 1, iteration: 16] Starting iteration... source=console
INFO[0030] [VU: 5, iteration: 16] Starting iteration... source=console

running (0m30.9s), 00/04 VUs, 83 complete and 0 interrupted iterations
k6_workshop ✓ [======================================] 00/04 VUs 30s 3 iters/s
INFO[0031] Hello, VU #0 has entered the test! source=console
running (0m30.0s), 0/5 VUs, 84 complete and 0 interrupted iterations
k6_workshop ✓ [======================================] 0/5 VUs 30s 2.78 iters/s
...
iterations.....................: 83 2.682393/s
vus............................: 4 min=3 max=4
Expand Down
Loading