Artillery-engine-playwright: issue with multiple batches of metrics for period when using pause

[vidhill]

I noticed that I was getting inconsistent values in my reports.

Most notably the report reporting that more VU's completed that created

Version info:

        ___         __  _ ____
  _____/   |  _____/ /_(_) / /__  _______  __ ___
 /____/ /| | / ___/ __/ / / / _ \/ ___/ / / /____/
/____/ ___ |/ /  / /_/ / / /  __/ /  / /_/ /____/
    /_/  |_/_/   \__/_/_/_/\___/_/   \__  /
                                    /____/


VERSION INFO:

Artillery: 2.0.14
Node.js:   v18.20.3
OS:        darwin

Running this command:

npx artillery run artillery-browser.yaml --dotenv=.env

I expected to see this happen:

explanation

I was seeing numerous, Warning: multiple batches of metrics for period warnings,

When trying to chase down the source of the multiple batches of metrics for period warnings I realised that they only occur when I include a pause phase,

Edit, upon further investigation, it appears the issue may not be related to pause, I am seeing a similar issue when I tried substituting pause with one user over a 60 second period: arrivalCount: 1 duration: 60

explanation

I have a feeling that some VUs are completing during the pause phase and perhaps the pause phase is not being recorded

Files being used:

config:
  target: "{{ $processEnvironment.BASE_URL }}"
  engines:
    playwright: {}
  processor: "./tests/index.js"
  phases:
    - name: constantArrival2
      duration: 10
      arrivalRate: 1
    - pause: 60
    - name: constantArrivalRecovery
      duration: 10
      arrivalRate: 1
scenarios:
  - engine: playwright
    testFunction: artilleryScript

Sample test

async function artilleryScript(page, vuContext) {
  await page.goto(vuContext.vars.target);
  await expect(page).toHaveTitle(/Home/);
}

module.exports = {
  artilleryScript,
};

Sample log output

Phase completed: constantArrival2 (index: 0, duration: 10s) 14:37:26(+0100)

Phase started: unnamed (index: 1, duration: 60s) 14:37:26(+0100)

--------------------------------------
Metrics for period to: 14:37:30(+0100) (width: NaNs)
--------------------------------------

No measurements recorded during this period


Warning: multiple batches of metrics for period 1717681040000 2024-06-06T13:37:20.000Z
Warning: multiple batches of metrics for period 1717681040000 2024-06-06T13:37:20.000Z
Phase completed: unnamed (index: 1, duration: 60s) 14:38:26(+0100)

[vidhill]

the width: NaNs suggests an some division error

[vidhill]

the width: NaNs suggests an some division error

I have checked, on line:

artillery/packages/artillery/lib/console-reporter.js

Line 203 in ca01f6f

(report.lastMetricAt - report.firstMetricAt) / 1000 +

Both report.lastMetricAt report.firstMetricAt both have a value of undefined in the case where width: NaNs is printed

[vidhill]

Any update on this,
If the reported metrics aren't correct, this is a kind of a fundamental problem.

I have been very impressed with the project, it would be a shame to need to find an alternative.