An option to disable the performance counter on demand.

[verrantom]

🤔 What's the problem you're trying to solve?

#1793 introduced a change whereby durations on the reports output as part of the cucumber_report.json can be decimal values. This is resulting in AWS CodeBuild's built in test report processor (written in Go) causing issues as the duration is no longer an integer.

✨ What's your proposed solution?

After speaking to @aurelien-reeves they suggested an option that would disable the performance counter on demand.

⛏ Have you considered any alternatives or workarounds?

An alternate workaround would be adding a specific step to our AWS CodeBuild pipeline to undo the changes made in the pull request linked above.

📚 Any additional context?

I raised this issue in the "help-cucumber-js" slack channel and spoke to Aurélien who suggested I raise this feature request.

This text was originally generated from a template, then edited by hand. You can modify the template here.

[davidjgoss]

@verrantom thanks for raising!

From reading up a bit on CodeBuild's test reporting, looks like it parses and uses the (legacy) JSON formatter output. On that basis I think I'd be fine with just changing the JSON formatter to round to integer milliseconds - arguably we've made a breaking change there (by changing it to a decimal) and CodeBuild is just a prominent victim.

@aurelien-reeves what do you think?

[aurelien-reeves]

Thanks for the request @verrantom 👍

@davidjgoss I am pretty sure that a lot of our users still rely a lot on the json formatter, and that they may have already taken advantage of the new accurate measures. That would be a regression to them.

And even if that is not the case, that would seem odd to not have the same value for duration from one formatter to another one.

I would prefer an option to deactivate the high precision measures.

[davidjgoss]

@aurelien-reeves so like:

"formatOptions": {
  "json": {
    "highPrecisionTimestamps": false
  }
}

I'd be cool with that if we're making false the default - so anyone who wants to use high precision with the JSON formatter can enable it, but it's off by default so it's not broken by default on CodeBuild.

[aurelien-reeves]

I was more thinking about a global option that would impact src/time.ts directly, and thus all formatters at once. Not only the JSON one.

And I would have suggest to opt-out of high-precision measures per default. But I would not object into opting-in for it. I think it was a good and reasonable improvement despite that regression with some third-party tools. And it was released as part of a new major version so we did not missed the breaking change in a minor or patched version here.

[davidjgoss]

After digging into this a little today:

• The precision of performance.now() can vary between platforms and can also be pared down at the system's discretion as a security measure. So I think it's fair to say no one can be relying on a specific level of precision.
• Our code for dealing with timing is a little messy and could use a thorough review. We have https://github.com/cucumber/cucumber-js/blob/main/src/runtime/stopwatch.ts and https://github.com/cucumber/cucumber-js/blob/main/src/time.ts doing similar work in different places

I think the best short-term course would be to Math.round the durations for the JSON formatter to fix that regression, and then do a more in-depth review of our timing code.

[davidjgoss]

Released https://github.com/cucumber/cucumber-js/releases/tag/v8.5.1 with a fix for this.