fix(benchmark): clear `BenchmarkResult.samples` array to reduce memory usage #6541

hi-ogawa · 2024-09-21T05:20:32Z

Description

closes Vitest has too much memory usage in the face of a large number of benches #6539

I think we should clear BenchmarkResult.samples by default on test runner side. I added benchmark.includeSamples option which allows returning samples, but it's disabled by default.

The fix is verified with this reproduction https://github.com/hi-ogawa/reproductions/tree/main/vitest-6539-bench-heap-oom

Plot of "used_heap_size"

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

It's really useful if your PR references an issue where it is discussed ahead of time. If the feature is substantial or introduces breaking changes without a discussion, PR might be closed.
Ideally, include a test that fails without this PR but passes with it.
Please, don't make changes to pnpm-lock.yaml unless you introduce a new test example.

Tests

Run the tests with pnpm test:ci.

Documentation

If you introduce new functionality, document it. You can run documentation with pnpm run docs command.

Changesets

Changes in changelog are generated from PR name. Please, make sure that it explains your changes in an understandable manner. Please, prefix changeset messages with feat:, fix:, perf:, docs:, or chore:.

…ult`

netlify · 2024-09-21T05:23:12Z

✅ Deploy Preview for vitest-dev ready!

Built without sensitive environment variables

Name	Link
🔨 Latest commit	`637fe11`
🔍 Latest deploy log	https://app.netlify.com/sites/vitest-dev/deploys/66ee57a21ac8230008e1635d
😎 Deploy Preview	https://deploy-preview-6541--vitest-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

AriPerkkio

Would be great to add test case for this in https://github.com/vitest-tests/ and include it in Ecosystem CI. We already have some similar cases there. This one could be benchmark-large.

For example the coverage-large is there to verify that we don't run into similar OOM crash when coverage results of large project are collected. We used to store them in memory - #4603.

The reporters-large is there to verify that Vitest doesn't get completely stuck when there's a lot of reporting to do: #4710.

packages/vitest/src/node/types/benchmark.ts

AriPerkkio · 2024-09-21T07:27:54Z

packages/vitest/src/defaults.ts

@@ -24,6 +24,7 @@ export const benchmarkConfigDefaults: Required<
 exclude: defaultExclude,
 includeSource: [],
 reporters: ['default'],
+ includeSamples: false,


Is this not breaking change? Though benchmarking is experimental feature 🤔

Right, this is a breaking change. We could start from default true, but I'm hoping that people haven't really needed to have raw samples. For example, since my last PR #5398. benchmark outputJson already omits raw samples.

@sheremet-va have we previously introduced breaking changes for experimental features in minor releases, or also on patch releases?

All the time

Co-authored-by: Ari Perkkiö <ari.perkkio@gmail.com>

hi-ogawa

Would be great to add test case for this in https://github.com/vitest-tests/ and include it in Ecosystem CI. We already have some similar cases there. This one could be benchmark-large.

Sounds good! I'll prepare one.

hi-ogawa · 2024-09-21T07:38:05Z

packages/vitest/src/defaults.ts

@@ -24,6 +24,7 @@ export const benchmarkConfigDefaults: Required<
 exclude: defaultExclude,
 includeSource: [],
 reporters: ['default'],
+ includeSamples: false,


Right, this is a breaking change. We could start from default true, but I'm hoping that people haven't really needed to have raw samples. For example, since my last PR #5398. benchmark outputJson already omits raw samples.

…o fix-benchmark-clear-samples

vitest-ecosystem-ci · 2024-09-23T06:32:28Z

📝 Ran ecosystem CI: Open

suite	result
vitest-benchmark-large	✅ success

AriPerkkio

Looks good to me! I've verified that the vitest-tests/benchmark-large crashes without this change.

aleclarson · 2024-10-01T19:13:41Z

When can we expect a new release? It'd be good to clarify the release cycle here in the docs.

hi-ogawa · 2024-10-02T02:26:07Z

@aleclarson I'm not sure if we'll have a stable release cycle (we'll discuss about it), but we just integrated pkg pr new, so you can grab one from main if you want to test it early https://github.com/vitest-dev/vitest/runs/30935774010

@hi-ogawa

##### [v2.1.2](https://github.com/vitest-dev/vitest/releases/tag/v2.1.2) ##### 🐞 Bug Fixes - Move `Vitest.setServer` to post `configureServer` hook to enable import analysis for workspace config loading - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6584 [<samp>(e7f35)</samp>](vitest-dev/vitest@e7f35214) - **benchmark**: - Clear `BenchmarkResult.samples` array to reduce memory usage - by [@hi-ogawa](https://github.com/hi-ogawa) and [@AriPerkkio](https://github.com/AriPerkkio) in vitest-dev/vitest#6541 [<samp>(a6407)</samp>](vitest-dev/vitest@a6407afc) - **browser**: - Fix dynamic import inside worker - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6569 [<samp>(ea2d4)</samp>](vitest-dev/vitest@ea2d429b) - Fix browser mock factory event race condition - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6530 [<samp>(f131f)</samp>](vitest-dev/vitest@f131f93b) - Serve ui assets as static - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6564 [<samp>(adcda)</samp>](vitest-dev/vitest@adcdaee8) - Update solidjs testing library lib - by [@CamilleTeruel](https://github.com/CamilleTeruel) in vitest-dev/vitest#6548 [<samp>(91442)</samp>](vitest-dev/vitest@91442dfc) - Use `data:` protocol on preview provider file upload - by [@userquin](https://github.com/userquin) in vitest-dev/vitest#6501 [<samp>(e9821)</samp>](vitest-dev/vitest@e9821f70) - Fix base for client script - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6510 [<samp>(f9528)</samp>](vitest-dev/vitest@f952874e) - Throw an error if "@vitest/browser/context" is imported outside of the browser mode - by [@sheremet-va](https://github.com/sheremet-va) in vitest-dev/vitest#6570 [<samp>(383f1)</samp>](vitest-dev/vitest@383f1791) - **coverage**: - Remove empty coverage folder on test failure too - by [@AriPerkkio](https://github.com/AriPerkkio) in vitest-dev/vitest#6547 [<samp>(1371c)</samp>](vitest-dev/vitest@1371ca6a) - Include `*.astro` by default - by [@AriPerkkio](https://github.com/AriPerkkio) in vitest-dev/vitest#6565 [<samp>(f8ff7)</samp>](vitest-dev/vitest@f8ff76a9) - `cleanOnRerun: false` to invalidate previous results - by [@AriPerkkio](https://github.com/AriPerkkio) in vitest-dev/vitest#6592 [<samp>(88bde)</samp>](vitest-dev/vitest@88bde99c) - **expect**: - Fix `toBeDefined` with `expect.poll` - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6562 [<samp>(f7da6)</samp>](vitest-dev/vitest@f7da6199) - **runner**: - Mark tests as skipped when `beforeAll` failed - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6524 [<samp>(fb797)</samp>](vitest-dev/vitest@fb79792d) - Support fixture parsing of lowered async syntax - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6531 [<samp>(b553c)</samp>](vitest-dev/vitest@b553c7d6) - Fix fixture parsing of lowered async syntax for non arrow functions - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6575 [<samp>(3de00)</samp>](vitest-dev/vitest@3de00ab6) - Guard test hook callback - by [@sheremet-va](https://github.com/sheremet-va) in vitest-dev/vitest#6604 [<samp>(14971)</samp>](vitest-dev/vitest@1497134e) - Run `onTestFinished` and `onTestFailed` during `retry` and `repeats` - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6609 [<samp>(c5e29)</samp>](vitest-dev/vitest@c5e29098) - **ui**: - List tests on ui when `--standalone` - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6577 [<samp>(d0bf8)</samp>](vitest-dev/vitest@d0bf89d3) - **vite-node**: - Fix esm false-detection inside comment - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6506 [<samp>(91f85)</samp>](vitest-dev/vitest@91f85997) - **vitest**: - Install dependencies with the same version when prompted - by [@sheremet-va](https://github.com/sheremet-va) in vitest-dev/vitest#6611 [<samp>(ed8b7)</samp>](vitest-dev/vitest@ed8b7c08) - Make env.SSR consistent between different pools - by [@sheremet-va](https://github.com/sheremet-va) in vitest-dev/vitest#6616 [<samp>(8a8d3)</samp>](vitest-dev/vitest@8a8d3f03) - Don't start a websocket server if api is disabled - by [@sheremet-va](https://github.com/sheremet-va) in vitest-dev/vitest#6617 [<samp>(82140)</samp>](vitest-dev/vitest@821400b8) - **workspace**: - Fix glob pattern detection - by [@hi-ogawa](https://github.com/hi-ogawa) in vitest-dev/vitest#6502 [<samp>(7727c)</samp>](vitest-dev/vitest@7727ca87) - Ignore DS_Store by default - by [@sheremet-va](https://github.com/sheremet-va) in vitest-dev/vitest#6571 [<samp>(d2a86)</samp>](vitest-dev/vitest@d2a86ff5) ##### [View changes on GitHub](vitest-dev/vitest@v2.1.1...v2.1.2)

refactor(benchmark): move sampleCount and median to `BenchmarkRes…

637fe11

…ult`

fix: clear samples

938204b

hi-ogawa changed the title ~~fix(benchmark): deal with a huge BenchmarkResult.samples array~~ fix(benchmark): clear BenchmarkResult.samples array to reduce memory usage Sep 21, 2024

test: add test

4a3dfb9

hi-ogawa marked this pull request as ready for review September 21, 2024 06:04

feat: add benchmark.includeSampels option

5e39ed6

AriPerkkio reviewed Sep 21, 2024

View reviewed changes

test: tweak

f1aa090

AriPerkkio reviewed Sep 21, 2024

View reviewed changes

chore: typo

65dc21d

Co-authored-by: Ari Perkkiö <ari.perkkio@gmail.com>

hi-ogawa commented Sep 21, 2024

View reviewed changes

hi-ogawa added 2 commits September 21, 2024 17:05

fix: free samples earlier

97ec6fe

Merge remote-tracking branch 'origin/fix-benchmark-clear-samples' int…

cfbd774

…o fix-benchmark-clear-samples

hi-ogawa mentioned this pull request Sep 23, 2024

feat: add benchmark-large vitest-dev/vitest-ecosystem-ci#24

Merged

AriPerkkio approved these changes Sep 23, 2024

View reviewed changes

sheremet-va approved these changes Sep 25, 2024

View reviewed changes

sheremet-va merged commit a6407af into vitest-dev:main Sep 25, 2024
14 of 16 checks passed

hi-ogawa deleted the fix-benchmark-clear-samples branch September 25, 2024 08:51

D-Sketon mentioned this pull request Sep 27, 2024

chore(deps): bump vitest from 1.5.2/2.0.5 to 2.1.1, fix unit test failing in Vitest 2 toss/es-toolkit#608

Merged

MarlonPassos-git mentioned this pull request Sep 28, 2024

chore: comparative benchmarks with lodash radashi-org/radashi#258

Draft

3 tasks

renovate bot mentioned this pull request Oct 3, 2024

Update all non-major dependencies 2BAD/mire#3

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(benchmark): clear `BenchmarkResult.samples` array to reduce memory usage #6541

fix(benchmark): clear `BenchmarkResult.samples` array to reduce memory usage #6541

hi-ogawa commented Sep 21, 2024 •

edited

Loading

netlify bot commented Sep 21, 2024

AriPerkkio left a comment

AriPerkkio Sep 21, 2024

hi-ogawa Sep 21, 2024

AriPerkkio Sep 25, 2024

sheremet-va Sep 25, 2024

hi-ogawa left a comment

hi-ogawa Sep 21, 2024

vitest-ecosystem-ci bot commented Sep 23, 2024 •

edited

Loading

AriPerkkio left a comment

aleclarson commented Oct 1, 2024

hi-ogawa commented Oct 2, 2024

fix(benchmark): clear BenchmarkResult.samples array to reduce memory usage #6541

fix(benchmark): clear BenchmarkResult.samples array to reduce memory usage #6541

Conversation

hi-ogawa commented Sep 21, 2024 • edited Loading

Description

Please don't delete this checklist! Before submitting the PR, please make sure you do the following:

Tests

Documentation

Changesets

netlify bot commented Sep 21, 2024

✅ Deploy Preview for vitest-dev ready!

AriPerkkio left a comment

Choose a reason for hiding this comment

AriPerkkio Sep 21, 2024

Choose a reason for hiding this comment

hi-ogawa Sep 21, 2024

Choose a reason for hiding this comment

AriPerkkio Sep 25, 2024

Choose a reason for hiding this comment

sheremet-va Sep 25, 2024

Choose a reason for hiding this comment

hi-ogawa left a comment

Choose a reason for hiding this comment

hi-ogawa Sep 21, 2024

Choose a reason for hiding this comment

vitest-ecosystem-ci bot commented Sep 23, 2024 • edited Loading

AriPerkkio left a comment

Choose a reason for hiding this comment

aleclarson commented Oct 1, 2024

hi-ogawa commented Oct 2, 2024

fix(benchmark): clear `BenchmarkResult.samples` array to reduce memory usage #6541

fix(benchmark): clear `BenchmarkResult.samples` array to reduce memory usage #6541

hi-ogawa commented Sep 21, 2024 •

edited

Loading

vitest-ecosystem-ci bot commented Sep 23, 2024 •

edited

Loading