Issues with vitest bench

[sheremet-va]

Describe the bug

There are some issues with the new bench function:

• several bench tests are shared between single suite, even if they are inside different describes:

describe('sort 3', () => {
  bench('normal 1', () => {
    const x = [1, 5, 4, 2, 3, 6];
    x.sort((a, b) => {
      return a - b;
    });
  });
});

describe('sort 2', () => {
  bench('normal 2', () => {
    const x = [1, 5, 4, 2, 3, 6];
    x.sort((a, b) => {
      return a - b;
    });
  });
});

Result:

 ✓ test/basic.bench.ts (2) 557ms
   ✓ sort 3 (1) 556ms
     · normal 1  515.09 ops/sec ±4.38% (257595 samples) fastest
   ✓ sort 2 (1) 556ms
     · normal 2  514.57 ops/sec ±5.11% (257595 samples) fastest

Single bench gives different results:

bench('normal 1', () => {
  const x = [1, 5, 4, 2, 3, 6];
  x.sort((a, b) => {
    return a - b;
  });
});

Result:

 ✓ test/basic.bench.ts (1) 557ms
   · normal 1  1,117.25 ops/sec ±4.17% (558625 samples) fastest

• options don't have any effect on benchmark

  bench(
    'normal 1',
    () => {
      const x = [1, 5, 4, 2, 3, 6];
      x.sort((a, b) => {
        return a - b;
      });
    },
    { time: 10000000 }
  );

bench/describe also doesn't support some API:

• bench.only (describe.only)
• bench.todo (describe.todo)

Reproduction

https://stackblitz.com/edit/vitest-dev-vitest-w6dzu4?file=test%2Fbasic.bench.ts

System Info

stackblitz

Used Package Manager

npm

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that reports the same bug to avoid creating a duplicate.
• Check that this is a concrete bug. For Q&A open a GitHub Discussion or join our Discord Chat Server.
• The provided reproduction is a minimal reproducible example of the bug.

[FINDarkside]

Single bench gives different results:

I think the underlying benchmark lib is flawed, though it has added sequential option which might provide correct results. tinylibs/tinybench#18

[Aslemammad]

@FINDarkside With the new update, it works pretty well. Btw, the implementation here is a little bit different than what was general in Tinybench.

[thecodrr]

This is still an issue. For some reason vitest bench gives way different results than tinybench:

vitest bench:

     name                   hz      min      max     mean      p75      p99     p995     p999      rme  samples
   · fdir 6.0.0 async  10.2217  34.1455   153.45  97.8306   151.56   153.45   153.45   153.45  ±34.13%       10
   · glob async         5.6354   159.46   228.05   177.45   186.79   228.05   228.05   228.05   ±8.27%       10   slowest
   · fast-glob async   35.7399  18.9385  79.6491  27.9799  24.3642  79.6491  79.6491  79.6491  ±26.89%       18   fastest
   · tiny-glob async   10.6964  90.8450  96.7699  93.4898  95.6707  96.7699  96.7699  96.7699   ±2.90%       10

tinybench:

┌─────────┬────────────────────┬────────────────────┐
│ (index) │     Task Name      │     Ops/second     │
├─────────┼────────────────────┼────────────────────┤
│    0    │ 'fdir 6.0.0 async' │ 75.99658025680434  │
│    1    │    'glob async'    │ 5.382662058310003  │
│    2    │ 'fast-glob async'  │ 49.264947273669335 │
│    3    │ 'tiny-glob async'  │ 8.152110562298246  │
└─────────┴────────────────────┴────────────────────┘

I also tried running each benchmark separately in vitest bench and the results get more accurate so the only explanation could be that vitest is running the benchmarks in parallel.

[Aslemammad]

I also tried running each benchmark separately in vitest bench and the results get more accurate so the only explanation could be that vitest is running the benchmarks in parallel.

I have the same feeling, because the last time I checked it was running them in parallel! But in tinybench with latest updates, it won't do that.

[Aslemammad]

@thecodrr In case you are interested in sending a PR, let me know, otherwise, I'll work on this soon.

[thecodrr]

Found the culprit:

vitest/packages/vitest/src/runtime/runners/benchmark.ts

Lines 91 to 97 in 0f86ff9

	Promise.all(benchmarkGroup.map(async (benchmark) => {
	await benchmark.meta.task!.warmup()
	const { setTimeout } = getSafeTimers()
	return await new Promise<BenchTask>(resolve => setTimeout(async () => {
	resolve(await benchmark.meta.task!.run())
	}))
	})).then((tasks) => {