[Docs] Explain how to use sharding #4269
Comments
|
Have you done any performance profiling via
Are you now using
Are you running these commands in parallel, e.g.: vitest run --shard=1/4 & \
vitest run --shard=2/4 & \
vitest run --shard=3/4 & \
vitest run --shard=4/4 & \
wait # https://man7.org/linux/man-pages/man2/waitpid.2.html
This utilizes only Note that this all depends on your setup. There's no general rule that says "using more than x amount of threads makes no difference" - unless there is a performance bottleneck on main thread of course. |
|
@AriPerkkio sorry, I missed your reply. Don't hesitate to ping me :) I will do some profiling this week. No, we don't use Yeah, I understand that we can use more threads, but we have some things in parallel to vitest too: webpack, eslint and so on. And in our case all of them kinda aligned to same time and usually finish at same time. But thanks for the recommendation anyway! |
|
Finally, I found some time to profile. ConfigurationI decided to upgrade all dependencies and node. So what we have right now: npx envinfo --system --npmPackages {vitest,@vitest/*,vite,@vitejs/*} --binaries --browsers
System:
OS: Linux 5.10 Debian GNU/Linux 11 (bullseye) 11 (bullseye)
CPU: (32) x64 Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz
Memory: 59.26 GB / 61.79 GB
Container: Yes
Shell: 5.1.4 - /bin/bash
Binaries:
Node: 21.1.0 - /usr/local/bin/node
Yarn: 3.5.1 - /usr/local/bin/yarn
npm: 10.2.0 - /usr/local/bin/npm
npmPackages:
@vitejs/plugin-react: 4.1.0 => 4.1.0
@vitest/ui: 0.34.6 => 0.34.6
vitest: 1.0.0-beta.3 => 1.0.0-beta.3And also dropped some dependencies, switched to import { createRequire } from 'node:module';
import path from 'node:path'
import { defineConfig } from 'vitest/config';
import react from '@vitejs/plugin-react';
process.env.TZ = 'utc';
process.env.ENV = 'test';
process.env.IS_VITEST = 'true';
process.env.BABEL_ENV = 'test';
process.env.NODE_ENV = 'development';
process.env.DOTENV_CONFIG_PATH = './.env.dist';
const require = createRequire(import.meta.url);
const babelConfig = require('./babel.config');
export default defineConfig({
test: {
include: [
'{client,server,scripts,test_utils,build}/**/*.{vitest,test,spec}.{ts,tsx,js}',
],
pool: 'vmThreads',
poolOptions: {
vmThreads: {
useAtomics: false,
memoryLimit: '500MB',
}
},
globals: true,
setupFiles: './test_utils/vitest.setup.ts',
testTimeout: 10_000,
teardownTimeout: 5_000,
server: {
deps: {
inline: ['@aviasales/ui', '@aviasales/analytics'],
cacheDir: './cache/vitest/server-deps-cachedir'
},
},
cache: {
dir: '.cache/vitest/cache-dir',
},
poolMatchGlobs: [
[
'**/client/src/product/assisted/services/smartlook/__tests__/smartlook.vitest.ts',
'threads',
],
[
'**/client/src/product/assisted/services/init/__tests__/init.vitest.ts',
'threads',
],
],
},
resolve: {
alias: [
{
find: /.*\.(css|scss)$/,
replacement: 'identity-obj-proxy',
},
// server/tsconfig.json paths aliases
{
find: '&selene',
replacement: path.resolve(__dirname, 'server/src'),
},
{
find: '&platform',
replacement: path.resolve(__dirname, 'client/src/platform'),
},
{
find: '&widgets',
replacement: path.resolve(__dirname, 'client/src/widgets'),
},
{
find: '&components',
replacement: path.resolve(__dirname, 'client/src/components'),
},
{
find: '&internal',
replacement: path.resolve(__dirname, 'client/src/internal'),
},
{
find: '&services',
replacement: path.resolve(__dirname, 'client/src/services'),
},
{
find: '&product',
replacement: path.resolve(__dirname, 'client/src/product'),
},
{
find: '&test_utils',
replacement: path.resolve(__dirname, 'test_utils'),
},
],
},
cacheDir: '.cache/vite',
plugins: [
react({
babel: {
...babelConfig,
babelrc: false,
configFile: false,
},
}),
],
});Results:4 shards in parallel by 3 threads in each: 125.00s (transform 14.74s, setup 5.21s, collect 343.05s, tests 11.34s, environment 1.11s, prepare 11.36s)
119.05s (transform 54.71s, setup 5.13s, collect 319.42s, tests 22.10s, environment 791ms, prepare 10.76s)
114.76s (transform 24.02s, setup 4.77s, collect 301.13s, tests 17.99s, environment 905ms, prepare 10.48s)
125.38s (transform 23.66s, setup 5.68s, collect 307.52s, tests 44.41s, environment 891ms, prepare 10.24s)12 threads 24 threads I captured some profiles and mentioned that a lot of time was spend for garbage collection. I tried to increase --max-semi-space-size (see nodejs/node#47277) to 64mb and 128mb. It led to time decreasion spent on GC from 16s to 8s (64mb) and 6s (128mb). It didn't make any diffrenece for total execution time, but on node v18.12.0 it allowed me to run more threads. Without this option vitest with 24 threads failed some tests with I didn't get meaningful insights from flamegraphs. But it's obvious that a lot of time spent on file transformation. Is it possible that shards allow to parallel transformations and it leads to faster execution?
There are cpu profiles: |
Each shard is a separate process with its own module graph so yes, it's running in parallel without affecting each other much. By default, I would expect Vitest to have a limit after which increasing the number of cores doesn't make a lot of sense because the main process is just overloaded by requests from other threads. Looks like sharding alleviates this problem. |
|
So there is nothing we can do? If so, I think we should mention in the documentation that shards could speed up tests. I think it makes sense to create a separate section about performance on website. Also, it would be nice to have some API to concat report artifacts like JUnit, JSON, and HTML which is a separate issue indeed. If you don't mind about this idea I can create issue for it and send PR. |
From the perspective of Vitest, we provide all the tools already but don't document it enough.
I think there is either an open PR or discussion about this, but yes it should be fine to open a new issue. I think playwright uses
Yes, we should probably have a separate guide page about sharding. Currently, it's only mentioned in the config docs with the assumption that people already know what to do with it. |
7rulnik
changed the title
|
So I renamed this issue a bit. And yeah, there is the issue about reports concatenation #5125 |
|
Let's add documentation in #5736. |
Describe the bug
First of all, I am not sure it's a bug, feel free to convert to discussion or close it.
So we experienced performance problems with Vitest after migration from Jest. I am aware of eperimentalVmThreads existence, but we faced it before it was introduced. So I decided to play a bit with
VITEST_MIN_THREADSandVITEST_MAX_THREADS. For all results captured from CI, we are using Amazon EC2 instances. In this case machines with 32 CPUs.Our vitest config:
We run vitest like this:
Amount of tests:
Here are the results:
As we can see it doesn't make sense to increase the amount of threads to more than ~8.
And as I imagine it, the bottleneck is the main vilest process that launches tinypool, but of course, it's just my assumption.
I don't know if is there any possibility to enhance it.
Also, we work around it via shards: we launch 4 shards with 3 threads in each and it takes ~3.5 minutes.
Reproduction
I can't provide repro since it's our private repo, but I can provide performance snapshots or even invite you to our repo probably without signing NDA and so on…
System Info
System: OS: Linux 5.10 Debian GNU/Linux 11 (bullseye) 11 (bullseye) CPU: (32) x64 AMD EPYC 7R13 Processor Memory: 59.22 GB / 61.53 GB Container: Yes Shell: 5.1.4 - /bin/bash Binaries: Node: 18.12.0 - /usr/local/bin/node Yarn: 3.5.1 - /usr/local/bin/yarn npm: 8.19.2 - /usr/local/bin/npm npmPackages: vitest: 0.34.5 => 0.34.5Used Package Manager
yarn
Validations
The text was updated successfully, but these errors were encountered: