-
-
Notifications
You must be signed in to change notification settings - Fork 14
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
fix(perf): improve validations, recursions, logs and imports (#346)
* fix(perf): improve validations, recursions, logs and imports * refactor: use callback for legacy compatibility * ci: add benchmarks * ci(benchmarks): compare poku (npm) and locally * ci(benchmarks): fix job name * ci(benchmarks): remove static tolerance * ci: ignore external errors from coverage * ci: ignore external errors from coverage * docs: add benchmark instructions * docs: fix typo * docs: fix sections
- Loading branch information
1 parent
29add1c
commit 89bcac3
Showing
38 changed files
with
6,445 additions
and
125 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| /lib | ||
| /benchmark | ||
| /ci | ||
| /CHANGELOG.md | ||
| /website |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,5 @@ | ||
| website/** linguist-documentation | ||
| test/docker/** linguist-vendored | ||
| fixtures/** linguist-vendored | ||
| benchmark/** linguist-vendored | ||
| src/bin/** linguist-language=TypeScript |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,44 @@ | ||
| name: '🎖️ CI — Benchmark' | ||
|
|
||
| on: | ||
| pull_request: | ||
| workflow_dispatch: | ||
|
|
||
| jobs: | ||
| benchmark: | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 5 | ||
| strategy: | ||
| fail-fast: false | ||
| name: Compare | ||
| steps: | ||
| - name: ➕ Actions - Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: ➕ Actions - Setup NodeJS | ||
| uses: actions/setup-node@v4 | ||
| with: | ||
| node-version: '22.x' | ||
|
|
||
| - name: ➕ Cache dependencies | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.npm | ||
| key: npm-linux-${{ hashFiles('package-lock.json') }} | ||
| restore-keys: npm-linux- | ||
|
|
||
| - name: ➕ Cache dependencies (Benchmark) | ||
| uses: actions/cache@v4 | ||
| with: | ||
| path: ~/.npm | ||
| key: npm-benchmark-${{ hashFiles('benchmark/package-lock.json') }} | ||
| restore-keys: npm-benchmark- | ||
|
|
||
| - name: 📦 Installing Dependencies | ||
| run: npm ci | ||
|
|
||
| - name: 🚀 Building Poku | ||
| run: npm run build | ||
|
|
||
| - name: 🎖️ Rock it | ||
| run: npm run benchmark |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| node_modules |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,35 @@ | ||
| # Benchmark | ||
|
|
||
| The benchmark is performed by comparing a simple success and a failure test, each in a respective file and using parallel/concurrent runs. | ||
|
|
||
| > [!important] | ||
| > | ||
| > **Poku** doesn't intend to be the best, but to offer a balance between high performance, compatibility, lightness and ease of use. | ||
| --- | ||
|
|
||
| The testers to be compared are chosen based on the three most downloaded test runners according to the **npm** weekly statistics _(2024/06/09)_: | ||
|
|
||
| - [**Jest**](https://www.npmjs.com/package/jest): 23,549,369 | ||
| - [**Mocha**](https://www.npmjs.com/package/mocha) + [**Chai**](https://www.npmjs.com/package/chai): 8,053,244 + 11,294,912 | ||
| - [**Vitest**](https://www.npmjs.com/package/vitest): 4,840,171 | ||
|
|
||
| --- | ||
|
|
||
| **Poku** is continuously tested ([**CI**](https://github.com/wellwelwel/poku/blob/main/.github/workflows/ci_benchmark.yml)) to ensure the following expectations: | ||
|
|
||
| - [x] **~4x** faster than [**Jest**](https://github.com/jestjs/jest) (v29.7.0) | ||
| - [x] **~3x** faster than [**Vitest**](https://github.com/vitest-dev/vitest) (v1.6.0) | ||
| - [x] **~1x** faster than [**Mocha**](https://github.com/mochajs/mocha) (v10.4.0) + [**Chai**](https://github.com/chaijs/chai) (v5.1.1) | ||
|
|
||
| --- | ||
|
|
||
| ## Running | ||
|
|
||
| To run the benchmark tests, follow these steps: | ||
|
|
||
| ```sh | ||
| npm ci | ||
| npm run build | ||
| npm run benchmark | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| import { execSync } from 'node:child_process'; | ||
| import Benchmark from 'benchmark'; | ||
|
|
||
| const suite = new Benchmark.Suite(); | ||
| const results = new Map(); | ||
|
|
||
| const exec = (command) => { | ||
| try { | ||
| execSync(command, { stdio: 'ignore' }); | ||
| } catch {} | ||
| }; | ||
|
|
||
| const test = () => { | ||
| const pokuResult = results.get('Poku (Local)'); | ||
|
|
||
| const tolerancesPerTester = { | ||
| Jest: 4, | ||
| 'Mocha + Chai': 1, | ||
| Vitest: 3, | ||
| Poku: 0.8, | ||
| }; | ||
|
|
||
| for (const [name, result] of results) { | ||
| if (name === 'Poku (Local)') continue; | ||
|
|
||
| const expectedRatio = tolerancesPerTester[name]; | ||
| const actualRatio = pokuResult.opsPerSec / result.opsPerSec; | ||
| const isAtLeastExpected = actualRatio >= expectedRatio; | ||
|
|
||
| if (!isAtLeastExpected) { | ||
| console.error( | ||
| `Poku (Local) isn't approximately ${expectedRatio.toFixed(2)}x faster than ${name}.` | ||
| ); | ||
| process.exit(1); | ||
| } | ||
| } | ||
| }; | ||
|
|
||
| suite | ||
| .add('Jest ', () => { | ||
| exec( | ||
| 'node --experimental-vm-modules ./node_modules/jest/bin/jest.js ./test/jest' | ||
| ); | ||
| }) | ||
| .add('Mocha + Chai ', () => { | ||
| exec('./node_modules/mocha/bin/mocha.js --parallel ./test/mocha'); | ||
| }) | ||
| .add('Vitest ', () => { | ||
| exec('./node_modules/vitest/vitest.mjs run ./test/vitest'); | ||
| }) | ||
| .add('Poku ', () => { | ||
| exec('./node_modules/poku/lib/bin/index.js --parallel ./test/poku'); | ||
| }) | ||
| .add('Poku (Local) ', () => { | ||
| exec('../lib/bin/index.js --parallel ./test/poku'); | ||
| }) | ||
| .on('cycle', (event) => { | ||
| const name = event.target.name.trim(); | ||
| const result = { | ||
| opsPerSec: event.target.hz, | ||
| percentage: event.target.stats.rme, | ||
| }; | ||
|
|
||
| results.set(name, result); | ||
| console.log( | ||
| `${event.target.name} x ${result.opsPerSec.toFixed(2)} ops/sec — ±${result.percentage.toFixed(2)}% (${event.target.stats.sample.length} runs sampled)` | ||
| ); | ||
| }) | ||
| .on('complete', function () { | ||
| const fatest = String(this.filter('fastest').map('name')).trim(); | ||
|
|
||
| console.log(`\n🚀 Fastest is \x1b[1m${fatest}\x1b[0m\n`); | ||
|
|
||
| if (!/^Poku/.test(fatest)) process.exit(1); | ||
|
|
||
| test(); | ||
| }) | ||
| .run({ async: true }); |
Oops, something went wrong.