Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: replace fast-write-atomic with steno #284

Closed
3 tasks done
SgtPooki opened this issue Dec 14, 2023 · 2 comments · Fixed by #285
Closed
3 tasks done

feat: replace fast-write-atomic with steno #284

SgtPooki opened this issue Dec 14, 2023 · 2 comments · Fixed by #285
Assignees
@SgtPooki
Labels
kind/enhancement A net-new feature or improvement to an existing feature

Comments

@SgtPooki
Copy link
Member

SgtPooki commented Dec 14, 2023 •
edited
Loading

See https://github.com/SgtPooki/fast-write-atomic#uint8array

Task Name ops/sec Average Time (ns) Margin Samples
Uint8Array - write-file-atomic 101 9853620.467916131 ±1.15% 1000
Uint8Array - fast-write-atomic 106 9421993.666924536 ±1.30% 1000
Uint8Array - atomically 101 9857824.475903064 ±1.16% 1000
Uint8Array - steno 2,937 340472.6759530604 ±4.23% 1000
Uint8Array - steno (cached writers) 3,037 329200.16099512577 ±6.35% 1000
Uint8Array - @sgtpooki/steno-patched 2,940 340053.3969886601 ±9.02% 1000
Uint8Array - @sgtpooki/steno-patched (cached writers) 2,566 389624.0490563214 ±8.57% 1000

Tasks
Beta Give feedback

related: ipfs/helia#346 (comment)
@SgtPooki SgtPooki added the need/triage Needs initial labeling and prioritization label Dec 14, 2023
@SgtPooki SgtPooki self-assigned this Dec 14, 2023
This was referenced Dec 14, 2023
Merged
Merged
@SgtPooki SgtPooki linked a pull request Dec 14, 2023 that will close this issue
fix: replace fast-atomic-write with steno #285
Merged
@SgtPooki SgtPooki changed the title chore: replace fast-write-atomic with steno feat: replace fast-write-atomic with steno Apr 22, 2024
@achingbrain achingbrain added kind/enhancement A net-new feature or improvement to an existing feature and removed need/triage Needs initial labeling and prioritization labels Aug 1, 2024
@ipfs ipfs deleted a comment from welcome bot Aug 1, 2024
achingbrain pushed a commit that referenced this issue Aug 9, 2024
@SgtPooki
fix: replace fast-atomic-write with steno (#285)
c6df8dd 
fixes #284

fast-write-atomic hasn't been updated in 5 years, is CJS, and is slower than steno (updated 2 months ago).

Benchmarks for various content-types & libraries (though we only use Uint8Arrays) can be found at https://github.com/SgtPooki/fast-write-atomic#benchmarks

However, there may be further room for improvement by moving to [fs.createWriteStream](https://nodejs.org/api/fs.html#fscreatewritestreampath-options)


```
╰─ ✔ ❯ hyperfine --parameter-list branch 284-chore-replace-fast-write-atomic-with-steno,main --setup "git switch {branch} && npm run reset && npm i && npm run build" --runs 20  -w 1 "npm run test:node"
Benchmark 1: npm run test:node (branch = 284-chore-replace-fast-write-atomic-with-steno)
  Time (mean ± σ):     27.212 s ±  0.832 s    [User: 34.810 s, System: 6.051 s]
  Range (min … max):   25.927 s … 29.324 s    20 runs

Benchmark 2: npm run test:node (branch = main)
  Time (mean ± σ):     42.971 s ±  0.637 s    [User: 35.297 s, System: 7.534 s]
  Range (min … max):   42.178 s … 44.796 s    20 runs

Summary
  npm run test:node (branch = 284-chore-replace-fast-write-atomic-with-steno) ran
    1.58 ± 0.05 times faster than npm run test:node (branch = main)
```

---

### Updated benchmarks of `npm run test` as of 2024-04-19

```
╭─    ~/code/work/protocol.ai/ipfs/js-stores    main ?1 
╰─ ✔ ❯ hyperfine --parameter-list branch main,test/not-same-event-loop-concurrency,284-chore-replace-fast-write-atomic-with-steno --setup "git switch {branch} && npm run reset && npm i && npm run build && cd packages/datastore-fs" "npm run test"
Benchmark 1: npm run test (branch = main)
  Time (mean ± σ):     99.415 s ±  2.918 s    [User: 69.659 s, System: 23.361 s]
  Range (min … max):   96.134 s … 105.200 s    10 runs

Benchmark 2: npm run test (branch = test/not-same-event-loop-concurrency)
  Time (mean ± σ):     103.456 s ±  3.186 s    [User: 74.442 s, System: 25.261 s]
  Range (min … max):   98.813 s … 108.429 s    10 runs

Benchmark 3: npm run test (branch = 284-chore-replace-fast-write-atomic-with-steno)
  Time (mean ± σ):     80.308 s ±  2.107 s    [User: 74.331 s, System: 22.228 s]
  Range (min … max):   78.219 s … 84.277 s    10 runs

Summary
  npm run test (branch = 284-chore-replace-fast-write-atomic-with-steno) ran
    1.24 ± 0.05 times faster than npm run test (branch = main)
    1.29 ± 0.05 times faster than npm run test (branch = test/not-same-event-loop-concurrency)                                                                                        [49m1.944s]

╭─    ~/code/work/protocol.ai/ipfs/js-stores    284-chore-re…c-with-steno ?1 
╰─ ✔ ❯ hyperfine --parameter-list branch main,test/not-same-event-loop-concurrency,284-chore-replace-fast-write-atomic-with-steno --setup "git switch {branch} && npm run reset && npm i && npm run build && cd packages/blockstore-fs" "npm run test"
Benchmark 1: npm run test (branch = main)
  Time (mean ± σ):     98.840 s ±  2.612 s    [User: 68.486 s, System: 22.585 s]
  Range (min … max):   97.005 s … 104.396 s    10 runs

Benchmark 2: npm run test (branch = test/not-same-event-loop-concurrency)
  Time (mean ± σ):     105.307 s ±  2.335 s    [User: 72.442 s, System: 24.766 s]
  Range (min … max):   101.167 s … 109.007 s    10 runs

Benchmark 3: npm run test (branch = 284-chore-replace-fast-write-atomic-with-steno)
  Time (mean ± σ):     77.012 s ±  1.829 s    [User: 74.442 s, System: 21.938 s]
  Range (min … max):   75.258 s … 80.825 s    10 runs

Summary
  npm run test (branch = 284-chore-replace-fast-write-atomic-with-steno) ran
    1.28 ± 0.05 times faster than npm run test (branch = main)
    1.37 ± 0.04 times faster than npm run test (branch = test/not-same-event-loop-concurrency)
```
@achingbrain
Copy link
Member

Fixed by #285

@achingbrain
Copy link
Member

Ah, no, #285 merged the steno impl into the PR with the tests

@achingbrain achingbrain reopened this Sep 13, 2024
github-actions bot pushed a commit that referenced this issue Sep 17, 2024
@semantic-release-bot
chore(release): 2.0.2 [skip ci]
1ad66fa 
## [blockstore-fs-v2.0.2](blockstore-fs-2.0.1...blockstore-fs-2.0.2) (2024-09-17)

### Tests

* add separate-thread concurrency test ([#305](#305)) ([5e3114e](5e3114e)), closes [#285](#285) [#284](#284)
github-actions bot pushed a commit that referenced this issue Sep 17, 2024
@semantic-release-bot
chore(release): 10.0.2 [skip ci]
06b7f79 
## [datastore-fs-v10.0.2](datastore-fs-10.0.1...datastore-fs-10.0.2) (2024-09-17)

### Tests

* add separate-thread concurrency test ([#305](#305)) ([5e3114e](5e3114e)), closes [#285](#285) [#284](#284)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@SgtPooki SgtPooki

Labels
kind/enhancement A net-new feature or improvement to an existing feature
Projects
No open projects
IPFS Shipyard Team
Status: 🏃‍♀️ In Progress
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: replace fast-atomic-write with steno ipfs/js-stores
2 participants
@achingbrain @SgtPooki