Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fs: make sure to write entire buffer #49211

Closed
wants to merge 0 commits into from
Closed

fs: make sure to write entire buffer #49211

wants to merge 0 commits into from

Conversation

atlowChemi
Copy link
Member

fs.write(v) is not guaranteed to write everything in a single
call. Make sure we don't assume so.

Refs: #42434 (comment)

@nodejs-github-bot nodejs-github-bot added fs Issues and PRs related to the fs subsystem / file system. needs-ci PRs that need a full CI run. labels Aug 17, 2023
anonrig
anonrig reviewed Aug 17, 2023
View reviewed changes
lib/internal/fs/streams.js Outdated Show resolved Hide resolved
lib/internal/fs/streams.js Outdated Show resolved Hide resolved
@mscdex mscdex added the needs-benchmark-ci PR that need a benchmark CI run. label Aug 17, 2023
@atlowChemi atlowChemi added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 20, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 20, 2023
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/53460/

@atlowChemi
Copy link
Member Author

benchmark output:

                                                                                              confidence improvement accuracy (*)    (**)   (***)
fs/write-stream-throughput.js size=1024 encodingType='asc' dur=5                                             -0.97 %       ±2.40%  ±3.19%  ±4.15%
fs/write-stream-throughput.js size=1024 encodingType='buf' dur=5                                              0.81 %       ±1.53%  ±2.03%  ±2.65%
fs/write-stream-throughput.js size=1024 encodingType='utf' dur=5                                              1.61 %       ±1.75%  ±2.33%  ±3.06%
fs/write-stream-throughput.js size=1048576 encodingType='asc' dur=5                                           0.09 %       ±2.32%  ±3.09%  ±4.02%
fs/write-stream-throughput.js size=1048576 encodingType='buf' dur=5                                          -0.38 %       ±0.60%  ±0.80%  ±1.03%
fs/write-stream-throughput.js size=1048576 encodingType='utf' dur=5                                          -0.38 %       ±1.50%  ±1.99%  ±2.60%
fs/write-stream-throughput.js size=2 encodingType='asc' dur=5                                                -0.31 %       ±2.03%  ±2.71%  ±3.53%
fs/write-stream-throughput.js size=2 encodingType='buf' dur=5                                                -0.53 %       ±1.74%  ±2.32%  ±3.02%
fs/write-stream-throughput.js size=2 encodingType='utf' dur=5                                                -0.13 %       ±2.22%  ±2.96%  ±3.85%
fs/write-stream-throughput.js size=65535 encodingType='asc' dur=5                                     **     -1.64 %       ±1.05%  ±1.40%  ±1.85%
fs/write-stream-throughput.js size=65535 encodingType='buf' dur=5                                             0.09 %       ±1.02%  ±1.37%  ±1.80%
fs/write-stream-throughput.js size=65535 encodingType='utf' dur=5                                             0.06 %       ±1.18%  ±1.57%  ±2.04%


Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 142 comparisons, you can thus
expect the following amount of false-positive results:
  7.10 false positives, when considering a   5% risk acceptance (*, **, ***),
  1.42 false positives, when considering a   1% risk acceptance (**, ***),
  0.14 false positives, when considering a 0.1% risk acceptance (***)

ronag
ronag reviewed Aug 21, 2023
View reviewed changes
Copy link
Member

@ronag ronag left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Existing tests pass and it's quite tricky make tests for this. You probably need to pass a custom "fs" implementation into the stream which will return EAGAIN.

@atlowChemi atlowChemi added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 22, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 22, 2023
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/53493/

@atlowChemi atlowChemi added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 22, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 22, 2023
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/53497/

@github-actions github-actions bot mentioned this pull request Aug 23, 2023
Open
27 tasks
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/53504/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/53505/

ronag
ronag reviewed Aug 23, 2023
View reviewed changes
lib/internal/fs/streams.js Outdated Show resolved Hide resolved
@atlowChemi atlowChemi force-pushed the write-all branch 2 times, most recently from 2c9b7e0 to 20d51d2 Compare August 23, 2023 13:44
@atlowChemi atlowChemi added the request-ci Add this label to start a Jenkins CI on a PR. label Aug 23, 2023
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Aug 23, 2023
@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/53521/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/53530/

@nodejs-github-bot
Copy link
Collaborator

CI: https://ci.nodejs.org/job/node-test-pull-request/53534/

@github-actions github-actions bot mentioned this pull request Aug 24, 2023
Open
27 tasks
@atlowChemi atlowChemi added author ready PRs that have at least one approval, no pending requests for changes, and a CI started. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. labels Aug 24, 2023
atlowChemi added a commit to atlowChemi/node that referenced this pull request Aug 24, 2023
@ronag @atlowChemi
fs: make sure to write entire buffer
feb5b0f 
fs.write(v) is not guaranteed to write everything in a single
call. Make sure we don't assume so.

PR-URL: nodejs#49211
Co-authored-by: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
@atlowChemi
Copy link
Member Author

Landed in feb5b0f

@atlowChemi atlowChemi closed this Aug 24, 2023
@atlowChemi atlowChemi deleted the write-all branch August 24, 2023 05:36
This was referenced Aug 25, 2023
Open
Open
Open
Open
Open
UlisesGascon pushed a commit that referenced this pull request Sep 10, 2023
@ronag @atlowChemi @UlisesGascon
fs: make sure to write entire buffer
88ba79b 
fs.write(v) is not guaranteed to write everything in a single
call. Make sure we don't assume so.

PR-URL: #49211
Co-authored-by: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
@UlisesGascon UlisesGascon mentioned this pull request Sep 10, 2023
Merged
@aduh95 aduh95 mentioned this pull request Sep 20, 2023
Closed
richardlau pushed a commit that referenced this pull request Sep 28, 2023
@ronag @atlowChemi @richardlau
fs: make sure to write entire buffer
7a3e1ff 
fs.write(v) is not guaranteed to write everything in a single
call. Make sure we don't assume so.

PR-URL: #49211
Co-authored-by: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
This was referenced Oct 6, 2023
Closed
Merged
alexfernandez pushed a commit to alexfernandez/node that referenced this pull request Nov 1, 2023
@ronag @atlowChemi @alexfernandez
fs: make sure to write entire buffer
febb608 
fs.write(v) is not guaranteed to write everything in a single
call. Make sure we don't assume so.

PR-URL: nodejs#49211
Co-authored-by: Chemi Atlow <chemi@atlow.co.il>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Reviewed-By: Robert Nagy <ronagy@icloud.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@benjamingr benjamingr benjamingr approved these changes

@ronag ronag ronag approved these changes

@anonrig anonrig Awaiting requested review from anonrig

Assignees
No one assigned
Labels
author ready PRs that have at least one approval, no pending requests for changes, and a CI started. commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. fs Issues and PRs related to the fs subsystem / file system. needs-benchmark-ci PR that need a benchmark CI run. needs-ci PRs that need a full CI run.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@atlowChemi @nodejs-github-bot @benjamingr @anonrig @ronag @mscdex