Skip to content

Conversation

pipobscure
Copy link
Contributor

@pipobscure pipobscure commented May 28, 2025

This addresses a bug in fs.watch when used as an AsyncIterator.

The issue is that when consuming the AsyncIteractor returned by fs.watch it yields a value. When using that value and in turn awaiting an asynchrounous action any events happening in the meantime will go missing. The reason is that between exiting the watch function by yielding and reentering it through the next round, the promise inside watch is already resolved. So any events generated will be duplicate resolutions of that promise and therefore ignored.

for await (let found of fs.watch('my dir')) {
  console.log(found);
  await sleep(10000); // any events happening during these 10seconds will  be missed;
}

More reaslistically than this minimal example is when the found files are actually read via await fs.readFile. Then there will be a lag. If the file events happening are happening close together, then the second one will be missed.

To fix this issue I added a queue to the watch function that new file events get pushed onto. The promise is no longer resolved with a value, but is simply the gate gating whether or not there are any events in the queue. The iterator awaits the promise and then yields the items from the queue so long as there are any. When the queue is empty and the watch is still running, then a new promise is created and awaited upon. This whould eliminate the problem entirely and one can now go asynchronous in the loop as long as one wants without missing events.

Verified that the added test fails with v24.0.0 and passes after the fix.

Based on feedback the queuing was made configurable with the maxQueue (default 2048) option determining the maximum size of the queue and the overflow option deciding to either ignore the issue or throw an Error (default: 'ignore'). To effectively get back the previous behavior one would have to pass { maxQueue: 1, overflow: 'ignore' }.

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or

(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or

(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.

(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@nodejs-github-bot nodejs-github-bot added fs Issues and PRs related to the fs subsystem / file system. needs-ci PRs that need a full CI run. labels May 28, 2025
@pipobscure pipobscure marked this pull request as ready for review May 28, 2025 22:49
Copy link

codecov bot commented May 28, 2025

Codecov Report

Attention: Patch coverage is 72.50000% with 11 lines in your changes missing coverage. Please review.

Project coverage is 90.15%. Comparing base (40d8983) to head (218d461).
Report is 18 commits behind head on main.

Files with missing lines Patch % Lines
lib/internal/fs/watchers.js 71.79% 11 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #58490      +/-   ##
==========================================
- Coverage   90.16%   90.15%   -0.01%     
==========================================
  Files         636      636              
  Lines      187891   187923      +32     
  Branches    36884    36882       -2     
==========================================
+ Hits       169408   169429      +21     
- Misses      11246    11247       +1     
- Partials     7237     7247      +10     
Files with missing lines Coverage Δ
lib/internal/errors.js 97.48% <100.00%> (+<0.01%) ⬆️
lib/internal/fs/watchers.js 85.16% <71.79%> (-0.63%) ⬇️

... and 30 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pipobscure pipobscure marked this pull request as draft May 29, 2025 10:33
@pipobscure pipobscure force-pushed the fsbug branch 2 times, most recently from 63d31e7 to 3512480 Compare May 29, 2025 11:23
@pipobscure pipobscure marked this pull request as ready for review May 29, 2025 11:23
buhski

This comment was marked as spam.

@pipobscure pipobscure force-pushed the fsbug branch 8 times, most recently from 16c2f33 to d5d6d5d Compare June 4, 2025 16:08
@joyeecheung
Copy link
Member

This almost LGTM % the suggestions. Though I'd appreciate if others from @nodejs/fs or maybe @jasnell @benjamingr can take a look and see if the new options look good.

Copy link
Contributor

@Ethan-Arrowood Ethan-Arrowood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other than the option values, this LGTM

Copy link
Member

@joyeecheung joyeecheung left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % missing updates to the docs

@pipobscure
Copy link
Contributor Author

Thanks for all the help, guidance & input @joyeecheung & @Ethan-Arrowood ❤️

@joyeecheung joyeecheung added the request-ci Add this label to start a Jenkins CI on a PR. label Jun 6, 2025
@joyeecheung joyeecheung added request-ci Add this label to start a Jenkins CI on a PR. and removed commit-queue-squash Add this label to instruct the Commit Queue to squash all the PR commits into the first one. labels Jun 9, 2025
@github-actions github-actions bot removed the request-ci Add this label to start a Jenkins CI on a PR. label Jun 9, 2025
@nodejs-github-bot
Copy link
Collaborator

@nodejs-github-bot
Copy link
Collaborator

@joyeecheung joyeecheung added the commit-queue Add this label to land a pull request using GitHub Actions. label Jun 10, 2025
@nodejs-github-bot nodejs-github-bot removed the commit-queue Add this label to land a pull request using GitHub Actions. label Jun 10, 2025
@nodejs-github-bot nodejs-github-bot merged commit 5f7dbf4 into nodejs:main Jun 10, 2025
60 checks passed
@nodejs-github-bot
Copy link
Collaborator

Landed in 5f7dbf4

seriousme pushed a commit to seriousme/node that referenced this pull request Jun 10, 2025
PR-URL: nodejs#58490
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Ethan Arrowood <ethan@arrowood.dev>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
targos pushed a commit that referenced this pull request Jun 16, 2025
PR-URL: #58490
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Ethan Arrowood <ethan@arrowood.dev>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
nodejs-github-bot added a commit that referenced this pull request Jun 16, 2025
Notable changes:

doc:
  * add islandryu to collaborators (Shima Ryuhei) #58714
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490
module:
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #58643
test_runner:
  * (SEMVER-MINOR) support object property mocking (Idan Goshen) #58438
url:
  * (SEMVER-MINOR) add fileURLToPathBuffer API (James M Snell) #58700

PR-URL: #58727
nodejs-github-bot added a commit that referenced this pull request Jun 23, 2025
Notable changes:

doc:
  * add islandryu to collaborators (Shima Ryuhei) #58714
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490
module:
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #58643
test_runner:
  * (SEMVER-MINOR) support object property mocking (Idan Goshen) #58438
url:
  * (SEMVER-MINOR) add fileURLToPathBuffer API (James M Snell) #58700

PR-URL: #58804
nodejs-github-bot added a commit that referenced this pull request Jun 24, 2025
Notable changes:

doc:
  * add islandryu to collaborators (Shima Ryuhei) #58714
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490
module:
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #58643
test:
  * fix test-timeout-flag after revert of auto subtest wait (Pietro Marchini) #58282
test_runner:
  * Revert "test_runner: remove promises returned by t.test() (Romain Menke) #58282
  * Revert "test_runner: remove promises returned by test() (Romain Menke) #58282
  * Revert "test_runner: automatically wait for subtests to finish (Romain Menke) #58282
  * (SEMVER-MINOR) support object property mocking (Idan Goshen) #58438
url:
  * (SEMVER-MINOR) add fileURLToPathBuffer API (James M Snell) #58700

PR-URL: #58813
RafaelGSS added a commit that referenced this pull request Jun 24, 2025
Notable changes:

doc:
  * add islandryu to collaborators (Shima Ryuhei) #58714
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490
module:
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #58643
test:
  * fix test-timeout-flag after revert of auto subtest wait (Pietro Marchini) #58282
test_runner:
  * (SEMVER-MINOR) support object property mocking (Idan Goshen) #58438
url:
  * (SEMVER-MINOR) add fileURLToPathBuffer API (James M Snell) #58700

PR-URL: #58813
RafaelGSS added a commit that referenced this pull request Jun 24, 2025
Notable changes:

doc:
  * add islandryu to collaborators (Shima Ryuhei) #58714
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490
module:
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #58643
test:
  * fix test-timeout-flag after revert of auto subtest wait (Pietro Marchini) #58282
test_runner:
  * (SEMVER-MINOR) support object property mocking (Idan Goshen) #58438
url:
  * (SEMVER-MINOR) add fileURLToPathBuffer API (James M Snell) #58700

PR-URL: #58813
aduh95 pushed a commit that referenced this pull request Jul 21, 2025
PR-URL: #58490
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Ethan Arrowood <ethan@arrowood.dev>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
aduh95 pushed a commit that referenced this pull request Jul 24, 2025
PR-URL: #58490
Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com>
Reviewed-By: Ethan Arrowood <ethan@arrowood.dev>
Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
nodejs-github-bot added a commit that referenced this pull request Jul 28, 2025
Notable changes:

deps:
  * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) #56350
  * (SEMVER-MINOR) update amaro to 1.0.0 (Node.js GitHub Bot) #56350
  * (SEMVER-MINOR) update amaro to 0.5.3 (Node.js GitHub Bot) #56350
  * (SEMVER-MINOR) update amaro to 0.5.2 (Node.js GitHub Bot) #56350
  * (SEMVER-MINOR) update amaro to 0.5.1 (Marco Ippolito) #56350
  * (SEMVER-MINOR) update amaro to 0.5.0 (nodejs-github-bot) #56350
doc:
  * (SEMVER-MINOR) add all watch-mode related flags to node.1 (Dario Piotrowicz) #58719
  * add islandryu to collaborators (Shima Ryuhei) #58714
  * (SEMVER-MINOR) add history entries to `--input-type` section (Antoine du Hamel) #56350
esm:
  * (SEMVER-MINOR) implement import.meta.main (Joe) #57804
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490
module:
  * (SEMVER-MINOR) improve typescript error message format (Marco Ippolito) #56350
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #56350
  * (SEMVER-MINOR) refactor commonjs typescript loader (Marco Ippolito) #56350
  * (SEMVER-MINOR) unflag --experimental-strip-types (Marco Ippolito) #56350
permission:
  * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) #58853
sqlite:
  * (SEMVER-MINOR) add support for readBigInts option in db connection level (Miguel Marcondes Filho) #58697
src,permission:
  * (SEMVER-MINOR) add support to permission.has(addon) (Rafael Gonzaga) #58951
test:
  * (SEMVER-MINOR) add test for async disposable worker thread (James M Snell) #58385
url:
  * (SEMVER-MINOR) add fileURLToPathBuffer API (James M Snell) #58700
watch:
  * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) #58719
worker:
  * (SEMVER-MINOR) make Worker async disposable (James M Snell) #58385

PR-URL: #59256
aduh95 added a commit that referenced this pull request Jul 28, 2025
Notable changes:

deps:
  * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) #56350
doc:
  * add islandryu to collaborators (Shima Ryuhei) #58714
esm:
  * (SEMVER-MINOR) implement `import.meta.main` (Joe) #57804
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490
module:
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #56350
  * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) #56350
permission:
  * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) #58853
sqlite:
  * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) #58697
src,permission:
  * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) #58951
url:
  * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) #58700
watch:
  * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) #58719
worker:
  * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) #58385

PR-URL: #59256
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
aduh95 added a commit that referenced this pull request Jul 31, 2025
Notable changes:

deps:
  * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) #56350
doc:
  * add islandryu to collaborators (Shima Ryuhei) #58714
esm:
  * (SEMVER-MINOR) implement `import.meta.main` (Joe) #57804
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490
module:
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #56350
  * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) #56350
permission:
  * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) #58853
sqlite:
  * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) #58697
src,permission:
  * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) #58951
url:
  * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) #58700
watch:
  * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) #58719
worker:
  * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) #58385

PR-URL: #59256
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
aduh95 added a commit that referenced this pull request Jul 31, 2025
Notable changes:

deps:
  * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) #56350
doc:
  * add islandryu to collaborators (Shima Ryuhei) #58714
esm:
  * (SEMVER-MINOR) implement `import.meta.main` (Joe) #57804
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490
module:
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #56350
  * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) #56350
permission:
  * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) #58853
sqlite:
  * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) #58697
src,permission:
  * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) #58951
url:
  * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) #58700
watch:
  * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) #58719
worker:
  * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) #58385

PR-URL: #59256
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
meteorqz6 pushed a commit to meteorqz6/node that referenced this pull request Aug 2, 2025
Notable changes:

deps:
  * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) nodejs#56350
doc:
  * add islandryu to collaborators (Shima Ryuhei) nodejs#58714
esm:
  * (SEMVER-MINOR) implement `import.meta.main` (Joe) nodejs#57804
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) nodejs#58490
module:
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) nodejs#56350
  * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) nodejs#56350
permission:
  * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) nodejs#58853
sqlite:
  * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) nodejs#58697
src,permission:
  * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) nodejs#58951
url:
  * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) nodejs#58700
watch:
  * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) nodejs#58719
worker:
  * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) nodejs#58385

PR-URL: nodejs#59256
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
panva pushed a commit to panva/node that referenced this pull request Aug 7, 2025
Notable changes:

deps:
  * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) nodejs#56350
doc:
  * add islandryu to collaborators (Shima Ryuhei) nodejs#58714
esm:
  * (SEMVER-MINOR) implement `import.meta.main` (Joe) nodejs#57804
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) nodejs#58490
module:
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) nodejs#56350
  * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) nodejs#56350
permission:
  * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) nodejs#58853
sqlite:
  * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) nodejs#58697
src,permission:
  * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) nodejs#58951
url:
  * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) nodejs#58700
watch:
  * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) nodejs#58719
worker:
  * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) nodejs#58385

PR-URL: nodejs#59256
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
mete0rfish pushed a commit to mete0rfish/node-contribute that referenced this pull request Aug 9, 2025
Notable changes:

deps:
  * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) nodejs#56350
doc:
  * add islandryu to collaborators (Shima Ryuhei) nodejs#58714
esm:
  * (SEMVER-MINOR) implement `import.meta.main` (Joe) nodejs#57804
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) nodejs#58490
module:
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) nodejs#56350
  * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) nodejs#56350
permission:
  * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) nodejs#58853
sqlite:
  * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) nodejs#58697
src,permission:
  * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) nodejs#58951
url:
  * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) nodejs#58700
watch:
  * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) nodejs#58719
worker:
  * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) nodejs#58385

PR-URL: nodejs#59256
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
panva pushed a commit to panva/node that referenced this pull request Aug 9, 2025
Notable changes:

deps:
  * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) nodejs#56350
doc:
  * add islandryu to collaborators (Shima Ryuhei) nodejs#58714
esm:
  * (SEMVER-MINOR) implement `import.meta.main` (Joe) nodejs#57804
fs:
  * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) nodejs#58490
module:
  * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) nodejs#56350
  * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) nodejs#56350
permission:
  * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) nodejs#58853
sqlite:
  * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) nodejs#58697
src,permission:
  * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) nodejs#58951
url:
  * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) nodejs#58700
watch:
  * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) nodejs#58719
worker:
  * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) nodejs#58385

PR-URL: nodejs#59256
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
fs Issues and PRs related to the fs subsystem / file system. needs-ci PRs that need a full CI run. semver-minor PRs that contain new features and should be released in the next minor version.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants