-
-
Notifications
You must be signed in to change notification settings - Fork 33.4k
fs: allow correct handling of burst in fs-events with AsyncIterator #58490
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #58490 +/- ##
==========================================
- Coverage 90.16% 90.15% -0.01%
==========================================
Files 636 636
Lines 187891 187923 +32
Branches 36884 36882 -2
==========================================
+ Hits 169408 169429 +21
- Misses 11246 11247 +1
- Partials 7237 7247 +10
🚀 New features to boost your workflow:
|
63d31e7
to
3512480
Compare
16c2f33
to
d5d6d5d
Compare
This almost LGTM % the suggestions. Though I'd appreciate if others from @nodejs/fs or maybe @jasnell @benjamingr can take a look and see if the new options look good. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Other than the option values, this LGTM
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM % missing updates to the docs
Thanks for all the help, guidance & input @joyeecheung & @Ethan-Arrowood ❤️ |
Landed in 5f7dbf4 |
PR-URL: nodejs#58490 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Ethan Arrowood <ethan@arrowood.dev> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
PR-URL: #58490 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Ethan Arrowood <ethan@arrowood.dev> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Notable changes: doc: * add islandryu to collaborators (Shima Ryuhei) #58714 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490 module: * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #58643 test_runner: * (SEMVER-MINOR) support object property mocking (Idan Goshen) #58438 url: * (SEMVER-MINOR) add fileURLToPathBuffer API (James M Snell) #58700 PR-URL: #58727
Notable changes: doc: * add islandryu to collaborators (Shima Ryuhei) #58714 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490 module: * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #58643 test_runner: * (SEMVER-MINOR) support object property mocking (Idan Goshen) #58438 url: * (SEMVER-MINOR) add fileURLToPathBuffer API (James M Snell) #58700 PR-URL: #58804
Notable changes: doc: * add islandryu to collaborators (Shima Ryuhei) #58714 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490 module: * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #58643 test: * fix test-timeout-flag after revert of auto subtest wait (Pietro Marchini) #58282 test_runner: * Revert "test_runner: remove promises returned by t.test() (Romain Menke) #58282 * Revert "test_runner: remove promises returned by test() (Romain Menke) #58282 * Revert "test_runner: automatically wait for subtests to finish (Romain Menke) #58282 * (SEMVER-MINOR) support object property mocking (Idan Goshen) #58438 url: * (SEMVER-MINOR) add fileURLToPathBuffer API (James M Snell) #58700 PR-URL: #58813
Notable changes: doc: * add islandryu to collaborators (Shima Ryuhei) #58714 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490 module: * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #58643 test: * fix test-timeout-flag after revert of auto subtest wait (Pietro Marchini) #58282 test_runner: * (SEMVER-MINOR) support object property mocking (Idan Goshen) #58438 url: * (SEMVER-MINOR) add fileURLToPathBuffer API (James M Snell) #58700 PR-URL: #58813
Notable changes: doc: * add islandryu to collaborators (Shima Ryuhei) #58714 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490 module: * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #58643 test: * fix test-timeout-flag after revert of auto subtest wait (Pietro Marchini) #58282 test_runner: * (SEMVER-MINOR) support object property mocking (Idan Goshen) #58438 url: * (SEMVER-MINOR) add fileURLToPathBuffer API (James M Snell) #58700 PR-URL: #58813
PR-URL: #58490 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Ethan Arrowood <ethan@arrowood.dev> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
PR-URL: #58490 Reviewed-By: Joyee Cheung <joyeec9h3@gmail.com> Reviewed-By: Ethan Arrowood <ethan@arrowood.dev> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com>
Notable changes: deps: * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) #56350 * (SEMVER-MINOR) update amaro to 1.0.0 (Node.js GitHub Bot) #56350 * (SEMVER-MINOR) update amaro to 0.5.3 (Node.js GitHub Bot) #56350 * (SEMVER-MINOR) update amaro to 0.5.2 (Node.js GitHub Bot) #56350 * (SEMVER-MINOR) update amaro to 0.5.1 (Marco Ippolito) #56350 * (SEMVER-MINOR) update amaro to 0.5.0 (nodejs-github-bot) #56350 doc: * (SEMVER-MINOR) add all watch-mode related flags to node.1 (Dario Piotrowicz) #58719 * add islandryu to collaborators (Shima Ryuhei) #58714 * (SEMVER-MINOR) add history entries to `--input-type` section (Antoine du Hamel) #56350 esm: * (SEMVER-MINOR) implement import.meta.main (Joe) #57804 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490 module: * (SEMVER-MINOR) improve typescript error message format (Marco Ippolito) #56350 * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #56350 * (SEMVER-MINOR) refactor commonjs typescript loader (Marco Ippolito) #56350 * (SEMVER-MINOR) unflag --experimental-strip-types (Marco Ippolito) #56350 permission: * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) #58853 sqlite: * (SEMVER-MINOR) add support for readBigInts option in db connection level (Miguel Marcondes Filho) #58697 src,permission: * (SEMVER-MINOR) add support to permission.has(addon) (Rafael Gonzaga) #58951 test: * (SEMVER-MINOR) add test for async disposable worker thread (James M Snell) #58385 url: * (SEMVER-MINOR) add fileURLToPathBuffer API (James M Snell) #58700 watch: * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) #58719 worker: * (SEMVER-MINOR) make Worker async disposable (James M Snell) #58385 PR-URL: #59256
Notable changes: deps: * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) #56350 doc: * add islandryu to collaborators (Shima Ryuhei) #58714 esm: * (SEMVER-MINOR) implement `import.meta.main` (Joe) #57804 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490 module: * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #56350 * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) #56350 permission: * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) #58853 sqlite: * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) #58697 src,permission: * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) #58951 url: * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) #58700 watch: * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) #58719 worker: * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) #58385 PR-URL: #59256 Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Notable changes: deps: * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) #56350 doc: * add islandryu to collaborators (Shima Ryuhei) #58714 esm: * (SEMVER-MINOR) implement `import.meta.main` (Joe) #57804 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490 module: * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #56350 * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) #56350 permission: * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) #58853 sqlite: * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) #58697 src,permission: * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) #58951 url: * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) #58700 watch: * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) #58719 worker: * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) #58385 PR-URL: #59256 Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Notable changes: deps: * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) #56350 doc: * add islandryu to collaborators (Shima Ryuhei) #58714 esm: * (SEMVER-MINOR) implement `import.meta.main` (Joe) #57804 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) #58490 module: * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) #56350 * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) #56350 permission: * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) #58853 sqlite: * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) #58697 src,permission: * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) #58951 url: * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) #58700 watch: * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) #58719 worker: * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) #58385 PR-URL: #59256 Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Notable changes: deps: * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) nodejs#56350 doc: * add islandryu to collaborators (Shima Ryuhei) nodejs#58714 esm: * (SEMVER-MINOR) implement `import.meta.main` (Joe) nodejs#57804 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) nodejs#58490 module: * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) nodejs#56350 * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) nodejs#56350 permission: * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) nodejs#58853 sqlite: * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) nodejs#58697 src,permission: * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) nodejs#58951 url: * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) nodejs#58700 watch: * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) nodejs#58719 worker: * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) nodejs#58385 PR-URL: nodejs#59256 Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Notable changes: deps: * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) nodejs#56350 doc: * add islandryu to collaborators (Shima Ryuhei) nodejs#58714 esm: * (SEMVER-MINOR) implement `import.meta.main` (Joe) nodejs#57804 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) nodejs#58490 module: * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) nodejs#56350 * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) nodejs#56350 permission: * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) nodejs#58853 sqlite: * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) nodejs#58697 src,permission: * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) nodejs#58951 url: * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) nodejs#58700 watch: * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) nodejs#58719 worker: * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) nodejs#58385 PR-URL: nodejs#59256 Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Notable changes: deps: * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) nodejs#56350 doc: * add islandryu to collaborators (Shima Ryuhei) nodejs#58714 esm: * (SEMVER-MINOR) implement `import.meta.main` (Joe) nodejs#57804 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) nodejs#58490 module: * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) nodejs#56350 * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) nodejs#56350 permission: * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) nodejs#58853 sqlite: * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) nodejs#58697 src,permission: * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) nodejs#58951 url: * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) nodejs#58700 watch: * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) nodejs#58719 worker: * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) nodejs#58385 PR-URL: nodejs#59256 Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Notable changes: deps: * (SEMVER-MINOR) update amaro to 1.1.0 (Node.js GitHub Bot) nodejs#56350 doc: * add islandryu to collaborators (Shima Ryuhei) nodejs#58714 esm: * (SEMVER-MINOR) implement `import.meta.main` (Joe) nodejs#57804 fs: * (SEMVER-MINOR) allow correct handling of burst in fs-events with AsyncIterator (Philipp Dunkel) nodejs#58490 module: * (SEMVER-MINOR) remove experimental warning from type stripping (Marco Ippolito) nodejs#56350 * (SEMVER-MINOR) unflag `--experimental-strip-types` (Marco Ippolito) nodejs#56350 permission: * (SEMVER-MINOR) propagate permission model flags on spawn (Rafael Gonzaga) nodejs#58853 sqlite: * (SEMVER-MINOR) add support for `readBigInts` option in db connection level (Miguel Marcondes Filho) nodejs#58697 src,permission: * (SEMVER-MINOR) add support to `permission.has(addon)` (Rafael Gonzaga) nodejs#58951 url: * (SEMVER-MINOR) add `fileURLToPathBuffer` API (James M Snell) nodejs#58700 watch: * (SEMVER-MINOR) add `--watch-kill-signal` flag (Dario Piotrowicz) nodejs#58719 worker: * (SEMVER-MINOR) make `Worker` async disposable (James M Snell) nodejs#58385 PR-URL: nodejs#59256 Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
This addresses a bug in
fs.watch
when used as an AsyncIterator.The issue is that when consuming the AsyncIteractor returned by
fs.watch
it yields a value. When using that value and in turn awaiting an asynchrounous action any events happening in the meantime will go missing. The reason is that between exiting the watch function by yielding and reentering it through the next round, thepromise
inside watch is already resolved. So any events generated will be duplicate resolutions of that promise and therefore ignored.More reaslistically than this minimal example is when the found files are actually read via
await fs.readFile
. Then there will be a lag. If the file events happening are happening close together, then the second one will be missed.To fix this issue I added a queue to the watch function that new file events get pushed onto. The promise is no longer resolved with a value, but is simply the gate gating whether or not there are any events in the queue. The iterator awaits the promise and then yields the items from the queue so long as there are any. When the queue is empty and the watch is still running, then a new promise is created and awaited upon. This whould eliminate the problem entirely and one can now go asynchronous in the loop as long as one wants without missing events.
Verified that the added test fails with v24.0.0 and passes after the fix.
Based on feedback the queuing was made configurable with the
maxQueue
(default2048
) option determining the maximum size of the queue and theoverflow
option deciding to eitherignore
the issue orthrow
an Error (default:'ignore'
). To effectively get back the previous behavior one would have to pass{ maxQueue: 1, overflow: 'ignore' }
.