Optimize haste map tracking of deleted files with Watchman. #8056
Conversation
Codecov Report
@@ Coverage Diff @@
## master #8056 +/- ##
==========================================
- Coverage 62.36% 62.35% -0.01%
==========================================
Files 262 262
Lines 10312 10316 +4
Branches 2491 2493 +2
==========================================
+ Hits 6431 6433 +2
Misses 3307 3307
- Partials 574 576 +2
Continue to review full report at Codecov.
|
There was a problem hiding this comment.
Ohh I like this change a lot! Nice work. I have one question inline about fresh instances.
I’m not sure if “deprecatedFiles” makes sense. Could we rename them to “removedFiles” (or “missingFiles”?) in a separate PR? What do you think?
@SimenB: is this good for you too?
| @@ -148,6 +158,8 @@ export = async function watchmanCrawl( | |||
| // files. | |||
| if (watchmanFileResults.isFresh) { | |||
| files = new Map(); | |||
| deprecatedFiles = new Map(data.files); | |||
There was a problem hiding this comment.
If watchman crashes and gives us a fresh instance, we just start from scratch, right? If so, this seems like unnecessary bookkeeping. Could you explain the idea behind this?
There was a problem hiding this comment.
If Watchman crashes, I'm just getting a full list of files. I need to also know what files were deleted because I have to pass them back for use here:
https://github.com/facebook/jest/blob/master/packages/jest-haste-map/src/index.ts#L642
There was a problem hiding this comment.
In this code path (when Watchman is fresh), that map has all existing files removed as we iterate through the files, so it ends up being just a list of removed files and nothing else.
|
@cpojer RE: Renaming from I'll submit a PR for it once this is merged in (feel free to provide any naming thoughts). |
|
@SimenB Do you want the rename in this PR or a separate one? Also, I don't think we have to worry about breaking changes here since it's all in internal methods and doesn't touch the public API. |
|
Oh, it's not exposed from |
|
@SimenB Rename didn't touch anything not already touched in this PR, pretty straightforward, so done now. Feel free to merge it in when build passes (and you're ready for the merge-- not sure what your process is). I can't. :) |
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Summary
This is a minor PR to improve the performance of tracking deleted files by taking advantage of Watchman when available.
Currently, deleted files are tracked within jest-haste-map by:
Benchmarking this operation against a large locally-generated test Haste Map of 300k~ files with one deletion, the operation currently takes about 150ms on my machine and grows linearly with more files tracked. Using Watchman makes it almost free and only grows with the number of files changed.
I've updated the non-Watchman implementation to also track deleted files within the crawler to keep the interface consistent, although that update is neutral on performance.
Test plan