Don't hash contents of every watched file #8
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
From https://github.com/gmethvin/directory-watcher?tab=readme-ov-file#configuration: > By default, DirectoryWatcher will try to prevent duplicate events (...). This is done by creating a hash for every file encountered and keeping that hash in memory. This might result in slower performance, because the library has to calculate the hash of the entire file. > ... > In the above example we use the last modified time hasher. This hasher is only suitable for platforms that have at least millisecond precision in last modified times from Java. It's known to work with JDK 10+ on Macs with APFS. The default is rather slow actually. On a 230mb directory with 1700 files, the initial watch call takes 2951ms. With the changes in this PR (which are the changes shown in their docs) it goes down to 2ms. 2951ms isn't terribly bad mind you... but I came across this problem when watching a 12.5gb folder with 180k files instead. The watcher never really started in that case. With these changes it started in 5659ms. Regarding the restrictions listed, APFS is the default on Macs since 10.13 (released late 2017). Perhaps this should not be the default, but I will leave that to your judgement.
|
@filipesilva Maybe you can bring down the loading time even more with a different fileTreeVisitor . I saw quite a big difference there as well. |
|
Need to benchmark that approach. Maybe the real PR here is to add something that supports a custom |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
From https://github.com/gmethvin/directory-watcher?tab=readme-ov-file#configuration:
The default is rather slow actually. On a 230mb directory with 1700 files, the initial watch call takes 2951ms. With the changes in this PR (which are the changes shown in their docs) it goes down to 2ms.
2951ms isn't terribly bad mind you... but I came across this problem when watching a 12.5gb folder with 180k files instead. The watcher never really started in that case. With these changes it started in 5659ms.
Regarding the restrictions listed, APFS is the default on Macs since 10.13 (released late 2017). Perhaps this should not be the default, but I will leave that to your judgement.