Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kqueue: Only make a single syscall to add all the watches #454

Merged
merged 3 commits into from
Jan 14, 2023

Conversation

alexkirsz
Copy link
Contributor

@alexkirsz alexkirsz commented Nov 16, 2022

Hey!

We use notify v4 in turbopack and we've recently tried updating to v5 to use the kqueue implementation, as we found it to be much faster than FSEvents at detecting updates.

However, we also noticed a very large drop in performance when watching large directories. After a bit of investigating, it turns out that notify makes a kevent syscall for each file it watches. Watching a folder with 15k files ends up taking about 11.5s.

This PR changes this behavior to ensure we only make a single kevent call when adding multiple files at the same time. This improves the above case to take 900ms.

I've also filed an issue with kqueue to fix another performance problem, which makes watching n files O(n²). This would further improve to 400ms.

Finally, while profiling the above issue, I noticed that during the 12s of watching, only 2 cores of my computer were working. I have a prototype version that uses multiple threads to open files, which might be faster when watching large directories, but it would also require changes to kqueue, so I'm filing it away for now.

@0xpr03
Copy link
Member

0xpr03 commented Nov 16, 2022

cc @xanderio who added this initially for BSD, my kqueue knowledge is kinda limited

@0xpr03
Copy link
Member

0xpr03 commented Nov 16, 2022

I'm intending to accept this. I'd like to have some confirmation from BSD & Apple folks that this actually works as intended for single files & recursive folders, since we're now in stable v5.

@0xpr03 0xpr03 merged commit d765321 into notify-rs:main Jan 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants