-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implementation help #2
Comments
This library is still in its infancy. It's probably already usable, but there are likely a lot of bugs. Using it to parallelize the transforms would be pretty amazing. Note that this code is really one-way, it does not implement a "passthrough" context. Doing this would be amazing. I probably do not have a lot of time to brainstorm right now... I'm focusing on shipping the next pino release with this in it! |
I'll just leave my thoughts so far, maybe it will help plant a thought seed 🌱 after some more thought, a few changes to the first layout. I think forks should send back(or send back an ack and forward to a write stream) their work every round instead of holding onto it to aggregate at the end - I came to this conclusion for a few reasons. Firstly and most intuitively, the total amount of work of sending finished work will be the same, so its better to chunk it if possible - the original reason for forks to hold onto finished work was that spamming relatively large messages would have overhead, which it may, but you'll have to pay that somewhere no matter what. Secondly, if forks were to hold their finished work, they could theoretically grow beyond memory limits. Thirdly, I originally thought forks should do their own file io, but it is probably better for the master to do it and transfer to workers threads - which should mean no need for cluster all together. I was originally thinking of a system that would be limited to message passing, but moving file io to master and transferring the memory is possible , so it should probably do that instead.
I think however this gets handled, in addition to the resolve algorithm, will be the two most important things that impact overall speed. |
Hi, i have been working on a fork of browserify and have been wanting to implement a worker pool for transforms using something like this so that it can keep up with esbuild. here is a repo with the bundler in tools/omnify/bundler. The speed increase (compared to browserify) is mostly coming from replacing acorn/detective with regex. https://github.com/jeremy-coleman/esbuild-vs-omnify-r3f . I think this would be a really good dogfood project for this lib and node streams in general. If you think it’d be interesting and would like to brainstorm , lmk.
The text was updated successfully, but these errors were encountered: