-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a GNU make jobserver implementation to Cargo #4110
Merged
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Add a GNU make jobserver implementation to Cargo
This commit adds a GNU make jobserver implementation to Cargo, both as a client of existing jobservers and also a creator of new jobservers. The jobserver is actually just an IPC semaphore which manifests itself as a pipe with N bytes of tokens on Unix and a literal IPC semaphore on Windows. The rough protocol is then if you want to run a job you read acquire the semaphore (read a byte on Unix or wait on the semaphore on Windows) and then you release it when you're done. All the hairy details of the jobserver implementation are housed in the `jobserver` crate on crates.io instead of Cargo. This should hopefully make it much easier for the compiler to also share a jobserver implementation eventually. The main tricky bit here is that on Unix and Windows acquiring a jobserver token will block the calling thread. We need to either way for a running job to exit or to acquire a new token when we want to spawn a new job. To handle this the current implementation spawns a helper thread that does the blocking and sends a message back to Cargo when it receives a token. It's a little trickier with shutting down this thread gracefully as well but more details can be found in the `jobserver` crate. Unfortunately crates are unlikely to see an immediate benefit of this once implemented. Most crates are run with a manual `make -jN` and this overrides the jobserver in the environment, creating a new jobserver in the sub-make. If the `-jN` argument is removed, however, then `make` will share Cargo's jobserver and properly limit parallelism. Closes #1744
- Loading branch information
commit cbf25a9b0ae5ac6f5b6da96e645b7fa6a75dc245
There are no files selected for viewing
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
s/i sno/is no
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting, looks like this transmute is always safe, because
Sender
is contravariant.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's what I originally thought yeah but actually I don't think so (unfortunately). I believe
Drop for Sender
may run destructors for items in the internal queue (not received yet), which means that if you persist aSender
beyond the lifetime of the item I think it'll access data outside of its lifetime.(not in this case though, the
Sender
should always go away with the stack frame.