Add Concurrent.available_processor_count that is cgroups aware #1038
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
eregon
reviewed
Jan 29, 2024
eregon
reviewed
Jan 29, 2024
casperisfine
force-pushed
the
cgroup-support
branch
from
bfd027b to
298d696
Compare
|
Arf: I'll fix 2.3 compat. |
casperisfine
force-pushed
the
cgroup-support
branch
2 times, most recently
from
2ad87f7 to
aecdebb
Compare
eregon
reviewed
Jan 29, 2024
eregon
reviewed
Feb 1, 2024
casperisfine
force-pushed
the
cgroup-support
branch
from
aecdebb to
165b061
Compare
eregon
reviewed
Feb 1, 2024
Closes: ruby-concurrency#1035 A running gag since the introduction of containerization is software that starts one process per logical or physical core while running inside a container with a restricted CPU quota and totally blowing up memory usage in containerized environments. The proper question to ask is how many CPU cores are usable, not how many the machine has. To do that we have to read the cgroup info from `/sys`. There is two way of doing it depending on the version of cgroups used. Co-Authored-By: usiegl00 <50933431+usiegl00@users.noreply.github.com>
casperisfine
force-pushed
the
cgroup-support
branch
from
165b061 to
1fd2bbc
Compare
eregon
reviewed
Feb 1, 2024
|
Thanks! |
casperisfine
pushed a commit
to Shopify/rails
that referenced
this pull request
May 29, 2024
It has to be reverted because the previous implementation wasn't cgroup aware so it would often start way too many processes on various shared hosting platforms. Thanks to ruby-concurrency/concurrent-ruby#1038 concurrent-ruby 1.3 now offer a cgroups aware method to detect how many processors we can actually use.
casperisfine
pushed a commit
to Shopify/rails
that referenced
this pull request
May 29, 2024
It has to be reverted because the previous implementation wasn't cgroup aware so it would often start way too many processes on various shared hosting platforms. Thanks to ruby-concurrency/concurrent-ruby#1038 concurrent-ruby 1.3 now offer a cgroups aware method to detect how many processors we can actually use.
|
Thanks for this! The PR's title has a different method name than the one that landed. I added PR #1049 to fix the CHANGELOG. |
xjunior
pushed a commit
to xjunior/rails
that referenced
this pull request
Jun 9, 2024
It has to be reverted because the previous implementation wasn't cgroup aware so it would often start way too many processes on various shared hosting platforms. Thanks to ruby-concurrency/concurrent-ruby#1038 concurrent-ruby 1.3 now offer a cgroups aware method to detect how many processors we can actually use.
st0012
added a commit
to getsentry/sentry-ruby
that referenced
this pull request
Jul 6, 2024
It's a new API introduced in concurrent-ruby 1.3.1, which works better in the container environment. ruby-concurrency/concurrent-ruby#1038 Since there are gems like sorbet-runtime that still use older versions of concurrent-ruby, we can't directly bump concurrent-ruby's requirement, but need to check if the method is available before calling it.
Set2005
pushed a commit
to Set2005/fix-association-initialize-order
that referenced
this pull request
Jul 8, 2024
It has to be reverted because the previous implementation wasn't cgroup aware so it would often start way too many processes on various shared hosting platforms. Thanks to ruby-concurrency/concurrent-ruby#1038 concurrent-ruby 1.3 now offer a cgroups aware method to detect how many processors we can actually use.
sl0thentr0py
pushed a commit
to getsentry/sentry-ruby
that referenced
this pull request
Jul 9, 2024
* Use Concurrent.usable_processor_count when it is available It's a new API introduced in concurrent-ruby 1.3.1, which works better in the container environment. ruby-concurrency/concurrent-ruby#1038 Since there are gems like sorbet-runtime that still use older versions of concurrent-ruby, we can't directly bump concurrent-ruby's requirement, but need to check if the method is available before calling it. * Update changelog
y-yagi
added a commit
to y-yagi/parallel_tests
that referenced
this pull request
Aug 6, 2024
In containerized environments, a number of CPU cores isn't the same as the available CPUs. In this case, we need to consider cgroups. `concurrent-ruby` now has the method to get that since v1.3.1. I think it's better to use this for setting more container environment friendly default value. Ref: ruby-concurrency/concurrent-ruby#1038
4 tasks
y-yagi
added a commit
to y-yagi/parallel
that referenced
this pull request
Aug 8, 2024
In containerized environments, a number of CPU cores isn't the same as the available CPUs. In this case, we need to consider cgroups. `concurrent-ruby` now has the method to get that since v1.3.1. I think it's better to use this for setting more container environment friendly default value. Ref: ruby-concurrency/concurrent-ruby#1038
y-yagi
added a commit
to y-yagi/parallel
that referenced
this pull request
Aug 8, 2024
In containerized environments, a number of CPU cores isn't the same as the available CPUs. In this case, we need to consider cgroups. `concurrent-ruby` now has the method to get that since v1.3.1. I think it's better to use this for setting more container environment friendly default value. Ref: ruby-concurrency/concurrent-ruby#1038
y-yagi
added a commit
to y-yagi/parallel
that referenced
this pull request
Aug 8, 2024
In containerized environments, a number of CPU cores isn't the same as the available CPUs. In this case, we need to consider cgroups. `concurrent-ruby` now has the method to get that since v1.3.1. I think it's better to use this for setting more container environment friendly default value. Ref: ruby-concurrency/concurrent-ruby#1038
y-yagi
added a commit
to y-yagi/parallel
that referenced
this pull request
Aug 8, 2024
In containerized environments, a number of CPU cores isn't the same as the available CPUs. In this case, we need to consider cgroups. `concurrent-ruby` now has the method to get that since v1.3.1. I think it's better to use this for setting more container environment friendly default value. Ref: ruby-concurrency/concurrent-ruby#1038 I keep `Parallel.processor_count` as is to keep supporting setting processor count via env(`PARALLEL_PROCESSOR_COUNT`).
y-yagi
added a commit
to y-yagi/parallel
that referenced
this pull request
Aug 8, 2024
In containerized environments, a number of CPU cores isn't the same as the available CPUs. In this case, we need to consider cgroups. `concurrent-ruby` now has the method to get that since v1.3.1. I think it's better to use this for setting more container environment friendly default value. Ref: ruby-concurrency/concurrent-ruby#1038 I keep `Parallel.processor_count` as is to keep supporting setting processor count via env(`PARALLEL_PROCESSOR_COUNT`).
y-yagi
added a commit
to y-yagi/parallel
that referenced
this pull request
Aug 8, 2024
In containerized environments, a number of CPU cores isn't the same as the available CPUs. In this case, we need to consider cgroups. `concurrent-ruby` now has the method to get that since v1.3.1. I think it's better to use this for setting more container environment friendly default value. Ref: ruby-concurrency/concurrent-ruby#1038 I keep `Parallel.processor_count` as is to keep supporting setting processor count via env(`PARALLEL_PROCESSOR_COUNT`).
DanielaVelasquez
pushed a commit
to DanielaVelasquez/rails
that referenced
this pull request
Oct 3, 2024
It has to be reverted because the previous implementation wasn't cgroup aware so it would often start way too many processes on various shared hosting platforms. Thanks to ruby-concurrency/concurrent-ruby#1038 concurrent-ruby 1.3 now offer a cgroups aware method to detect how many processors we can actually use.
jeremy
added a commit
to jeremy/rails
that referenced
this pull request
Nov 14, 2024
… count (We already require concurrent-ruby 1.3.1+ with support for this.) References: * rails@f719787 * ruby-concurrency/concurrent-ruby#1038
|
Hi! The PR title says the new API was named Would someone mind updating the PR title to match? |
eregon
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Closes: #1035
A running gag since the introduction of containerization is software that starts one process per logical or physical core while running inside a container with a restricted CPU quota and totally blowing up memory usage in containerized environments.
The proper question to ask is how many CPU cores are usable, not how many the machine has. To do that we have to read the cgroup info from
/sys. There is two way of doing it depending on the version of cgroups used.Co-Authored-By: @usiegl00