Increase limits_cpu for the short_running_workers #615
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Seems like that when we have spikes, above the limits_cpu, we sometimes got an hard time limit exception for the celery tasks.
majamassarini
force-pushed
the
increase-cpu-limits-for-short-running-workers
branch
from
5943e59 to
1f61425
Compare
|
Build succeeded. ✔️ pre-commit SUCCESS in 1m 41s |
mfocko
approved these changes
Nov 28, 2024
|
Build succeeded (gate pipeline). ✔️ pre-commit SUCCESS in 1m 40s |
softwarefactory-project-zuul
bot
merged commit c093640
into
packit:main
3 of 4 checks passed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I am monitoring the hard time limits exceptions since a while, seems like that when we have spikes both in the short running workers and in the long running we sometimes get an hard time limit exception for the celery tasks.
We have spike above the requested limits (as you can see in the today picture) mainly for the short running workers.
We have exceptions for the spikes even when the single worker cpu usage is below the
limits_cpuand I am wondering if this limit applies to the sum of all our replicas or just for the single replica?I would say it is the limit for all the replicas together and in this case we need to increase much more the
limits_cpuboth for the long running and the short running pods (I would put there 1G oflimits_cpufor both kind of workers in this case).One of the highest cpu requests (for just a single short running worker) I registered:
