Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temporary fix to narrow the scope of cached TaskRuns to prevent OOMs #336

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

mshaposhnik
Copy link
Collaborator

@mshaposhnik mshaposhnik commented Nov 7, 2024

We have OOMs on prod, with mpc container consuming over 8Gb RAM. This is supposedly due ti high number of TaskRuns on cluster (>10000, and we cache them all)
This PR limits caches TRs to just those belonging to MPC

Possibe fix for https://issues.redhat.com/browse/KFLUXINFRA-887

Signed-off-by: Max Shaposhnyk <mshaposh@redhat.com>
@mshaposhnik mshaposhnik changed the title Temporary fix to norrow the scope of cached TaskRuns to prevent OOMs Temporary fix to narrow the scope of cached TaskRuns to prevent OOMs Nov 7, 2024
Copy link
Member

@ifireball ifireball left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm trying to understand what would be the downside of doing this. Worst case - if we need to reconcile a task that is not in the cache, then we end up with slower reconciliation for it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants