Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase timeout for metric purging #15312

Merged
merged 3 commits into from
Jun 14, 2017
Merged

Increase timeout for metric purging #15312

merged 3 commits into from
Jun 14, 2017

Conversation

jntullo
Copy link

@jntullo jntullo commented Jun 6, 2017

BZ: https://bugzilla.redhat.com/show_bug.cgi?id=1445498

Metric purging currently uses the default timeout of 10 minutes which is not long enough for an environment with thousands of VMs. This is causing metrics tables to grow in size so large that they eventually fill the entire VMDB.

This PR adds a queue_timeoutto the settings for metrics_history and performance.history so that it can be altered as needed, and increases it by default to "20.minutes"

Thoughts @yrudman?
@miq-bot bug, core/queue

@miq-bot
Copy link
Member

miq-bot commented Jun 6, 2017

@jntullo unrecognized command 'bug', ignoring...

Accepted commands are: add_label, assign, close_issue, move_issue, remove_label, rm_label, set_milestone

@jntullo
Copy link
Author

jntullo commented Jun 6, 2017

@miq-bot add_label bug, core/queue

Jillian Tullo added 2 commits June 6, 2017 09:31
@@ -895,6 +896,7 @@
:keep_hourly_performances: 6.months
:keep_realtime_performances: 4.hours
:purge_window_size: 1000
:queue_timeout: 20.minutes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we also need to increase performance_realtime_purging_interval to avoid overlapping in scheduler ?

@@ -1169,7 +1169,7 @@
:orchestration_stack_retired_interval: 10.minutes
:performance_collection_interval: 3.minutes
:performance_collection_start_delay: 5.minutes
:performance_realtime_purging_interval: 15.minutes
:performance_realtime_purging_interval: 25.minutes
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is still 5 minutes gap between timeout and next scheduler loop, does it make sense
to make purging interval very close to timeout (may be 21.minutes) ?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yrudman yeah, that's a good idea. I wasn't sure if we could keep the current 5 minute gap.

@miq-bot
Copy link
Member

miq-bot commented Jun 6, 2017

Checked commits jntullo/manageiq@da8d96b~...75f603c with ruby 2.2.6, rubocop 0.47.1, and haml-lint 0.20.0
3 files checked, 0 offenses detected
Everything looks fine. 👍

@jntullo
Copy link
Author

jntullo commented Jun 8, 2017

@miq-bot add_label euwe/yes, fine/yes

Copy link
Member

@gtanzillo gtanzillo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 LGTM

@gtanzillo gtanzillo added this to the Sprint 63 Ending Jun 19, 2017 milestone Jun 14, 2017
@gtanzillo gtanzillo merged commit 6271921 into ManageIQ:master Jun 14, 2017
@simaishi
Copy link
Contributor

Fine backport details:

$ git log -1
commit 6332db37fc931b7c4acc135d0a56dfe96f55d883
Author: Gregg Tanzillo <gtanzill@redhat.com>
Date:   Wed Jun 14 11:12:57 2017 -0400

    Merge pull request #15312 from jntullo/bz/purge_timeout
    
    Increase timeout for metric purging
    (cherry picked from commit 62719214d86bf68b169571af9a761fea6331f5f4)
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1462358

simaishi pushed a commit that referenced this pull request Jun 16, 2017
@simaishi
Copy link
Contributor

@jntullo Cherry-pick results in conflicts on all files 😞 Would you mind creating a PR for Euwe?

@jntullo
Copy link
Author

jntullo commented Jul 28, 2017

@simaishi will do!

@simaishi
Copy link
Contributor

simaishi commented Aug 2, 2017

Backported to Euwe via #15702

@jntullo jntullo deleted the bz/purge_timeout branch November 28, 2017 19:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants