Introduce report result purging timer #13044
Conversation
@kbrock Will users have to truncate or purge this table manually before this code is run? I could imagine the first scheduled one could hit the message timeout if the table hasn't been pre-purged before. |
|
@jrafanie even if a user is purging for the first time and they choose 'purge all but 5 newest` (historically slower) - it should still only take a few minutes. Aah, they are purging associated records, including blobs. this may take a bit. I'll play with this |
ok, yeah @kbrock, I was basing my comment on this:
If we have lots of results, is it possible to timeout the queue message...? |
We have cut down considerably the amount of data going over the wire and the amount of data that is being serialized. running tests to see how expensive it is to delete a lot |
|
@jrafanie |
| @@ -1383,6 +1383,7 @@ | |||
| :performance_rollup_purging_start_delay: 5.minutes | |||
| :policy_events_purge_interval: 1.day | |||
| :poll: 15.seconds | |||
| :report_result_purge_interval: 1.days | |||
There was a problem hiding this comment.
this may be too frequent. any suggestions would be great
There was a problem hiding this comment.
Is 1.month too infrequent? If so, how is 7.days?
kbrock
force-pushed
the
purge_report_results
branch
from
6c317d9
to
90957ae
Compare
|
done. changed to 1/month |
| @@ -213,6 +213,12 @@ def schedules_for_scheduler_role | |||
| enqueue :miq_alert_evaluate_hourly_timer | |||
| end | |||
|
|
|||
| # Schedule - Prune old reports Timer | |||
| every = worker_setting_or_default(:report_result_purge_interval, 1.day) | |||
There was a problem hiding this comment.
This default should probably match what's in the settings.yml
| @@ -1383,6 +1383,7 @@ | |||
| :performance_rollup_purging_start_delay: 5.minutes | |||
| :policy_events_purge_interval: 1.day | |||
| :poll: 15.seconds | |||
| :report_result_purge_interval: 1.months | |||
There was a problem hiding this comment.
The thing I don't like about 1.month is that if the server is restarted, the timer also restarts...not sure how often appliances are restarted.
|
<pr_mergeability_checker />This pull request is not mergeable. Please rebase and repush. |
kbrock
force-pushed
the
purge_report_results
branch
from
90957ae
to
15ad153
Compare
|
fixed merge conflict. |
The resources required for report result purging has been reduced, so make sure this is scheduled. https://bugzilla.redhat.com/show_bug.cgi?id=1348625
kbrock
force-pushed
the
purge_report_results
branch
from
15ad153
to
2f78249
Compare
|
Checked commit kbrock@2f78249 with ruby 2.2.5, rubocop 0.37.2, and haml-lint 0.16.1 |
| @@ -117,6 +117,10 @@ def policy_event_purge_timer | |||
| queue_work(:class_name => "PolicyEvent", :method_name => "purge_timer", :zone => nil) | |||
| end | |||
|
|
|||
| def miq_report_result_purge_timer | |||
| queue_work(:class_name => "MiqReportResult", :method_name => "purge_timer", :zone => nil) | |||
There was a problem hiding this comment.
- nil zone
- schedules_for_scheduler_role
This means the singular regional "scheduler" role schedule worker will run this and queue it for the deletion of old report results for the whole region. If you were concerned with who should run it, I think the above confirms you did it right.
There was a problem hiding this comment.
Note, I'm not sure how you could really test this cleanly
|
Euwe backport conflict: $ git diff
diff --cc config/settings.yml
index 0f79392,8ed260b..0000000
--- a/config/settings.yml
+++ b/config/settings.yml
@@@ -1364,6 -1392,8 +1364,11 @@@
:performance_rollup_purging_start_delay: 5.minutes
:policy_events_purge_interval: 1.day
:poll: 15.seconds
++<<<<<<< HEAD
++=======
+ :resync_rhn_mirror: 12.hours
+ :report_result_purge_interval: 1.week
++>>>>>>> 1bd9279... Merge pull request #13044 from kbrock/purge_report_results
:server_log_stats_interval: 5.minutes
:server_stats_interval: 60.seconds
:service_retired_interval: 10.minutes |
|
@kbrock Please resolve conflict and make Euwe-specific PR (referencing this one) or suggest other PRs to backport. |
|
Backported to Euwe via #13429 |
The resources required to purge report results used to be quite large.
So it was not scheduled
In many databases, report results is the second or third largest table (by memory usage)
before
customers manually purged the report results table
after
the report results table is checked once per week.
concerns
While these can be run by any worker, I am not sure which worker needs to schedule it.
So I put it under the
schedules_for_scheduler_rolehttps://bugzilla.redhat.com/show_bug.cgi?id=1348625