Unique EmsRefresh.refresh targets if there are over 1,000 targets

[agrare]

RFC on if we should put back the unique check for ems_refresh targets

cc @Fryguy @Ladas

https://bugzilla.redhat.com/show_bug.cgi?id=1516401

[Ladas]

So the biggest problem might be in OpenShift provider, but:

1. We queue only DELETED events now, which are smaller
2. We disable the InventoryCollectorWorker queuing if the Refresh worker dies

That means we should always be at quite low amount of targets queued, then the uniq per queuing is not that bad.

---

What this solves though is, that we just saw how refresh worker failure could have lead to having 2M targets queued, where it's just like 400 unique targets possible. The 2M targets were around 4GB in memory, killing all priority, Generic and Refresh workers that met it.

cc @cben

[agrare]

So I tried queueing 2 million targets for amazon as a test:

ems = ManageIQ::Providers::Amazon::CloudManager.first
EmsRefresh.queue_refresh(2_000_000.times.map { ems })

The resulting queue item had ["msg_data", "<104000008 bytes of binary data>"] so around 100MB of data.

The RefreshWorker grew from 392MB to 1.2GB when dequeueing it.

[agrare]

@Fryguy we saw this on a 5.9.0.9 appliance so the watch collector was disabled. I still think there is something else going on to allow this to happen (refresh worker should have dequeued this) but this certainly isn't helping matters.

[agrare]

Okay @Fryguy this will uniq the target list every 1000 targets. 1000 ManagerRefresh::Targets marshal dumped is 32 KiB

[Fryguy]

Prefer Settings.ems_refresh.max_targets_uniq_threshold and get rid of the || 1000, because we will always have a value.

[Fryguy]

On that note, why make it configurable at all?

[Ladas]

hm, right, maybe this config doesn't make that much of a sense

[Ladas]

👍

[miq-bot]

Checked commit agrare@d100b56 with ruby 2.3.3, rubocop 0.47.1, haml-lint 0.20.0, and yamllint 1.10.0
1 file checked, 0 offenses detected
Everything looks fine. 👍

[agrare]

Linked to the new bug to track just this issue
@Fryguy can you take another look?

[simaishi]

Gaprindashvili backport details:

$ git log -1
commit d20081cd1cd829e03289734521bc37de6186b6a1
Author: Jason Frey <fryguy9@gmail.com>
Date:   Mon Nov 27 12:03:04 2017 -0500

    Merge pull request #16432 from agrare/add_back_ems_refresh_unique_target
    
    Unique EmsRefresh.refresh targets if there are over 1,000 targets
    (cherry picked from commit fdb09c20fdb778fb3ef0238044b2ce907f43e1ff)
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1517962