steps/infra: serialize master and worker domain sets for libvirt

[abhinavdahiya]

We are seeing race like symptoms when creating multiple domainsets in parallel.
For more info: dmacvicar/terraform-provider-libvirt#402
The issue suggests that its most proabaly a libvirt bug that we can avoid by
serializing master and worker domain sets.

[wking]

The issue suggests that its most probably a libvirt bug that we can avoid by
serializing master and worker domain sets.

How much do we care about working around libvirt bugs? A patch for this one appears to have been out since April; can we just ask users to stay up to date? That said, it's a small-enough change (although I don't really understand how it works, won't individual master domains still be created in parallel?). So I'm fine either way.

[abhinavdahiya]

@wking

A patch for this one appears to have been out since April; can we just ask users to stay up to date?

The latest version available for fedora 28 by default is 4.1.0 (through dnf)... I'm not sure if we want everybody to go out and install new libvirt through rpms... ? @crawford says that might me a little difficult to enforce on everybody...

That said, it's a small-enough change (although I don't really understand how it works, won't individual master domains still be created in parallel?)

I don't understand terraform enough, but it looks like terraform installs domains in one set one after the other; but 2 different sets go at once maybe. :/

[wking]

The latest version available for fedora 28 by default is 4.1.0 (through dnf)...

Looks like 4.1.0 is from 2018-03-05, so before the April libvirt/libvirt@4d7384eb. Has the patch not been backported to Fedora 28? I don't know anything about Fedora package maintenance (yet?). Backporting the patch for Fedora 28 seems like a fix that would help more people than landing a workaround just for this project. But if we have a number of Fedora 28 users and getting Fedora 28 patched would take too long, then yeah, we'll want to land the workaround here to tide us over.

[crawford]

@wking I think we should do both. Backporting would help users and, in the meantime, we can carry this workaround.

/approve

[wking]

/lgtm

[openshift-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya, crawford, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [abhinavdahiya,crawford,wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[wking]

I've filed rhbz#1626680 requesting a backport to Fedora 28. We can drop this workaround a bit after that gets addressed.

[wking]

The latest version available for fedora 28 by default is 4.1.0...

@abhinavdahiya, it seems like libvirt-4.1.0-5.fc28 had fixed this on 2018-08-30 (see here and here). Can you confirm? If so, can we revert this workaround?