contrib/pkg/awstagdeprovision: Allow for OR filters

[wking]

Builds on #46; review that first.

AWS docs for filters are a bit sparse, but the AWS docs have:

Combining search filters

In general, multiple filters with the same key field (for example, tag:Name, search, Instance State) are automatically joined with OR. This is intentional, as the vast majority of filters would not be logical if they were joined with AND. For example, you would get zero results for a search on Instance State=running AND Instance State=stopped. In many cases, you can granulate the results by using complementary search terms on different key fields, where the AND rule is automatically applied instead. If you search for tag: Name:=All values and tag:Instance State=running, you get search results that contain both those criteria. To fine-tune your results, simply remove one filter in the string until the results fit your requirements.

Unfortunately, there seems to be no way to represent OR filters in ec2's Filter structure. With the approach I have here, resources that match multiple filters will be fetched multiple times, and may have parallel, racing delete attempts. But we should be robust in the face of racing delete attempt, and hopefully one of the deletes will go through before too long ;).

It would be possible to adjust our locally-filtered types (which use filterObjects) to avoid this issue for those types. I may do that in follow-up work, but for now I'm treating all of our types the same way.

[abhinavdahiya]

553c02a 8de157d merged

6e7bacc doesn't look like necessary and can be merged separate to the PR, so just drop it maybe?

7bdaa6a /lgtm

But i'm confused.. why force only one use the first filter? 7bdaa6a#diff-6df0d52da59598889c66f03dd6a7bc67R35 has aws-tag-deprovision KEY=VALUE ... ... means more than one pair might be accepted ?

[wking]

I've pushed 7bdaa6a -> f59dd98, rebasing onto master so GitHub only shows the new-to-this-PR commit in the diff.

But i'm confused.. why force only one use the first filter? 7bdaa6a#diff-6df0d52da59598889c66f03dd6a7bc67R35 has aws-tag-deprovision KEY=VALUE ... ... means more than one pair might be accepted ?

The hiveutil command has accepted multiple filters since it landed, so nothing new there (although this was only documented in #46). This PR is adding OR support to the library, but not exposing that support in hiveutil. You can still list multiple KEY=VALUE, and they'll all get fed into Filters[0] (and be AND-ed together by the library).

[joelddiaz]

@wking i'm seeing a panic when i tried to run this

[jdiaz@goomba hive (wking-or-tag-deletion %)]$ ./bin/hiveutil aws-tag-deprovision --cluster-name fakecluster --loglevel=debug tectonicClusterID=fakeuuid kubernetes.io/cluster/fakecluster=shared kubernetes.io/cluster/fakecluster=owned
panic: assignment to entry in nil map

goroutine 1 [running]:
main.parseFilter(0x0, 0x7ffde7f5a96b, 0x1a, 0x0, 0x16c5b77)
        /home/jdiaz/projects/hive/src/github.com/openshift/hive/contrib/cmd/hiveutil/awstagdeprovision.go:88 +0xa4

[wking]

panic: assignment to entry in nil map

Oops. Fixed in f59dd98 -> 971d238.

[abhinavdahiya]

/approve

[joelddiaz]

We should allow passing of multiple key=value tags that we can OR against.

[wking]

We should allow passing of multiple key=value tags that we can OR against.

Currently you can pass multiple to AND. Did you have a command-line syntax in mind for OR?

[joelddiaz]

I was thinking that each command line argument is what we OR against. No fancy combinations of ORs with ANDs...you just get OR. Do we have a need for more complex matching than ORing against a list of key=value pairs at this time?

[wking]

I was thinking that each command line argument is what we OR against.

hiveutil has supported multiple KEY=VALUE pairs ANDed together since it landed (notes here). You're suggesting I change that to use OR instead?

Do we have a need for more complex matching than ORing against a list of key=value pairs at this time?

The installer doesn't, but the installer doesn't use hiveutil ;). Folks who want to destroy clusters should be using openshift-install destroy cluster. Do you know who uses hiveutil and what their use-cases are?

[joelddiaz]

Yes, I'm suggesting to switch from ANDing the list of key=value pairs to ORing.

@dgoodwin what do you think about changing the default away from ANDing the list to ORing (and taking away the ability to do any kind of ANDing until we find a use-case for it)?

[dgoodwin]

I suspect pretty much everyone using it would be specifying single tags, but it's a little hard to know for sure and a tiny bit risky. It's probably fine to do so but we might want to communicate it a little bit.

Would a --or option seem ok?

[joelddiaz]

Are you suggesting AND-by-default, and --or will switch to 'OR all of thes key=value pairs passed as parameters'?

[dgoodwin]

Yes, but I am up for either, mild risk in changing behavior from AND to OR on folks who might be using it, but I don't think there are a lot.

[joelddiaz]

Being that the number of people using this at this time is still probably quite low, I'd argue that we should move to OR-by-default since it solves our usecase more cleanly (we can put the OR-by-default loud and clear in the '--help' output). I think we make the change now and if needed it should be easier to add AND after the fact than retrofit OR into a previously AND-by-default world.

[joelddiaz]

@wking i'm more than willing to make the proposed changes if you don't want to. you want to incorporate the OR-by-default changes?

[wking]

I've pushed 971d238 -> 7522689, adding filter qualifiers to some debug logging and converting hiveutil to the OR semantics.

[joelddiaz]

This looks good. Just one change to raise the number of goroutines to wait on.

[joelddiaz]

/lgtm