Support e2e runs despite custom taints on nodes

[johnSchnake]

Upstream tracking issue for the ability to run e2e tests despite having custom taints on nodes.

kubernetes/kubernetes#74282

Would resolve things like #571 and #507

[johnSchnake]

For clarity; putting this on the milestone for v0.14.2 as it is being worked on. The feature won't be supported (even if it merges) until the next k8s release (I don't think this will get cherry picked; I could be wrong). So it will be a v0.15.0 feature eventually.

[johnSchnake]

As I was working on this I actually ran into another dev that had a WIP PR to solve the same issue: kubernetes/kubernetes#76654

His approach allows the user to provide label selectors for nodes for which being non-schedulable is not an error condition in the tests.

[johnSchnake]

Moved to next milestone for tracking.

[johnSchnake]

Slid in priority and missed the v1.15 k8s code freeze; will be shooting for v1.16 and more recent PR variation is here: kubernetes/kubernetes#78500

[hectorj2f]

Won't this be included as part of the 0.15.2 version ? We're also facing this issue.

[johnSchnake]

@hectorj2f this is an upstream k8s issue, not just a Sonobuoy issue so we are limited by their release/freeze schedule.

kubernetes/kubernetes#81043 just merged, fixing the issue (allowing you to work around it I should say) but will be in v1.17 of k8s. So until then you just have to either untaint the nodes or set the flag for allowing more not-ready nodes. (I think you can get context on those two approaches by looking at the original k8s ticket for this).

[hectorj2f]

Alright @johnSchnake. V1.17 sounds like too far in time.