E2E Tests - Flakes in Istio and AllInOne Tests

[shaneutt]

Current Behavior

A test flake appears to be showing up in our Istio nightly E2E tests:

 === CONT  TestIstioWithKongIngressGateway
    istio_test.go:308: 
        	Error Trace:	istio_test.go:308
        	Error:      	Condition never satisfied
        	Test:       	TestIstioWithKongIngressGateway
    istio_test.go:92: cleaning up istio test cluster 1c6f3d90-a772-40e6-8171-923ad6a75a67

And our all-in-one tests:

 === CONT  TestDeployAllInOneDBLESSNoLoadBalancer
    all_in_one_test.go:337: 
        	Error Trace:	all_in_one_test.go:337
        	            				all_in_one_test.go:137
        	Error:      	Condition never satisfied
        	Test:       	TestDeployAllInOneDBLESSNoLoadBalancer

[shaneutt]

I was able to reproduce the problem with the TestDeployAllInOneDBLESSNoLoadBalancer test. It appears this one (unlike the Istio one) is consistent. The problem is due to health checks for the ingress controller never becoming healthy because the health checks rely on the LoadBalancer type service becoming provisioned in order to consider the kong deployment healthy. It would seem that things have changed around this recently so I'm going to review the recent changes that started triggering this failure and determine if the best course of action is to change that health check, or change the test.