Integration test failed - timeout in multi_tenancy

[awrichar]

Failed in integration run 315 (TestEthereumMultipartyE2ESuite, besu, postgres)

https://github.com/hyperledger/firefly/runs/8028671474?check_suite_focus=true#step:6:2488

Performed reset, waited for FireFly to come up, but then failed to connect websocket.

    restclient.go:109: 2022-08-26T02:29:07.217696924Z: ==> POST /reset map[]: {}
    restclient.go:116: 2022-08-26T02:29:07.218936618Z: <== 204
    restclient.go:109: 2022-08-26T02:29:07.218978418Z: ==> POST /reset map[]: {}
    restclient.go:116: 2022-08-26T02:29:07.221382306Z: <== 204
    restclient.go:109: 2022-08-26T02:29:07.221468206Z: ==> GET /namespaces/default/status map[]: null
Websocket 127.0.0.1:5001 closing, error: websocket: close 1006 (abnormal closure): unexpected EOF
    restclient.go:116: 2022-08-26T02:29:07.577115935Z: <== 200
    restclient.go:109: 2022-08-26T02:29:07.577204835Z: ==> GET /namespaces/default/status map[]: null
Websocket 127.0.0.1:5000 closing, error: websocket: close 1006 (abnormal closure): unexpected EOF
    restclient.go:109: 2022-08-26T02:29:12.580516414Z: ==> GET /namespaces/default/status map[]: null
    restclient.go:109: 2022-08-26T02:29:17.584558307Z: ==> GET /namespaces/default/status map[]: null
    restclient.go:116: 2022-08-26T02:29:18.001983709Z: <== 200
    restclient.go:143: 
        	Error Trace:	/home/runner/work/firefly/firefly/test/e2e/runners/restclient.go:143
        	            				/home/runner/work/firefly/firefly/test/e2e/runners/multi_tenancy.go:123
        	Error:      	Received unexpected error:
        	            	unexpected EOF
        	Test:       	TestEthereumMultipartyE2ESuite/TestMultiTenancy
    restclient.go:109: 2022-08-26T02:29:18.002570006Z: ==> GET /namespaces/default/operations map[created:[>1661480942620516909]]: null
    restclient.go:917: 
        	Error Trace:	/home/runner/work/firefly/firefly/test/e2e/runners/restclient.go:917
        	            				/home/runner/work/firefly/firefly/test/e2e/runners/e2e.go:233
        	            				/home/runner/work/firefly/firefly/test/e2e/runners/multi_tenancy.go:45
        	            				/home/runner/work/firefly/firefly/test/e2e/runners/suite.go:153
        	            				/home/runner/work/firefly/firefly/test/e2e/runners/panic.go:642
        	            				/home/runner/work/firefly/firefly/test/e2e/runners/testing.go:756
        	            				/home/runner/work/firefly/firefly/test/e2e/runners/restclient.go:143
        	            				/home/runner/work/firefly/firefly/test/e2e/runners/multi_tenancy.go:123
        	Error:      	Received unexpected error:
        	            	Get "http://127.0.0.1:5000/api/v1/namespaces/default/operations?created=%3E1661480942620516909": EOF
        	Test:       	TestEthereumMultipartyE2ESuite/TestMultiTenancy

[awrichar]

We're only checking for the old namespaces to be up... seems like maybe a race condition where the newly created namespaces are still initializing.