samples.instanceadmin.test_instanceadmin: test_add_and_delete_cluster failed

[flaky-bot]

This test failed!

To configure my behavior, see the Flaky Bot documentation.

If I'm commenting on this issue too often, add the flakybot: quiet label and
I will stop commenting.

---

commit: 1e51285
buildURL: Build Status, Sponge
status: failed

Test output

args = (parent: "projects/python-docs-samples-tests/instances/instanceadmin-636-1625735059"
cluster_id: "instanceadmin-920"
c...ocation: "projects/python-docs-samples-tests/locations/us-central1-a"
  serve_nodes: 1
  default_storage_type: SSD
}
,)
kwargs = {'metadata': [('x-goog-request-params', 'parent=projects/python-docs-samples-tests/instances/instanceadmin-636-1625735059'), ('x-goog-api-client', 'gl-python/3.6.13 grpc/1.38.1 gax/1.30.0')]}
@six.wraps(callable_)
def error_remapped_callable(*args, **kwargs):
    try:


      return callable_(*args, **kwargs)


.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py:67:

self = <grpc._channel._UnaryUnaryMultiCallable object at 0x7efeafef4fd0>

request = parent: "projects/python-docs-samples-tests/instances/instanceadmin-636-1625735059"

cluster_id: "instanceadmin-920"

cl... location: "projects/python-docs-samples-tests/locations/us-central1-a"

serve_nodes: 1

default_storage_type: SSD

}
timeout = None

metadata = [('x-goog-request-params', 'parent=projects/python-docs-samples-tests/instances/instanceadmin-636-1625735059'), ('x-goog-api-client', 'gl-python/3.6.13 grpc/1.38.1 gax/1.30.0')]

credentials = None, wait_for_ready = None, compression = None
def __call__(self,
             request,
             timeout=None,
             metadata=None,
             credentials=None,
             wait_for_ready=None,
             compression=None):
    state, call, = self._blocking(request, timeout, metadata, credentials,
                                  wait_for_ready, compression)


  return _end_unary_response_blocking(state, call, False, None)


.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py:946:

state = <grpc._channel._RPCState object at 0x7efeafe0f978>

call = <grpc._cython.cygrpc.SegregatedCall object at 0x7efeadd1f508>

with_call = False, deadline = None
def _end_unary_response_blocking(state, call, with_call, deadline):
    if state.code is grpc.StatusCode.OK:
        if with_call:
            rendezvous = _MultiThreadedRendezvous(state, call, None, deadline)
            return state.response, rendezvous
        else:
            return state.response
    else:


      raise _InactiveRpcError(state)


E           grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:

E           	status = StatusCode.UNAVAILABLE

E           	details = "The instance is currently being changed, please try again."

E           	debug_error_string = "{"created":"@1625735069.728455632","description":"Error received from peer ipv4:142.250.99.95:443","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"The instance is currently being changed, please try again.","grpc_status":14}"

E           >
.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py:849: _InactiveRpcError
The above exception was the direct cause of the following exception:
capsys = <_pytest.capture.CaptureFixture object at 0x7efeafe89240>

dispose_of = <function dispose_of..disposal at 0x7efeb0eee2f0>
def test_add_and_delete_cluster(capsys, dispose_of):
    dispose_of(INSTANCE)

    # This won't work, because the instance isn't created yet
    instanceadmin.add_cluster(PROJECT, INSTANCE, CLUSTER2)
    out = capsys.readouterr().out
    assert f"Instance {INSTANCE} does not exist" in out

    # Get the instance created
    instanceadmin.run_instance_operations(PROJECT, INSTANCE, CLUSTER1)
    capsys.readouterr()  # throw away output

    # Add a cluster to that instance


  instanceadmin.add_cluster(PROJECT, INSTANCE, CLUSTER2)


test_instanceadmin.py:131:

instanceadmin.py:158: in add_cluster

cluster.create()

../../google/cloud/bigtable/cluster.py:301: in create

"cluster": cluster_pb,

../../google/cloud/bigtable_admin_v2/services/bigtable_instance_admin/client.py:1001: in create_cluster

response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)

.nox/py-3-6/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py:145: in call

return wrapped_func(*args, **kwargs)

.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py:69: in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

value = None

from_value = <_InactiveRpcError of RPC that terminated with:

status = StatusCode.UNAVAILABLE

details = "The instance is currently...l.cc","file_line":1066,"grpc_message":"The instance is currently being changed, please try again.","grpc_status":14}"



???

E   google.api_core.exceptions.ServiceUnavailable: 503 The instance is currently being changed, please try again.

:3: ServiceUnavailable

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

When run at the same commit (1e51285), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

[flaky-bot]

Oops! Looks like this issue is still flaky. It failed again. 😬

I reopened the issue, but a human will need to close it again.

---

commit: aa5b606
buildURL: Build Status, Sponge
status: failed

Test output

args = (parent: "projects/python-docs-samples-tests/instances/instanceadmin-825-1626080660"
cluster_id: "instanceadmin-148"
c...ocation: "projects/python-docs-samples-tests/locations/us-central1-a"
  serve_nodes: 1
  default_storage_type: SSD
}
,)
kwargs = {'metadata': [('x-goog-request-params', 'parent=projects/python-docs-samples-tests/instances/instanceadmin-825-1626080660'), ('x-goog-api-client', 'gl-python/3.6.13 grpc/1.38.1 gax/1.31.0')]}
@six.wraps(callable_)
def error_remapped_callable(*args, **kwargs):
    try:


      return callable_(*args, **kwargs)


.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py:67:

self = <grpc._channel._UnaryUnaryMultiCallable object at 0x7fd07d3026d8>

request = parent: "projects/python-docs-samples-tests/instances/instanceadmin-825-1626080660"

cluster_id: "instanceadmin-148"

cl... location: "projects/python-docs-samples-tests/locations/us-central1-a"

serve_nodes: 1

default_storage_type: SSD

}
timeout = None

metadata = [('x-goog-request-params', 'parent=projects/python-docs-samples-tests/instances/instanceadmin-825-1626080660'), ('x-goog-api-client', 'gl-python/3.6.13 grpc/1.38.1 gax/1.31.0')]

credentials = None, wait_for_ready = None, compression = None
def __call__(self,
             request,
             timeout=None,
             metadata=None,
             credentials=None,
             wait_for_ready=None,
             compression=None):
    state, call, = self._blocking(request, timeout, metadata, credentials,
                                  wait_for_ready, compression)


  return _end_unary_response_blocking(state, call, False, None)


.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py:946:

state = <grpc._channel._RPCState object at 0x7fd07d2fc550>

call = <grpc._cython.cygrpc.SegregatedCall object at 0x7fd07c19a348>

with_call = False, deadline = None
def _end_unary_response_blocking(state, call, with_call, deadline):
    if state.code is grpc.StatusCode.OK:
        if with_call:
            rendezvous = _MultiThreadedRendezvous(state, call, None, deadline)
            return state.response, rendezvous
        else:
            return state.response
    else:


      raise _InactiveRpcError(state)


E           grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:

E           	status = StatusCode.UNAVAILABLE

E           	details = "The instance is currently being changed, please try again."

E           	debug_error_string = "{"created":"@1626080671.204595768","description":"Error received from peer ipv4:74.125.142.95:443","file":"src/core/lib/surface/call.cc","file_line":1066,"grpc_message":"The instance is currently being changed, please try again.","grpc_status":14}"

E           >
.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py:849: _InactiveRpcError
The above exception was the direct cause of the following exception:
capsys = <_pytest.capture.CaptureFixture object at 0x7fd07d2cf6a0>

dispose_of = <function dispose_of..disposal at 0x7fd07e3652f0>
def test_add_and_delete_cluster(capsys, dispose_of):
    dispose_of(INSTANCE)

    # This won't work, because the instance isn't created yet
    instanceadmin.add_cluster(PROJECT, INSTANCE, CLUSTER2)
    out = capsys.readouterr().out
    assert f"Instance {INSTANCE} does not exist" in out

    # Get the instance created
    instanceadmin.run_instance_operations(PROJECT, INSTANCE, CLUSTER1)
    capsys.readouterr()  # throw away output

    # Add a cluster to that instance


  instanceadmin.add_cluster(PROJECT, INSTANCE, CLUSTER2)


test_instanceadmin.py:131:

instanceadmin.py:158: in add_cluster

cluster.create()

../../google/cloud/bigtable/cluster.py:301: in create

"cluster": cluster_pb,

../../google/cloud/bigtable_admin_v2/services/bigtable_instance_admin/client.py:1001: in create_cluster

response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)

.nox/py-3-6/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py:145: in call

return wrapped_func(*args, **kwargs)

.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py:69: in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

value = None

from_value = <_InactiveRpcError of RPC that terminated with:

status = StatusCode.UNAVAILABLE

details = "The instance is currently...l.cc","file_line":1066,"grpc_message":"The instance is currently being changed, please try again.","grpc_status":14}"



???

E   google.api_core.exceptions.ServiceUnavailable: 503 The instance is currently being changed, please try again.

:3: ServiceUnavailable

[tseaver]

This failure is not (as I had earlier assumed) fixable by polling the LRO returned from Cluster.create with a long-enough timeout. Rather, it is being raised directly from the CreateCluster API, before the LRO is constructed. That API has no retry semantics configured in the proto, which means we can't expect it to be retried automagically.

instanceadmin.py:158: in add_cluster
    cluster.create()
../../google/cloud/bigtable/cluster.py:301: in create
    "cluster": cluster_pb,
../../google/cloud/bigtable_admin_v2/services/bigtable_instance_admin/client.py:1001: in create_cluster
    response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
.nox/py-3-6/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py:145: in __call__
    return wrapped_func(*args, **kwargs)
.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py:69: in error_remapped_callable
    six.raise_from(exceptions.from_grpc_error(exc), exc)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

value = None
from_value = <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.UNAVAILABLE
	details = "The instance is currently...l.cc","file_line":1066,"grpc_message":"The instance is currently being changed, please try again.","grpc_status":14}"

None of the API-invoking Cluster and Instance CRUD methods expose retry or timeout parameters, which means that we can't even add a custom retry in the sample to fix this flake. @kolea2 should we update them to allow such usage?

[tseaver]

@kolea2 Seeing that samples for other API clients (automl, datalabeling, translate) are using the backoff library, I guess we could update the samples which flake here (particularly adding a cluster to a newly-created instance) to use it, e.g.:

import backoff

...

    @backoff.on_exception(backoff.expo, exceptions.ServiceUnavailable)
    def do_create_cluster(cluster):
        return cluster.create()

    operation = do_create_cluster(cluster)
    operation.result(timeout=30)