spanner.cloud-client.backup_sample_test: test_update_backup failed

[flaky-bot]

Note: #3241 was also for this test, but it was closed more than 10 days ago. So, I didn't mark it flaky.

---

commit: 7a169f2
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/spanner/cloud-client/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/spanner/cloud-client/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/spanner/cloud-client/.nox/py-3-7/lib/python3.7/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline expired before operation could complete."
	debug_error_string = "{"created":"@1594896296.181900111","description":"Error received from peer ipv4:74.125.135.95:443","file":"src/core/lib/surface/call.cc","file_line":1055,"grpc_message":"Deadline expired before operation could complete.","grpc_status":4}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/workspace/spanner/cloud-client/backup_sample_test.py", line 93, in test_update_backup

backup_sample.update_backup(INSTANCE_ID, BACKUP_ID)

File "/workspace/spanner/cloud-client/backup_sample.py", line 232, in update_backup

backup.update_expire_time(new_expire_time)

File "/workspace/spanner/cloud-client/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/spanner_v1/backup.py", line 249, in update_expire_time

api.update_backup(backup_update, update_mask, metadata=metadata)

File "/workspace/spanner/cloud-client/.nox/py-3-7/lib/python3.7/site-packages/google/cloud/spanner_admin_database_v1/gapic/database_admin_client.py", line 1297, in update_backup

request, retry=retry, timeout=timeout, metadata=metadata

File "/workspace/spanner/cloud-client/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 143, in call

return wrapped_func(*args, **kwargs)

File "/workspace/spanner/cloud-client/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func

on_error=on_error,

File "/workspace/spanner/cloud-client/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target

return target()

File "/workspace/spanner/cloud-client/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout

return func(*args, **kwargs)

File "/workspace/spanner/cloud-client/.nox/py-3-7/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

File "", line 3, in raise_from

google.api_core.exceptions.DeadlineExceeded: 504 Deadline expired before operation could complete.

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

When run at the same commit (7a169f2), this test passed in one build (Build Status, Sponge) and failed in another build (Build Status, Sponge).

[busunkim96]

@larkee Was the issue with low default timeout (googleapis/python-spanner#83) resolved?

[tmatsuo]

@busunkim96
I think so. However, tests around LRO are all inevitably flaky. I would not worry much if these types of failures happening few times a year.