dataproc.create_cluster_test: test_cluster_create failed

[flaky-bot]

This test failed!

To configure my behavior, see the Build Cop Bot documentation.

If I'm commenting on this issue too often, add the buildcop: quiet label and
I will stop commenting.

---

commit: d671543
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/dataproc/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/dataproc/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/dataproc/.nox/py-3-8/lib/python3.8/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1591012437.589734564","description":"Error received from peer ipv4:74.125.31.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/workspace/dataproc/create_cluster_test.py", line 44, in test_cluster_create

create_cluster.create_cluster(PROJECT_ID, REGION, CLUSTER_NAME)

File "/workspace/dataproc/create_cluster.py", line 62, in create_cluster

operation = cluster_client.create_cluster(project_id, region, cluster)

File "/workspace/dataproc/.nox/py-3-8/lib/python3.8/site-packages/google/cloud/dataproc_v1/gapic/cluster_controller_client.py", line 289, in create_cluster

operation = self._inner_api_calls["create_cluster"](

File "/workspace/dataproc/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/gapic_v1/method.py", line 143, in call

return wrapped_func(*args, **kwargs)

File "/workspace/dataproc/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py", line 281, in retry_wrapped_func

return retry_target(

File "/workspace/dataproc/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/retry.py", line 184, in retry_target

return target()

File "/workspace/dataproc/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout

return func(*args, **kwargs)

File "/workspace/dataproc/.nox/py-3-8/lib/python3.8/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

File "", line 3, in raise_from

google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

[flaky-bot]

Test passed for commit e7e6f93 (Build Status, Sponge)! Closing this issue.

[flaky-bot]

Looks like this issue is flaky. 😟

I'm going to leave this open and stop commenting.

A human should fix and close this.

---

commit: 572d951
buildURL: Build Status, Sponge
status: failed

Test output

Traceback (most recent call last):
  File "/workspace/dataproc/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
    return callable_(*args, **kwargs)
  File "/workspace/dataproc/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 826, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/workspace/dataproc/.nox/py-3-6/lib/python3.6/site-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
	status = StatusCode.DEADLINE_EXCEEDED
	details = "Deadline Exceeded"
	debug_error_string = "{"created":"@1591178408.468149281","description":"Error received from peer ipv4:172.217.203.95:443","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Deadline Exceeded","grpc_status":4}"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):

File "/workspace/dataproc/create_cluster_test.py", line 44, in test_cluster_create

create_cluster.create_cluster(PROJECT_ID, REGION, CLUSTER_NAME)

File "/workspace/dataproc/create_cluster.py", line 62, in create_cluster

operation = cluster_client.create_cluster(project_id, region, cluster)

File "/workspace/dataproc/.nox/py-3-6/lib/python3.6/site-packages/google/cloud/dataproc_v1/gapic/cluster_controller_client.py", line 290, in create_cluster

request, retry=retry, timeout=timeout, metadata=metadata

File "/workspace/dataproc/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/gapic_v1/method.py", line 143, in call

return wrapped_func(*args, **kwargs)

File "/workspace/dataproc/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func

on_error=on_error,

File "/workspace/dataproc/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/retry.py", line 184, in retry_target

return target()

File "/workspace/dataproc/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/timeout.py", line 214, in func_with_timeout

return func(*args, **kwargs)

File "/workspace/dataproc/.nox/py-3-6/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 59, in error_remapped_callable

six.raise_from(exceptions.from_grpc_error(exc), exc)

File "", line 3, in raise_from

google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

[tmatsuo]

@bradmiro

JFYI, here is what I would do:

1. All the RPC is flaky. Look at the code and if any RPCs are not retried, make a change and let it retry.

2. If we still see DeadlineExceededErrors too often, talk with the service provider to discuss if the current Deadline value is appropriate.

[bradmiro]

Thanks Takashi, I'm currently investigating this at b/158098991.