Bigtable operation never completes after error in grpc._cython.cygrpc.plugin_get_metadata #3061
Description
I'm working with a user to debug an issue in the following code:
https://gist.github.com/garye/1c273182ff7a27881396cd12931236a4
Once in a while when this code runs, the read_rdc_item_data function seems to hang at the table.read_rows call on line 94. It does not seem to be related to a particular query and no issue is visible on the bigtable server side. It does not seem easily reproducible on demand. The potentially interesting logging around when the incident occurs is:
Exception socket.error: error(101, 'Network is unreachable') in 'grpc._cython.cygrpc.plugin_get_metadata' ignored
and also
Exception oauth2client.client.HttpAccessTokenRefreshError: HttpAccessTokenRefreshError('Invalid response 502.',) in 'grpc._cython.cygrpc.plugin_get_metadata’ ignored
Because of concerns around thread safety of the client a new one is created before each request. More efficient solutions are under consideration.