Pubsub 0.37.2: Observed non-recoverable stream error 400

[ghost]

Hello! Firstly, thank you for this awesome library!

I am running around 40 pubsub client workers in an EKS cluster. After more than a day of running uninterrupted, one of my pubsub workers observed a non-recoverable error. (The rest are still running fine)

OS - ubuntu:16.04
Python version - Python 3.5.2
Pubsub version - google-cloud-pubsub==0.37.2

Stack Trace:

INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 504 Deadline Exceeded
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed recoverable stream error 504 Deadline Exceeded
INFO:google.cloud.pubsub_v1.subscriber._protocol.bidi:Re-established stream
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:Observed non-recoverable stream error 400 Invalid arguments provided: the number of ack ids to modify must be equal to the number of ack deadlines.
INFO:google.cloud.pubsub_v1.subscriber._protocol.streaming_pull_manager:RPC termination has signaled streaming pull manager shutdown.
INFO:google.cloud.pubsub_v1.subscriber._protocol.bidi:Thread-ConsumeBidirectionalStream exiting
INFO:google.cloud.pubsub_v1.subscriber._protocol.leaser:Thread-LeaseMaintainer exiting.
INFO:google.cloud.pubsub_v1.subscriber._protocol.heartbeater:Thread-Heartbeater exiting.

Code snippet

flow_control = pubsub_v1.types.FlowControl(max_messages=500)
subscription = sub_client.subscribe("pubsub_subscription", process_message, flow_control=flow_control)

Where process_message batches messages for a few minutes and acks them all when their payloads have been inserted into a database.

Any help would be appreciated here!