504 deadline exceeded

[kthulsidoss]

Hi,

Greetings. Way back i had reported 504 deadlines exceeded issue and we are trying with the synchronous call.Now that the issue has surfaced in the customer environment when we are trying to test the pub-sub mechanism. I have glanced the possible solutions and found an entry here :
#9 -->" I found the change that has been done in PR 9279 and it's related to initial_rpc_timeout_millis parameter of messaging in pubsub/google/cloud/pubsub_v1/gapic/subscriber_client_config.py file." it seems to me to over come this issue either use google pub sub 1.0.0 or change the parameter mentioned in subscriber_clinet_config.py.
Could you please let me know how do i
1.uninstall google pub sub google_cloud_pubsub-1.0.2 and make use of google_cloud_pubsub-1.0.0.
or
how to find the installed folder in GCP and locate this subscriber_client_config.py in the installed directories ?
some details
OS :Red Hat 4.8.5-39
Python : 3.6
Google pub sub :google_cloud_pubsub-1.0.2

[kthulsidoss]

@plamut can you help in finding these files in GCP ?

[plamut]

Until the RPC timeout is reverted back to the old value, an older version of the PubSub library can be installed by explicitly specifying that version. For example, once can run pip on the target machine:

$ pip install google-cloud-pubsub==1.0.0

This will automatically uninstall any existing pubsub version, and replace it with pubsub version 1.0.0. The library gets installed into the target Python's site-packages directory, e.g. for Python 3.7:

<python_dir>/lib/python3.7/site-packages/google/cloud/pubsub_v1/

Does your application work normally with PubSub v1.0.0?

[kthulsidoss]

@plamut after setting the opt/secadm/xyzecadm/env/lib/python3.6/site-packages/google/cloud/pubsub_v1/gapic/subscriber_client_config.py as mentioned in the support channel i dont see the 504 exception how ever i see that i get a message
google.api_core.exceptions.InvalidArgument: 400 You have not specified an ack ID in the request.

[plamut]

@kthulsidoss Could this be a different issue that just surfaced after fixing the first one? A missing ACK ID in the request does not seem directly related to the timeout that was previously occurring.

Does the issue also occur if trying the out-of-the-box client version 1.0.0?

[kthulsidoss]

@plamut it is blaffling to me , can you please list out the steps to check for the basic flow is set up properly like the topic & subscription.
No i did not change the google pub sub and using the same one : google_cloud_pubsub-1.0.2

[kthulsidoss]

@plamut here is the exact log:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INVALID_ARGUMENT
details = "You have not specified an ack ID in the request."
debug_error_string = "{"created":"@1584355980.626180381","description":"Error received from peer ipv4:172.217.27.202:443","file":"src/core/lib/surface/call.cc","file_l
ine":1056,"grpc_message":"You have not specified an ack ID in the request.","grpc_status":3}"

[kthulsidoss]

@plamut there were code that was over written and hence this spurious issue has occured.

[plamut]

@kthulsidoss In order to install a different pubsub version to try it out (the timeout reportedly does not occur prior to 1.0.1), the following command can be run from the console:

$ python -m pip install google-cloud-pubsub==1.0.0

(depending on the environment, the command might have to be run with superuser privileges)

This might be the fastest way to get around the reported timeout issue, at least until it gets fixed in one of the future versions.

---

As for the general steps, the quickstart from the docs can be helpful.

Assuming the project, the account, etc. have all been set up successfully, here are the steps for receiving the first message:

• A topic needs to be created (code sample)
• A subscription to that topic needs to be created (code sample)
• A couple of messages should be published to the topic (code sample)
• A subscriber can then pull these messages, either asynchronously (recommended by default), or synchronously. A synchronous pull code sample: link.

[kthulsidoss]

@plamut now that the 504 deadline issue is no longer seen after making the following changes
i changed to opt//env/lib/python3.6/site-packages/google/cloud/pubsub_v1/gapic/subscriber_client_config.py

initial_rpc_timeout_millis parameter of messaging in pubsub/google/cloud/pubsub_v1/gapic/subscriber_client-config.py chaging to 25000 .

How ever the API is throwing the response as 502 bad gateway . Any help to overcome this issue and obtain http 200 response ?
i see from google that increasing the buffer size "Nginx 502 Bad Gateway. Solved by increasing buffer." solved the issue ? would that help in our case too ?

[plamut]

@kthulsidoss Are 502 errors only temporary, or do they happen for all requests?

You might try with increasing the buffer sizes, yes, and examining the logs to see why the error occurs (is your application behind nginx too busy perhaps?).

It's hard to tell without a reproducible code sample, would it be possible to provide one? The error might be caused by something outside of the PubSub client library itself.

[kthulsidoss]

@plamut i see the threads on this particular issue :
1.https://stackoverflow.com/questions/19539501/nginx-502-bad-gateway-solved-by-increasing-buffer-why?rq=1
2.https://stackoverflow.com/questions/30470448/nginx-uwsgi-websockets-502-bad-gateway-upstream-prematurely-closed-connection-wh
so these parameters were closely looked into but nothing applied to make them go away however i made the no of messages to 1 in the code
NUM_MESSAGES = 1 , this helped in making the 502 error (bad gateway ) gone.
Can you please let us know how this is helping.

[plamut]

@kthulsidoss It's hard to tell without a code sample or some other details, as the issue could very well be be lurking somewhere outside of the client library. Could you please provide one? And how is the customer environment configured, is nginx indeed connected to the backend correctly? Does a request reach it?

Out of the box I was not able to reproduce the 502 error, unfortunately.

[plamut]

Fixed here by reverting the timeout to the previous value, closing this.