Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add retry on 429 error code #455

Merged
merged 3 commits into from
Sep 2, 2024
Merged

Conversation

hovaesco
Copy link
Member

@hovaesco hovaesco commented Mar 1, 2024

Description

Add retry on 429 error code

Non-technical explanation

Release notes

( ) This is not user-visible or docs only and no release notes are required.
( ) Release notes are required, please propose a release note for me.
( ) Release notes are required, with the following suggested text:

* Fix some things. ({issue}`issuenumber`)

@cla-bot cla-bot bot added the cla-signed label Mar 1, 2024
@hovaesco hovaesco requested review from aalbu and hashhar March 1, 2024 15:11
lambda response: getattr(response, "status_code", None)
in (502, 503, 504),
in (429, 502, 503, 504),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add any delays for the retry? Immediate retry will just make the problem worse (not sure if that's already built in)

Copy link
Member

@electrum electrum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn’t part of the Trino protocol. Before we add this here, it needs to be specified in the protocol, which will require more details. How long should the client wait? Is there a standard header for this?

@electrum
Copy link
Member

electrum commented Mar 1, 2024

This behavior needs to be specified in https://trino.io/docs/current/develop/client-protocol.html

There is a standard Retry-After header.

@hovaesco hovaesco self-assigned this Jul 1, 2024
@hovaesco hovaesco force-pushed the hovaesco/retry-429 branch from d18dae0 to dfa6840 Compare July 5, 2024 12:26
trino/client.py Outdated
@@ -887,7 +895,11 @@ def decorated(*args, **kwargs):
try:
result = func(*args, **kwargs)
if any(guard(result) for guard in conditions):
handle_retry.retry(func, args, kwargs, None, attempt)
if result.status_code == 429 and "Retry-After" in result.headers:
handle_retry_sleep = _RetryAfterSleep(result.headers.get("Retry-After"))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a method to convert header value to seconds. The header can be a point-in-time too. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After#syntax in which case this code will throw.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've added a method _parse_retry_after_header for parsing Retry-After header.

@hovaesco hovaesco force-pushed the hovaesco/retry-429 branch from dfa6840 to 872a6f4 Compare July 8, 2024 13:17
@hovaesco hovaesco force-pushed the hovaesco/retry-429 branch from 872a6f4 to 0e54f25 Compare July 8, 2024 13:19
Copy link
Member

@hashhar hashhar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM for the client.

As David said earlier we should document the expectation from other clients for HTTP 429 in the Trino client protocol docs. Please address that in a follow-up.

Copy link
Member

@hashhar hashhar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trino protocol docs were updated. So this is now good to merge.

@hashhar hashhar merged commit a875667 into trinodb:master Sep 2, 2024
12 checks passed
@hashhar hashhar mentioned this pull request Sep 12, 2024
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

4 participants