Retry-After amount

[felix-hilden]

On our Discord server there has been a discussion about the Retry-After headers Spotify is sending. Reportedly they don't match the actual retry amount or the header is missing altogether (possibly also on Spotify's tracker spotify/web-api#542 and spotify/web-api#1267).

I tried to reproduce it, and managed to get the unwanted repeating retries, but not the missing retry header. Here's the code injected to RetryingSender.

def send(self, request: Request) -> Response:
    if self.is_async:
        return self._async_send(request)

    tries = self.retries + 1
    delay_seconds = 1
    rate_limited = False    # Track whether the last request was also a retry

    while tries > 0:
        r = self.sender.send(request)

        if r.status_code == 429:
            seconds = r.headers['Retry-After']
            time.sleep(int(seconds))
            print(rate_limited, seconds)
            rate_limited = True
        elif r.status_code >= 500 and tries > 1:
            rate_limited = False
            tries -= 1
            time.sleep(delay_seconds)
            delay_seconds *= 2
        else:
            return r

By tracking whether the last request was also a retry, we can see that indeed almost half of the requests are retried again.

from tekore import Spotify, util
from tekore.sender import RetryingSender, PersistentSender
conf = util.config_from_environment(return_refresh=True)
token = util.refresh_user_token(conf[0], conf[1], conf[3])

spotify = Spotify(token, sender=RetryingSender(sender=PersistentSender()))

while True:
    spotify.track('4EPfPUkzXddsQu6mcimezc')

...
False 1
False 4
True 1
False 4
True 1
False 3
True 1
False 4
True 1
False 4
True 1
False 1
False 1
False 4
True 1

It also seems that the retry amount in the second request is always 1, and that the repeating retry occurs more often with longer retry times. The requests are never retried thrice. And when looking at a longer series of values, the amount of retries and the share of second retries seems to have a pattern.

secs | # retries | % 2nd retries
-----+-----------+--------------
  1  | 45        | 0.244
  2  | 8         | 0.500
  3  | 4         | 0.500
  4  | 62        | 1.000
  5  | 26        | 0.846

Everything works now because the second retries are performed, but by increasing the retry time by 1, the unnecessary retries are avoided entirely and we wait for less. For 145 retries, first retries took 450 seconds and second retries 100 seconds, but when adding the extra second the total was only 513 seconds.

---

So, given the possibility of missing headers and second retries, and the result of adding an extra second above, I think we should do two things:

• Make the retry longer by 1 second
• Have a default retry of 1 second