429 HTTP Client Error - may just need help understanding how to manage it

[synman]

Describe the bug
I'm experiencing an HTTP 429 response code issue with a rather large photos library I've had in Google Photos since the service originally launched. I've got indexed photos in this library going back to the late 1990s (I imported them at some point along the way).

I apologize in advance if this issue is redundant. I did take a look through closed issues / commits / etc and see this has come up a couple times in the past. #202

I started getting the below error about midway through 2019 processing (2020 and 2021 made it).

03-07 17:02:16 gphotos.GooglePhotosDownload INFO     downloading 218480 photos/2008/05/May 2008 140.JPG
03-07 17:02:16 gphotos.GooglePhotosDownload ERROR    FAILURE 143397 downloading photos/2008/05/May 2008 141.JPG - 429 Client Error: Too Many Requests for url: https://lh3.googleusercontent.com/lr/AFBm1_ZcGCDI3YIBGDiX_roNSvd2iQ1rPNag1Oe_ayFyaDYlFQebk_I0fDoH5Ph1zdW5hrykS38pYTLaBQ_BpOjtOrXhN7jEgV7C4QD8ER1aeuy1pzlhflmsj12e-lwuYXt2ORt_C7APZawiQVfgMc2PyhC2I7uKH0GEEmWFU7qYrXT3lXnzUSXlJQDL8stvK7JIeLHCDdCG_zvllgmhLdzBKvTNXjHRyEvBpIhAGockKz05G9UBcjIGCuUpO5vAGpW0PDiTk06OI0G_YCQ5EHlhwR8OPtivAM49LFr7DR4iIjc5E-d6HhXqs_IXNX5dXNTn5yNemKOEE5F27ZxoPVLMSv9umrUWGHicQmN6i6a6Ppx6HIlf1AZ5otFgJkbZqnu5B_qNsWohijfY2XkzUksQ0bOio__IQGk5X60g7_Ko3t4mh7XqaizAGVm-W1GFN5-NN9e0IpH2kfrm_IHioy6ZnldklxgqshA4yYRGHtfj0KRjLPERGb0LPtnBVq_j2aC6qegQx5Hk7tq735JmBxJaxo1-IV4CnZl388K-7Q91Jf9LIoL6FhvG61jqq8QQwHxUaYchUG2fzLx2rALm341PEtSs3ml9VK7QL4wkhfXvQqg48zNvoqTKF8s_nFYiUvW3nTk1hU-cNlpkTYJiQnOZ_O_vbEWuo04AwxQO7tP4CLwfD3uu5Ar93trSLKyCV6IATsx7UJ0z5RWKHlzBXV10dcaZ9dl0BfnWFq2uf7CxX9Q_dOXiRANG4Cw1qJFy7KG825dI2Cg_rqgb5sKPTL2j04TJrM-h0jFmlKK4ECINFPgGJ6xeEaJgiaFV9y-EU-tJ4_-QE825KVsD=d
03-07 17:02:16 gphotos.BadIds DEBUG    BAD ID AEarUuLpy9yy3UF__RWzSXxvNsUqRrQM1a2oAUJnL9fszkGzxuWZOA_I5adb-ALZCJQTlYt1L_SxkSJDEyRmQ9vN2xD1YEWRNw for photos/2008/05/May 2008 141.JPG
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/gphotos/GooglePhotosDownload.py", line 256, in do_download_file
    response.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://lh3.googleusercontent.com/lr/AFBm1_ZcGCDI3YIBGDiX_roNSvd2iQ1rPNag1Oe_ayFyaDYlFQebk_I0fDoH5Ph1zdW5hrykS38pYTLaBQ_BpOjtOrXhN7jEgV7C4QD8ER1aeuy1pzlhflmsj12e-lwuYXt2ORt_C7APZawiQVfgMc2PyhC2I7uKH0GEEmWFU7qYrXT3lXnzUSXlJQDL8stvK7JIeLHCDdCG_zvllgmhLdzBKvTNXjHRyEvBpIhAGockKz05G9UBcjIGCuUpO5vAGpW0PDiTk06OI0G_YCQ5EHlhwR8OPtivAM49LFr7DR4iIjc5E-d6HhXqs_IXNX5dXNTn5yNemKOEE5F27ZxoPVLMSv9umrUWGHicQmN6i6a6Ppx6HIlf1AZ5otFgJkbZqnu5B_qNsWohijfY2XkzUksQ0bOio__IQGk5X60g7_Ko3t4mh7XqaizAGVm-W1GFN5-NN9e0IpH2kfrm_IHioy6ZnldklxgqshA4yYRGHtfj0KRjLPERGb0LPtnBVq_j2aC6qegQx5Hk7tq735JmBxJaxo1-IV4CnZl388K-7Q91Jf9LIoL6FhvG61jqq8QQwHxUaYchUG2fzLx2rALm341PEtSs3ml9VK7QL4wkhfXvQqg48zNvoqTKF8s_nFYiUvW3nTk1hU-cNlpkTYJiQnOZ_O_vbEWuo04AwxQO7tP4CLwfD3uu5Ar93trSLKyCV6IATsx7UJ0z5RWKHlzBXV10dcaZ9dl0BfnWFq2uf7CxX9Q_dOXiRANG4Cw1qJFy7KG825dI2Cg_rqgb5sKPTL2j04TJrM-h0jFmlKK4ECINFPgGJ6xeEaJgiaFV9y-EU-tJ4_-QE825KVsD=d
03-07 17:02:16 gphotos.GooglePhotosDownload INFO     downloading 218481 photos/2008/05/May 2008 139.JPG
03-07 17:02:17 gphotos.GooglePhotosDownload ERROR    FAILURE 143398 downloading photos/2008/05/May 2008 140.JPG - 429 Client Error: Too Many Requests for url: https://lh3.googleusercontent.com/lr/AFBm1_abZkWfLhbyjizpCG0Qg0o4GpyCaf_Px1ACan2ZV2XDlgNy_NMt-K8HO5pwyXLyNvlXZ3-ekct-__a9PXKsuxh50emrGc_-q1qZXPItuqnqgoBa4gUmMCYVYL6_Zz8UNBbFEISwcGpyPVyfe19a8Nrvouw5-b_aPdp5ae1xri-DxxFlpBl9hFpmAnUbxdvwVmFW6Dx06nzJITgtJu-SzSFR1wn6nfEotcWtVWhoS-9yPOD0q2vqwSGqI2iHObqDXSlo_DBNZdLHGzOK3aWWDxX4pgg4kLZBsQYNcyBT7dhXLrf9RNVdjq2ngzSZU0v4uwXJK8XDgRdhmfizTSHKocWfM9_Yw3Si6PvwRAtLF_Y9wed05f3_TauFTEIrhbmfZ8GFH0WeeFD14y_wp7NWiRrfTGfgWOHlh19QGvIb_DSrYfIAFbDYbCaC_rRp1mu-wqJaWoBvHjF4joBNqLL03vjEpQG6ZXIpoAPkZUY8cAb6tuozgPokw3p7yF2CgNYlpDgccbFtVLFMEAGo6rOJk6pK-rtP77Dud8RfkT2aa4yombVx8z0D_6tCnItHVIzT7cSASJN27tNzAkPbY6O1zLA5N9B97mQ315QkW3gP_nBsYCWL8QDxaBHaIQjeDlW6XkUixvKWqlnDEWMGmDfI4C50C_jHbIzDqP5MxJAHyE1TsuPhPQfCaU31v0R6GYOiJeW-rEC2p5c95nWb1rnM3idbW9VRpxXe6Ot9Tz71c-gVAJFYWzuTstIoXVuNv0cv5A8PBy7erGlNbaTsMVX-6iovUfywXAOeVjb7wQqRWf13VU0DATeJ5DQxPWSAChd2zyW7-bZ3Z0rl=d
03-07 17:02:17 gphotos.BadIds DEBUG    BAD ID AEarUuIV6O0ttFkCSKt5A6ENCIiGla3B5Z73ARShbyWEUUkQ8CadX3JjQWq4FDTn8tbB0GDnsAasAavnNHvNd3rPkM615CNo4A for photos/2008/05/May 2008 140.JPG
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.7/site-packages/gphotos/GooglePhotosDownload.py", line 256, in do_download_file
    response.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

I'm guessing these failures are getting written to my gphotos.bad_ids.yaml file. Perhaps I'm mistaken.

What is the best course of action for me to take to work around this? Should I just let it keep running? Do I need to delete my bad_ids.yaml file at some point?

I imagine at some point I'll reach a steady state for this library and it'll settle down. This could be a bad assumption too as I'm still coming up to speed on the underlying issue.

To Reproduce
Simply run gphoto-sync on a big library and wait for Google to complain

[gilesknap]

Hello there. My library goes back to 1987 and is about 150GB and it downloads OK. However, if I were to guess at your issue it is that you have a very fast connection and you are exceeding the quotas on Google's API.

There are two quotas - a max calls per second and a max calls per 24 hours. The first of those is supposed to be handled in the code using exponential backoff. But if you exceed the 2nd I don't think this will work. I'm also unsure how effective the backoff code is since it is hard to test.

luckily, if this is your issue then you can slow things down using --max-threads. Choose a lower number than the default and see how you get on. It would also help to wait for 12 hours or so since you last got the error.

[synman]

I am capped at about 300 mb/s from my ISP. I'll take the container schedule offline, wait a day, and try again with the suggested max threads setting until I have the library synched.

My library is a bit over 700GB (has video).

[synman]

I deleted gphotos.bad_ids.yaml, set --max-threads to 5, and am rerunning the job now. Making quite a bit more progress today. I definitely fell into 24 hour block type situation.

[gilesknap]

OK this does not come up often - but I should probably document the issue for those with fast connections and big libraries.

[qrkourier]

I am running into this problem with a fast connection, large library, and limited to a single thread. I didn't find another way to throttle gphotos-sync to further reduce the likelihood of hitting the rate limit which symptom is HTTP 429 error response.

[gilesknap]

@qrkourier yes there is no other way to throttle. Do you think that is an issue? I'm not sure how fast a connection would need to be to hit this problem on a single thread but it must be more than 1GBps.

[qrkourier]

I'm operating at less than <=1Gbps and am hitting the rate limit and subsequent exception with one concurrent thread and will track progress in #320.