[Bug]: "Match All Authors" with a big library will hit rate limits on Audible API and start getting 429s

[FreedomBen]

Describe the issue

Expected behaviour
Ideally the requests would be queued server-side to throttle them to avoid exceeding the rate limit threshold

Actual behaviour
All lookups start to fail with 429s from Audible API. If your library is big enough, you may even get an IP ban

Steps to reproduce the issue

1. Load hundreds of books from different authors into a library
2. Visit the "authors" page
3. Click "Match All Authors"
4. Observe backend 429 errors from Audible API after several dozen lookups

Audiobookshelf version

v2.2.4. I can update and re-test in a little while but for the next week or so I need it to be perfectly stable as I'm traveling

How are you running audiobookshelf?

Docker

[FreedomBen]

Would a client-side fix in client/components/app/BookShelfToolbar.vue method matchAllAuthors() to add a small delay between POST requests be a reasonable (and acceptable for a PR) place to implement this?

Given that the endpoint is currently synchronous, it seems like a much bigger (and contract breaking) effort to turn it async and update the client accordingly, but I'm quite unfamiliar with this codebase so maybe there's something with the web socket that could be done without too much effort.

[advplyr]

Yeah I think a client-side fix is best here for the reason you pointed out. PR would be great.

We can do it server side with web socket events but probably more effort then it is worth at this stage. We're working on a new model to handle long running tasks so in the future we could make a match all authors route.

[advplyr]

One issue that just occurred to me is if the user navigates while the loop is running

[sebclark]

Perhaps a second button "Search Missing Authors". To lower the search volume.

[Dr-Blank]

To lower the api calls further, maybe implement a maintenance task to search only the newly added authors, for e.g after a library scan, queue an author's update scan if new authors were added.

It will be high volume for the initial scan, but only searching for authors that have been added recently, probably after the last update, should only be queried, and also only authors not previously match in case they got an update. Similar to scan and force re-scan feature separation in case of library.

And I do not know how the code structure is right now, but shouldn't this make more sense as a server-side feature instead of client side? I mean server is responsible for maintaining the db of books, scanning and updating, why not authors too, just send an update call to server and let it handle. For larger libraries, you have to keep the tab open, also it uses a lot of resources, not to mention all the notifications of foo author found, bar author not found freezing up my system briefly every now and then.

[TheSylus]

Any news on this? My library is pretty large so doing a rescan is already pretty impossible.

[ZLoth]

Describe the issue

In some of my AudioBookShelf library, I have hundreds of authors. When I attempt to do an author match using Authors → Match All Authors, I get a match on some authors, but errors on others, including well-known authors. A rescan results in some of those missed authors being matched up. When I review the servers logs, I see the following:

2024-05-05 12:44:57.626 - INFO - [Audnexus] Searching for author "https://api.audnex.us/authors/B004AIN4W4?region=us"
2024-05-05 12:44:57.705 - ERROR - [Audnexus] Author request failed for B004AIN4W4 AxiosError: Request failed with status code 429
2024-05-05 12:44:57.730 - INFO - [Audnexus] Searching for author "https://api.audnex.us/authors?name=David+Javerbaum&region=us"
2024-05-05 12:44:57.821 - ERROR - [Audnexus] Author ASIN request failed for David Javerbaum AxiosError: Request failed with status code 429
2024-05-05 12:44:58.060 - DEBUG - [Audnexus] Looking up author by name Day Keene
2024-05-05 12:44:58.061 - INFO - [Audnexus] Searching for author "https://api.audnex.us/authors?name=Day+Keene&region=us"
2024-05-05 12:44:58.133 - ERROR - [Audnexus] Author ASIN request failed for Day Keene AxiosError: Request failed with status code 429

This tells me that AudioBookShelf is sending too many requests to api.audnex.us within a short period of time and needs to slow down and be rate limited to maybe one request every .25 to .5 seconds. Having author's being matched up is not a time-critical function, and waiting several minutes for the matching to complete should not be an unexpected condition. It may warrant a warning message when the administrator clicks on the Match All Authors that it may take some time to complete.

Screenshot from logs screen:

Steps to reproduce the issue

1. Go to a library with large number of authors
2. Go to Authors
3. Click on March All Authors
4. Wait until red error messages appear.
5. Go to Settings → Logs
6. Filter for Warn-level messages

Audiobookshelf version

2.9.0

How are you running audiobookshelf?

Docker

[ZLoth]

Thank you. Looking forward to the release! @advplyr

[advplyr]

Fixed in v2.11.0

[ZLoth]

Thank you!

[Death916]

Not sure if this is fully fixed. Ran a match today after updating to 2.16 and got rate limited. Getting too many requewsts in the docker logs
` url: 'https://openlibrary.org//works/OL19340214W.json',

  data: undefined

},

request: <ref *1> ClientRequest {

  _events: [Object: null prototype],

  _eventsCount: 7,

  _maxListeners: undefined,

  outputData: [],

  outputSize: 0,

  writable: true,

  destroyed: true,

  _last: true,

  chunkedEncoding: false,

  shouldKeepAlive: true,

  maxRequestsOnConnectionReached: false,

  _defaultKeepAlive: true,

  useChunkedEncodingByDefault: false,

  sendDate: false,

  _removedConnection: false,

  _removedContLen: false,

  _removedTE: false,

  strictContentLength: false,

  _contentLength: 0,

  _hasBody: true,

  _trailer: '',

  finished: true,

  _headerSent: true,

  _closed: true,

  socket: [TLSSocket],

  _header: 'GET /works/OL19340214W.json HTTP/1.1\r\n' +

    'Accept: application/json, text/plain, */*\r\n' +

    'User-Agent: axios/0.27.2\r\n' +

    'Host: openlibrary.org\r\n' +

    'Connection: keep-alive\r\n' +

    '\r\n',

  _keepAliveTimeout: 0,

  _onPendingData: [Function: nop],

  agent: [Agent],

  socketPath: undefined,

  method: 'GET',

  maxHeaderSize: undefined,

  insecureHTTPParser: undefined,

  joinDuplicateHeaders: undefined,

  path: '/works/OL19340214W.json',

  _ended: true,

  res: [IncomingMessage],

  aborted: false,

  timeoutCb: null,

  upgradeOrConnect: false,

  parser: null,

  maxHeadersCount: null,

  reusedSocket: true,

  host: 'openlibrary.org',

  protocol: 'https:',

  _redirectable: [Writable],

  [Symbol(shapeMode)]: false,

  [Symbol(kCapture)]: false,

  [Symbol(kBytesWritten)]: 0,

  [Symbol(kNeedDrain)]: false,

  [Symbol(corked)]: 0,

  [Symbol(kOutHeaders)]: [Object: null prototype],

  [Symbol(errored)]: null,

  [Symbol(kHighWaterMark)]: 16384,

  [Symbol(kRejectNonStandardBodyWrites)]: false,

  [Symbol(kUniqueHeaders)]: null

},

data: {

  status: 429,

  message: 'Too Many Requests. Consider using https://openlibrary.org/developers/dumps.'

}

}

}`

[advplyr]

Open library isn't used for matching authors so that is unrelated to this