Add user agent string to feed requests #3099
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hey folks, I work on Pinecast (https://pinecast.com). We recently had a customer write in that their audiobookshelf server stopped updating feeds from us. It turns out that we'd seen a huge uptick in requests from the default Axios user agent string, which caused requests with that user agent to get served a captcha. My suspicion is that companies have started scraping more aggressively.
Podcast apps have (by and large) done a good job of providing descriptive
User-Agent
headers. Identifying your app allows podcasters to get informed analytics data and helps hosting companies like mine to identify and fix problems more easily.In this PR, I'm adding the
user-agent
HTTP request header. I've chosen a value that is fairly straightforward: the name of the project, a link to the project, andlike iTMS
to indicate that the client should be treated similarly to Apple's podcast crawler. I do not have strong feelings about this UA string, but I'll suggest that if you decide to go with something else, you avoid adding a version number. There's a balance between user privacy (less information) and transparency (more descriptive information) and compatibility (more information that looks like another user agent).This PR does not update the client calls to download media. I am not too familiar with the project and do not want to regress any features. However, I strongly suspect this function will need a similar update:
https://github.com/advplyr/audiobookshelf/blob/master/server/utils/fileUtils.js#L254
I'm glad to make that change as well, if you think it's appropriate.