Fix 403 errors from crates.io

[yan12125]

No description provided.

[yan12125]

As a side note, from https://tools.ietf.org/html/rfc7231:

A user agent SHOULD send a User-Agent field in each request unless specifically configured not to do so.

[lilydjwg]

On Thu, Jul 04, 2019 at 07:29:58AM -0700, Chih-Hsuan Yen wrote: As a side note, from https://tools.ietf.org/html/rfc7231: > A user agent SHOULD send a User-Agent field in each request unless specifically configured not to do so.

This can be forwarded to tornado.

…

-- Best regards, lilydjwg

[yan12125]

This can be forwarded to tornado.

I agree.

UPDATE: Opened tornadoweb/tornado#2702

[yan12125]

AUR tests are still as flaky as usual :/

[lilydjwg]

I'd like to set the user agent in session so we don't need to set it per request. Also, the version can be added to the user agent string.

[yan12125]

I'd like to set the user agent in session so we don't need to set it per request. Also, the version can be added to the user agent string.

I'll check it out later.

By the way, do you think it's a good idea to elevate regex's user_agent to a gloabl option?

[lilydjwg]

A global user_agent is good but should be separate from the regex one. The regex one is to cope with sites that behave poorly with unexpected user agents (e.g. bilibili).

[yan12125]

Turns out global user_agent needs quite a few changes. I give up the idea this time.

[lilydjwg]

pytest-httpbin doesn't work with nightly. Also it brings in too many dependencies. I'll remove it.

[yan12125]

Well, I still think it's better not to test against the actual httpbin.org site.

By the way, the Python 3.8 issue is already fixed in git-master of werkzeug: pallets/werkzeug#1551.