HTTP error 403: forbidden

[mesfff]

ran into this issue when i tried to get this working, couldnt find a fix for it in the readme file so thought id ask here

403

[avalyra]

also having this issue

edit: setting useragent to this exactly worked:

useragent = Mozilla/5.0

found in reply to #1239

[Galileo007]

edit: setting useragent to this exactly worked:

useragent = Mozilla/5.0

This worked for me, thanks.

[photonometric]

Thanks for making an issue that actually has the error name in the title! x3 Made it easy to find the solution quickly.

I just updated to v20230105 [commit 5d0b370 (Feb 4)] and got a long red HTTPError 403: Forbidden. Confirmed session cookies were still good.

Changing the useragent line in config.ini to useragent = Mozilla/5.0 fixed the 403 errors instantly, in all recent versions.

Here's the 403 error log from the 20230105 versions

Trying to log in with saved cookie 
Error at doLogin(): (<class 'mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper'>, <httperror_seek_wrapper (urllib.error.HTTPError instance) at 0x7f871779fb40 whose wrapped object = <HTTPError 403: 'Forbidden'>>, <traceback object at 0x7f87179e2940>) 
Traceback (most recent call last):
File "PixivUtil2.py", line 1524, in doLogin
  result = __br__.loginUsingCookie()
File "/home/minty/PixivUtil2-master/PixivBrowserFactory.py", line 347, in loginUsingCookie
  res = self.open_with_retry('https://www.pixiv.net/en')  # + self._locale)
File "/home/minty/PixivUtil2-master/PixivBrowserFactory.py", line 207, in open_with_retry
  res = self.open(url, data, timeout)
File "/usr/local/lib/python3.8/dist-packages/mechanize/_mechanize.py", line 257, in open
  return self._mech_open(url_or_request, data, timeout=timeout)
File "/usr/local/lib/python3.8/dist-packages/mechanize/_mechanize.py", line 313, in _mech_open
  raise response
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 403: Forbidden

Cannot Login! 

Just for diagnostics, I tried changing my VPN (JPN nodes) and also going back to previous install [v20221029 commit 3771035 (Nov 25)]. Still got 403, but the earlier version gave a shorter error output. Changing to useragent = Mozilla/5.0 fixed the login even in this older version.

Login error v20221029 + 3771035 (Nov 25)

Error at doLogin(): (<class 'mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper'>, <httperror_seek_wrapper (urllib.error.HTTPError instance) at 0x7f08d0a1d740 whose wrapped object = <HTTPError 403: 'Forbidden'>>, <traceback object at 0x7f08d09fc680>) 
Cannot Login! 

Others have had this issue starting in December, though I have had no problems until today. This problem is due apparently to how Cloudflare verifies "safe" users using the useragent (web browser ID field); it is a mystery why it comes and goes, but has been an issue for similar downloading programs like gallery-dl as well. See linked threads in the next post for more (confusing) info, if desired.

Note this 403 error looks similar to other red text login errors involving username/password or PHPSESSID session cookies. These usually say something like HTTPError 302: 'The HTTP server returned a redirect error that would lead to an infinite loop. The last 30x error message was Found. This is a user error, see Wiki: Cookie Login for clear instructions.

Issue #814 is a (very long) discussion of these more "normal" login problems, and how they were solved 2-3 years ago (which led to the diagnosis of this related problem)

[photonometric]

Some comments and suggestions regarding this issue: This is the first time changing the user agent has worked for me; modifying the long useragent (OS/rev/FF version) helped some people fix OAuth back in Oct 2020, but mine was an API procedural fix at that time. But I guess at this point, it is worth it for everyone to change to this simplified Mozilla/5.0 ID, to "vaccinate" against the seemingly random periods when Cloudflare causes problems with more verbose agent IDs.

There have been quite a few threads about this issue recently [good discussions in #1209 and #1239 | simpler troubleshooting in #1240 #1241 #1242]. Maybe it would be a good idea to update documentation? It would help users to be able to look up the HTTPError 403: Forbidden text (maybe with an explanation of the difference to cookie errors 301/302), and get a clear statement that the solution is simply to change useragent = Mozilla/5.0. Or better yet, a notice that everyone should change to this proactively, since the Cloudflare issue has an excellent chance of affecting most/all users at some point.

This kind of notice would be helpfully visible in some of these places perhaps?

• the readme.md
• GitHub landing (perhaps at top if this is ongoing (really just contents of readme.md)
• Wiki
• Release Notes (even if the fix isn't part of the latest Jan 05 release, per se),
• pinned issue. Actually this issue thread is the first one that states the error and the solution in a clear way, without requiring reading a small maze of other related issues and potential/deprecated fixes.
• A special version release? With corrected default useragent in auto-generated config.ini. It is currently: useragent = Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36

Sorry if these suggestions sound a bit drastic or pushy ^^;; [I'm working on using Git on my linux VM for editing documentation at least, but I'm still a ways from confident in such tasks] I make these suggestions humbly, because in my view (as a user for about a decade), this is the most difficult and frustrating species of error for normal users to deal with:

• a) It appears and then is randomly "fixed" (except not), based entirely on opaque server-side/cloud security conditions, making basic identification and understanding extremely problematic
• b) it closely resembles both "normal" login/PHPSESSID user errors (which are still quite confusing to identify and navigate correctly, even for experienced users)—as well as the more code/API-related OAuth issues from the past 5 years. These similar issues lead users to attempt familiar troubleshooting actions (or inaction) which do not apply here
• c) codebase troubleshooting (even without touching the code) requires a high degree of experience with GitHub and PixivUtil2, as well as experience navigating potentially conflicting/unrelated historical issue threads

So I think this is really the kind of issue—like the various cookie and API problems before it—which requires this kind of action toward user notification/instruction.

[Nandaka]

pinned the issue for now using #1239

I don't have free time due to real time issues.

[XaviMH]

As already commented by @photonometric (in this issue, #1245 (comment)), and @Nanoka (in a similar issue, #1239 (comment)), I'd like to address that changing the user-agent to a simple Mozilla 5.0 solved my (very recent) HTTP Error: 403 Forbidden problem as well (which proved to be difficult to assess for me):

[Network]
useProxy = False
proxyAddress = 
useragent = Mozilla/5.0   <----------
useRobots = True
timeout = 60
retry = 3
retryWait = 5
downloadDelay = 5
checkNewVersion = True
notifyBetaVersion = True
openNewVersion = True
enableSSLVerification = True

[amarney4862]

Thanks for this, the program didn't start without this fix so it should probably have a place in the readme.

[GGrant3]

Couldn't for the life of me understand why it wouldn't authenticate. You just cured me of a lot of stress. It worked in the past... so weird.