Login/Download Issues see here (Oct-20)

[NHOrus]

Edited by @Nandaka: Please follow this step first

https://github.com/Nandaka/PixivUtil2/wiki#pixiv-login-using-cookie

https://github.com/Nandaka/PixivUtil2/wiki#fanbox-login-using-cookie

Prerequisites

• Did you read FAQ section?
• Did you test with the latest releases or commit ?

Description

Trying to start PixivUtil to get pictures, connection fails

Steps to Reproduce

Start up configured PixivUtil2

Expected behavior:

Program starts normally, downloads work

Actual behavior:

Cannot Login!
press enter to exit

2020-10-14 09:01:16,420 - PixivUtil20201009-beta1 - INFO - ###############################################################
2020-10-14 09:01:16,420 - PixivUtil20201009-beta1 - INFO - Starting with argument: [E:\conf\pixiv\pixivutil\PixivUtil2.exe].
2020-10-14 09:01:16,423 - PixivUtil20201009-beta1 - INFO - Setting log level to: DEBUG
2020-10-14 09:01:16,423 - PixivUtil20201009-beta1 - INFO - No default cookie jar available, creating... 
2020-10-14 09:01:16,565 - PixivUtil20201009-beta1 - INFO - Using default DB Path: E:\conf\pixiv\pixivutil\db.sqlite
2020-10-14 09:01:16,567 - PixivUtil20201009-beta1 - INFO - Using Username: nhorus
2020-10-14 09:01:16,567 - PixivUtil20201009-beta1 - INFO - Trying to log in with saved cookie
2020-10-14 09:01:16,765 - PixivUtil20201009-beta1 - ERROR - Error at doLogin(): (<class 'mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper'>, <httperror_seek_wrapper (urllib.error.HTTPError instance) at 0x41b9fa0 whose wrapped object = <HTTPError 403: 'Forbidden'>>, <traceback object at 0x041E3148>)
2020-10-14 09:01:16,767 - PixivUtil20201009-beta1 - ERROR - Traceback (most recent call last):
  File "PixivUtil2.py", line 1077, in doLogin
    last_image_id = -1
  File "PixivBrowserFactory.pyc", line 317, in loginUsingCookie
  File "PixivBrowserFactory.pyc", line 206, in open_with_retry
  File "mechanize\_mechanize.pyc", line 253, in open
  File "mechanize\_mechanize.pyc", line 309, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 403: Forbidden

2020-10-14 09:01:16,767 - PixivUtil20201009-beta1 - ERROR - Cannot Login!
2020-10-14 09:01:16,768 - PixivUtil20201009-beta1 - ERROR - Traceback (most recent call last):
  File "PixivUtil2.py", line 1077, in doLogin
    last_image_id = -1
  File "PixivBrowserFactory.pyc", line 317, in loginUsingCookie
  File "PixivBrowserFactory.pyc", line 206, in open_with_retry
  File "mechanize\_mechanize.pyc", line 253, in open
  File "mechanize\_mechanize.pyc", line 309, in _mech_open
mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper: HTTP Error 403: Forbidden

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "PixivUtil2.py", line 1278, in main
    parsed_page = BeautifulSoup(page.read().decode("utf-8"), features="html5lib")
  File "PixivUtil2.py", line 1084, in doLogin
    raise
PixivException.PixivException: 0 Cannot Login!, hasDumpPage=N, pageData=None

Versions

You can get this information from executing PixivUtil2.py --help.
Latest version available in https://github.com/Nandaka/PixivUtil2/releases

[RighteousAnger]

I just got this recently also, from the latest PixivDownloader

Error at doLogin(): (<class 'mechanize._response.get_seek_wrapper_class..httperror_seek_wrapper'>, <httperror_seek_wrapper (urllib.error.HTTPError instance) at 0x4282d60 whose wrapped object = <HTTPError 403: 'Forbidden'>>, <traceback object at 0x04292A68>)
Cannot Login!

[gnarf1975]

I have the same problem

Error at doLogin(): (<class 'mechanize._response.get_seek_wrapper_class..httperror_seek_wrapper'>, <httperror_seek_wrapper (urllib.error.HTTPError instance) at 0x3835bb0 whose wrapped object = <HTTPError 403: 'Forbidden'>>, <traceback object at 0x03864968>)
Cannot Login!

[kody180]

Same problem here too.

Error at doLogin(): (<class 'mechanize._response.get_seek_wrapper_class..httperror_seek_wrapper'>, <httperror_seek_wrapper (urllib.error.HTTPError instance) at 0x3b60778 whose wrapped object = <HTTPError 403: 'Forbidden'>>, <traceback object at 0x03B6F0C8>)
Cannot Login!

[Nandaka]

Maybe because of this, Pixiv want you to change your password?

Also, update your cookie again, looks like the format changed.

Works fine after changing my password and updating the cookie.

[polujm9]

Same problem.

I had tried change PHPSESSID and not working.

[Nandaka]

@polujm9 try to update your password in pixiv website and try to match the useragent value with your browser.

[jwshields]

Changing my password on the website and updating my config with the new PW, PHPSESSID, and UA appear to have fixed this for me.
Edit: FWIW, I did not receive anything from Pixiv that would have prompted me/suggested that I should change my password.

[polujm9]

Changing Password works for me.

[Tarried]

Changing password, changing session, none of that worked for me. I ended up creating a new account, and that worked just fine.

Interestingly the new session ID was shorter since it wasn't preceded by '10XXXXXX_'.

[todesschnitzel]

Same error here too. I changed my PW, PHPSESSID and User Agent, but i still get the error:

Error at doLogin(): (<class 'mechanize._response.get_seek_wrapper_class..httperror_seek_wrapper'>, <httperror_seek_wrapper (urllib.error.HTTPError instance) at 0x3bf03d0 whose wrapped object = <HTTPError 403: 'Forbidden'>>, <traceback object at 0x03BE95C8>)
Cannot Login!

I also gone through this: #505

Anything else i can try?

[skyvory]

I logged in and the prompt to change password appears, so I changed with that and updated the cookie (which formatted differently) and it works again.
Maybe try to logout and login back and see if the prompt appears. Perhaps a simple password change if you're already logged in wouldn't work. Can't guarantee though as I didn't try it myself.

[pxssy]

Did not get prompt to change password. Changed nonetheless, new cookie, does not work. :(

[ItaloKnox]

Didn't get the prompt, but changing the password worked. Added the new cookie and password to the config and it worked just as always. Back to pulling from multiple instances at once with no issues.

[fanboy01]

I have the same problem like everyone above.

From the looks of it, it seems the solutions seem to be hit or miss,

Unfortunately, none of the solutions worked for me.

Tried changing password and updating the cookie and user agent - no luck

Hell, I tried making a new account and that didn't work either.

[Hamuko]

I also got it to work by changing my Pixiv password and then updating my configuration with the new password, new cookie and current user agent.

[Baramanga]

I got the same problem:
Error at doLogin(): (<class 'mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper'>, <httperror_seek_wrapper (urllib.error.HTTPError instance) at 0x3e7b3d0 whose wrapped object = <HTTPError 403: 'Forbidden'>>, <traceback object at 0x03E81AC8>) Cannot Login!
and changing password doesn't work

[ee092884]

I currently test out a way to download in full mode normally
When you change the password and play this screen, use F12 to directly view the new PHPSESSID
Can change the same password
Don’t go to the pixiv homepage to check the PHPSESSID
Will not display the number of ID "XXXXXXX_" but only the command line after _
EDIT: also check your browser user-agent to match with config.ini

Current solution

[viravera]

Same issue. I've done the following:

logged out and back in. No prompt to change password.
Updated cookie/useragent
Changed password
Updated cookie/useragent again
no luck; problem persists.

The error I get is the same as the others:
Error at doLogin(): (<class 'mechanize._response.get_seek_wrapper_class.<locals>.httperror_seek_wrapper'>, <httperror_seek_wrapper (urllib.error.HTTPError instance) at 0x7f8fa3adae18 whose wrapped object = <HTTPError 403: 'Forbidden'>>, <traceback object at 0x7f8fa3929348>) Cannot Login!

[DisasterInbound]

I'm adding my comment here with the same issue and the previous steps didn't work:

-Logged out and back in (I was not prompted to make any password change)

• Changed password
• Logged out and back in
• Copy new cookie
  -Updated user agent

i don't want to create a new account so I didn't do that.

Error code:
'Error at doLogin(): (<class 'mechanize._response.get_seek_wrapper_class..httperror_seek_wrapper'>, <httperror_seek_wrapper (urllib.error.HTTPError instance) at 0x3717a48 whose wrapped object = <HTTPError 403: 'Forbidden'>>, <traceback object at 0x03734768>)
Cannot Login!'

[bluerthanever]

I managed to save the page retrieved in doLogin and it turns out to be as the picture above shows.
Looks like they put more effort in anti-scraping.

[bluerthanever]

it seems to be cloudflare.... geez.
I hate cloudflare.

[aksskl]

@polujm9 try to update your password in pixiv website and try to match the useragent value with your browser.

Worked for me using v20201009-beta1 and I used Chrome 86.0.4240.75 to get my PHPSESSID value

First I changed my password and PHPSESSID cookie value to see if that alone would work. It did not.

Then used this site (https://www.whatismybrowser.com/detect/what-is-my-user-agent) to detect my useragent and replaced whatever was in the config.ini WORKED

[RighteousAnger]

@polujm9 try to update your password in pixiv website and try to match the useragent value with your browser.

Worked for me using v20201009-beta1 and I used Chrome 86.0.4240.75 to get my PHPSESSID value

First I changed my password and PHPSESSID cookie value to see if that alone would work. It did not.

Then used this site (https://www.whatismybrowser.com/detect/what-is-my-user-agent) to detect my useragent and replaced whatever was in the config.ini WORKED

Hit or miss for sure...
using v20201009-beta1 and I used Chrome 86.0.4240.75 (Latest)
I go to Pixiv website to change password with no prompt encountered
Put in my new password along with the new cookie using Chrome in the config and logged out
Headed over to detected my useragent and copy what was there and replaced in the config
Open up the downloader and still same error results...I also tried on two different accounts, no avail. (Giving me the same error) Something must've changed recently no...?

[Kotters]

I managed to get it working again. I had no prompt to change my password, but doing so provided me with a new PHPSESSID that is in a new format. The old format had 6 digits and an underscore prepended (######_RestOfID). The new format does not.

Note that I had previously attempted setting a matching useragent and logging out and in to generate a new PHPSESSID. Before I changed my password, it simply gave me a new PHPSESSID in the old format, which did not work.

[maxiy01]

Changed password, updated cookie, logged out. Problem persists.

[AgentThirteen]

Keeping the SESSID unique auth token (the part that is after the underscore) and removing your user ID as well as the underscore will allow you to log in just fine, for now at least. You'll still have to manually provide your user ID to PixivUtil for this to work. I suspect this is part of #807 as user IDs classify as trackable information but @bluerthanever raised a valid concern with the verification process. pixiv probably didn't even think ahead nor test it at all as usual.

I cannot guarantee this will help solve the issue as I tried both a rather old release and the latest one, both run into #817 when trying to download something. Nothing seems to generate the new format or any prompt for that matter, no matter the browser, method or password complexity so for now stuck with this like everyone else encountering it.

If someone manages to grab a link to the password change prompt, maybe that would also force reformat the stored cookie instead of pushing the ID to force a manual check which is what seems to be happening.

[bluerthanever]

Ok. I didn't do any test yesterday. But here is what I did, and I have no problem for now.

1. Deleted all cookie in browser with extension, not logging out (not sure if there are any differences)
2. Logged in again in browser, somehow this time they asked for recaptcha, and password update.
3. Updated the new PHPSESSID in config.in and viola.

Edit:
Didn't try out any of the menu items yet. And the new PHPSESSID seems to be in new format as well.

[afterdelight]

For firefox:
https://user-images.githubusercontent.com/836867/140603623-18fcd9af-e7a1-4952-a494-cbf3cdfb59a7.png
PAssword fill with anything. it will login using ur PHPSESSID.

[izzyseo]

I'm having this issue too. I'm on Mac tho.

[afterdelight]

use gallery-dl, its better

[biggestsonicfan]

@userskill The usage is in the FAQ of the Readme, Question# 3.

@afterdelight Your comment here is not helpful. While gallery-dl supports pixiv, for content marked as 18+ gallery-dl will still want login credentials. The refresh_token is still required by gallery-dl in this circumstance.

[afterdelight]

here, read this how to login to pixiv with gallery-dl. i havent have a successful attempt with pixivutil2

mikf/gallery-dl#1304 (comment)

[jwshields]

I know I commented early on in this thread saying that updating my password worked for me, but I seem to be having troubles again.
Reading through the entire thread, I've changed my password about 15 times now. Each time I'm getting a PHPSESSID in the form of ######_xxxxxxxxxxxx.
I've tried logging out and back in for a clean session, I've cleared my cookies, I've tried logging in with a separate browser... But still can't seem to log in using the cookie. CloudFlare is giving me a 403. Haven't seen any prompt for a captcha or anything in my browsing/attempts to log in.
Am I missing something obvious here?

[biggestsonicfan]

I've changed my password about 15 times now

Why was this necessary?

[jwshields]

I've changed my password about 15 times now

Why was this necessary?

Having performed the steps in #814 (comment), on the page in the screenshot, I am still getting the incorrect cookie. Also, the comments right below this linked one state that multiple changes may be needed in order to get the correct format of cookie/PHPSESSID.

And performing the steps in https://github.com/Nandaka/PixivUtil2#faqs Q3 do not result in success either...
Hence my asking if I am missing something, since I know this issue ticket has been open for a long time and lots of people have struggled with it, and it seems Pixiv has changed formats/methods a few times throughout the thread...

EDIT: well, I poked around some of the other closed threads about login issues. I toggled my language setting on Pixiv back and forth a few times, and suddenly the cookie works. Then it prompted me to go to the API page and grab the code from the callback request. Things seem to be working for now.

Edit 2: Well, after a while of working, a started getting some 5xx errors and captchas from CloudFlare. Did the whole jig again with password resets, language updates, logging in/out, etc; additionally, tried this method linked in another issue.
I've found that when using the PHPSESSID, the XXXXXX_xxxxxxxxx value, I routinely get a 403 when attempting to log in, but with the second method of generating a refresh token and access token, if I cleared my UN/PW, and only put the access token in under the cookie field, I was receiving 200 response codes, but PixivUtil errored out saying it couldn't find my user id.
So I'm pretty much at a loss here; again, hence why I'm posting in this issue and asking if there's something I'm missing.

[Nandaka]

Created wiki page for login using cookie for pixiv and fanbox. I hope this is useful

https://github.com/Nandaka/PixivUtil2/wiki#pixiv-login-using-cookie
https://github.com/Nandaka/PixivUtil2/wiki#fanbox-login-using-cookie

Please open an issue request if you have any feed back to the wiki page.

[jwshields]

Wiki page looks good.
Followed the directions though, running 20220804, CloudFlare is still giving me a 403.
I copied UN/PW/user-agent. Could my IP be blacklisted by CF? Even though Pixiv works in browser?

[Nandaka]

Weird, I didn't see any cloudflare, and the script didn't have special handling for it (except for oAuth validation). Can you try to run using vpn or proxy to check if it is blocked on ip address level (or if you can run from different network/isp).

[jwshields]

Tested via socks5 proxy through an SSH tunnel to two of my VPSs, one in AWS, one in GCP.
This is the output from the test from the AWS VPS: https://gist.github.com/jwshields/f5c66581536a1344f241b1db94cb496c
I've replaced a few values in the text with ___REDACTED___ - lines after 38 are the attempt to login to Pixiv.
I don't have any other residential connection that I could test from, I don't think.. But the errors I'm getting are the same from my home connection, and both of the VPSs. Just a 403 from CF.

Looking around on other Github projects, I think it may be tied to some other cookies like device_token - maybe some others too, but I don't know the significance of them. Things like __cf_bm stand out.

[Nandaka]

updated the cookie handling, so it can also import the full cookie string (only for pixiv.net, fanbox not there yet).

ensure you copy the full string from the browser (not only the PHPSESSIONID, screen shot from firefox).

old format still supported (only the value of PHPSESSIONID).

[biggestsonicfan]

Thank you for going above and beyond here, hopefully this will help folks out who are having a difficult time logging in! I have the scripts running right now but once it's finished I will test on linux and report back if something breaks!

[jwshields]

Seconded and highly agreed, thank you for all the work you do @Nandaka - it's much appreciated.

An update on my login situation: I've got the most recent commit and tested the new solution with it. Still getting 403s, but I was able to do some more in-depth debugging..
I added two lines in PixivBrowserFactory.py - after line 210 in the open_with_retry function.

print(res.read().decode("utf-8"))
res.seek(0)

This allowed me to see the output of the page that CF is serving me with the 403 (It seems the except clause was causing it to just fail outright and not dump the page.)
Turns out I'm getting the CloudFlare DDOS protection page, "Checking your browser before accessing ....."
Thing is, I can access Pixiv just fine locally using Firefox, Chrome, and a few others, but copying all the cookies exactly into pixivutil gives me a 403.
Could the mechanize lib somehow be messing with the data being sent in the request? Or making it malformed in some way?
That's really about the only thing I can think of at this point, but I've been staring at my screen for a bit too long today.

I'm still doing some digging to see what I can do to get around it, but maybe my account (or IP) is just blacklisted or something...
I've never gotten a CloudFlare check or Google Captcha in my browser ever for Pixiv, which makes this just so much more peculiar to me.
And, I can't say I've seen any other CF protected site perform a browser check during normal web usage in recent times...

One last thing! When invoking .\PixivOAuthBrowser.py to get the access token and refresh token, I am able to get those values, but upon putting them info my config file and running pixivutil, I get a plain ol' login page. (At least it's not a 4xx 😄)

Going to try and wrap my head around the oauth bits and open with cookie.. Could maybe a plain login suffice at that stage? IE, no refresh/auth token, use oauth to generate them, insert into config, attempt login with cookie, fails and presents a login page, attempt a regular login with UN/PW?

Sorry if I'm rambling at this point. Just wanted to get some info out.
I'll post any additional updates or workarounds that I can find;

[Nandaka]

yeah, because I didn't get the Cloudflare error, so I can't help on this part.

There is a library to handle the cloudflare thing. Alternatively, you can also try to replace mechanize with requests library and send me the pull request 😛

[jwshields]

If you're willing to review/accept such a large change to the project, then I'm more than willing to investigate & implement it!

Though, like I mentioned, still want to do some more digging to see if I can work around it without rewriting the entire application 😆

[fireattack]

I've encountered this Cloudflare issue with other sites.

Using requests won't fix it. Copying the exact HTTP request (down to identical headers and what not) will still trigger it.

They did something captcha-ish to detect automated tools from humans. Things like cloudscraper can work sometimes but that's not a guarantee.

I ended up using selenium last time I have to deal with it, for what it's worth.

[Nandaka]

@jwshields if it is limited to https://github.com/Nandaka/PixivUtil2/blob/master/PixivBrowserFactory.py, then should be ok.

It has additional code PixivHelper.py to handle proxy (this might need to be change too, or just move it inside the PixivBrowserFactory.py)

The on in PixivDownloadHandler is just to catch the HTTPError.

Not sure if you can do shortcut on the helper to use the cloudscraper request, instead of mechanize.request for testing.

[jwshields]

I've been a bit busy with work the last few days, but should have some time freeing up tomorrow and over the weekend; I'll do a deep dive into it all then.
Thank you for the pointers!

[TheDuriel]

Version 20220924

Getting basically the same error as in when trying to log in #1172

Logging in...
Traceback (most recent call last):
  File "PixivBrowserFactory.pyc", line 488, in login
TypeError: 'NoneType' object is not subscriptable
Error at login(): (<class 'TypeError'>, TypeError("'NoneType' object is not subscriptable"), <traceback object at 0x03A277E8>)
Error at doLogin(): (<class 'TypeError'>, TypeError("'NoneType' object is not subscriptable"), <traceback object at 0x03A16F28>)
Cannot Login!
press enter to exit.

Error stays the same regardless of if the credentials are provided in the terminal or the config file.

First version on which I encountered this was way back in pixivutil20220311-beta3