v14: JSONDecodeError when trying to log in

[GiladKupfer]

when I put my detail it gave me this error

"Please log in to continue.
Fatal error encountered:

Traceback (most recent call last):
File "main.py", line 169, in
File "asyncio\base_events.py", line 616, in run_until_complete
File "twitch.py", line 423, in run
File "twitch.py", line 443, in _run
File "twitch.py", line 986, in get_auth
File "twitch.py", line 208, in validate
File "twitch.py", line 233, in _validate
File "twitch.py", line 125, in login
File "aiohttp\client_reqrep.py", line 1119, in json
File "json_init.py", line 357, in loads
File "json\decoder.py", line 340, in decode
json.decoder.JSONDecodeError: Extra data: line 1 column 128 (char 127)

Exiting...

Application Terminated.
Close the window to exit the application."

[NaniDaFuck]

ya same thing

[DevilXD]

Hello o/

It looks like it's Twitch's fault. The passport endpoint started to return 400s with a messed up (duplicated) JSON payload. Here's the raw response that fails to decode:

b'{"error":"Please update your app to continue","error_code":5023,"error_description":"client is not supported for this feature"}{"error":"Please update your app to continue","error_code":5023,"error_description":"client is not supported for this feature"}'

The JSON decoder breaks because there's two root objects instead of one.

Anyway, that's not really the main issue here. Looks like Twitch has started to deal with this client ID as well, and I expect it's only going to get worse from here. It's probably going to get protected by the integrity crap just like the main one did. For now, the logins have stopped working, and there's nothing I can do about it. The official Android app is probably going to use a different endpoint for logins, and thus this client ID is no longer allowed to use the normal endpoint.

[NaniDaFuck]

is there no way to login now :/

[Huiaei]

Perhaps consider manually filling in cookies? I've seen many similar projects use this method.

[GiladKupfer]

Perhaps consider manually filling in cookies? I've seen many similar projects use this method.
how do i manually put cookies

[Huiaei]

Perhaps consider manually filling in cookies? I've seen many similar projects use this method.
how do i manually put cookies

Sorry, this is just a suggestion. I've opened the cookie.jar and although it has the relevant characters in it, vscode doesn't fully open it, so I don't know where to start with the changes.

[DevilXD]

@Huiaei It's not possible. While I can agree that I've seen some of those "many similar projects" that have used this technique, it can't be applied now. It's because Twitch has made changes that require integrity interaction for the web client ID on desktop, and thus your web browser can't be used - you could scrape the authentication information, but not the integrity token, as it isn't stored in a cookie. Even then, you'd have no real way to refresh the token (which can change at pretty much any time), without having your browser open just for this purpose, and then somehow automate interaction with it, which isn't easily possible either.

The miner now uses an android app client ID to bypass the integrity system, and since there's no way to login as an android app via the PC browser, there's no cookie I could scrape either. It's a different platform, so it's not possible.

[EffortlessFury]

Would this be possible if you were running a headless browser?

[DevilXD]

I'm not sure if you missed my point or not. All headless browsers are currently detected by Twitch as bots and denied any login or drop claiming interaction. The web client was the first to receive it, now the android app has received one as well. Trying to login with anything other than a legit browser or app is not going to work, simple as that. There's no existing workaround for the /protected_login endpoint, and the /login one the miner uses, is only going to produce a token that fails any and every Twitch's integrity check. There is nothing I can do about this, unfortunately. If you haven't seen #40 yet, that almost killed this project entirely, I'd recommend checking it out for more info.

[EffortlessFury]

Ah I see, I didn't realize headless browsers were included in what you were discussing.

[LegendAdmin]

@Huiaei It's not possible. While I can agree that I've seen some of those "many similar projects" that have used this technique, it can't be applied now. It's because Twitch has made changes that require integrity interaction for the web client ID on desktop, and thus your web browser can't be used - you could scrape the authentication information, but not the integrity token, as it isn't stored in a cookie. Even then, you'd have no real way to refresh the token (which can change at pretty much any time), without having your browser open just for this purpose, and then somehow automate interaction with it, which isn't easily possible either.

The miner now uses an android app client ID to bypass the integrity system, and since there's no way to login as an android app via the PC browser, there's no cookie I could scrape either. It's a different platform, so it's not possible.

I'm in the same situation, so there is no way to log in now?

[DevilXD]

@LegendAdmin No, there's not. Using a browser cookie will cause points and drop claiming to error out.

Please, if you're new to the issue, see #40 for more information. This is a continuation of the story, really.

[LegendAdmin]

@LegendAdmin No, there's not. Using a browser cookie will cause points and drop claiming to error out.

Please, if you're new to the issue, see #40 for more information. This is a continuation of the story, really.

I looked, but I think it's different from what I encountered, and my error is exactly the same as this one.
However, I solved it now, the process goes like this:
I used the old V12 version at the beginning. I used it for a long time, but it kept getting an error some time ago, but I didn't have time to worry about it. Later, I went to Github and looked at it, only to find that 2 big version updates have been made recently. Then I downloaded the latest version, because the old version will report an error, I was afraid that Twitch judged my account as a robot, so I deleted the previous cookies and tried to log in again, but I encountered this error, no matter what
No matter how many times I try to log in, the result is the same, finally I restored the previous cookies and it actually works again (so I still don't understand why my account can't log in in the software, but I can use it now)

[LegendAdmin]

@DevilXD I thought I was done, but when I rebooted my computer, there was a new problem. Here is a new error code:

Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 169, in <module>
  File "asyncio\base_events.py", line 616, in run_until_complete
  File "twitch.py", line 423, in run
  File "twitch.py", line 484, in _run
  File "inventory.py", line 191, in claim
  File "inventory.py", line 109, in claim
  File "inventory.py", line 125, in _claim
  File "twitch.py", line 1059, in gql_request
exceptions.MinerException: GQL error: [{'message': 'failed integrity check'}]

Also, now I can still only log in with the old cookie file, and if I delete the file and log in again, I still only get the error message I mentioned before.
I re-read your release description, I think it's possible that the CAPTCHA verification is stuck, I'll try to wait for some time before logging in again to see if it works (perhaps you can consider adding the ability to manually pass the verification in v15, although I don't know if this is possible)

[Windows200000]

@DevilXD I thought I was done, but when I rebooted my computer, there was a new problem. Here is a new error code:

Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 169, in <module>
  File "asyncio\base_events.py", line 616, in run_until_complete
  File "twitch.py", line 423, in run
  File "twitch.py", line 484, in _run
  File "inventory.py", line 191, in claim
  File "inventory.py", line 109, in claim
  File "inventory.py", line 125, in _claim
  File "twitch.py", line 1059, in gql_request
exceptions.MinerException: GQL error: [{'message': 'failed integrity check'}]

Also, now I can still only log in with the old cookie file, and if I delete the file and log in again, I still only get the error message I mentioned before. I re-read your release description, I think it's possible that the CAPTCHA verification is stuck, I'll try to wait for some time before logging in again to see if it works (perhaps you can consider adding the ability to manually pass the verification in v15, although I don't know if this is possible)

DevilXD already addressed this. While the cookies work for the login, twitch doesn't allow you to actually mine without the integrity check.

@Huiaei It's not possible. While I can agree that I've seen some of those "many similar projects" that have used this technique, it can't be applied now. It's because Twitch has made changes that require integrity interaction for the web client ID on desktop, and thus your web browser can't be used - you could scrape the authentication information, but not the integrity token, as it isn't stored in a cookie. Even then, you'd have no real way to refresh the token (which can change at pretty much any time), without having your browser open just for this purpose, and then somehow automate interaction with it, which isn't easily possible either.

The miner now uses an android app client ID to bypass the integrity system, and since there's no way to login as an android app via the PC browser, there's no cookie I could scrape either. It's a different platform, so it's not possible.

[DevilXD]

@LegendAdmin If you're getting the "failed integrity check" message, that means you are still using the old cookie and haven't read the important note I've attached under v13 and v14 release notes. And because of this issue, you won't be able to generate a new cookie that gets rid of that error. You're can only blame yourself for not reading properly here, really. I can't help you with it.

[LegendAdmin]

@LegendAdmin If you're getting the "failed integrity check" message, that means you are still using the old cookie and haven't read the important note I've attached under v13 and v14 release notes. And because of this issue, you won't be able to generate a new cookie that gets rid of that error. You're can only blame yourself for not reading properly here, really. I can't help you with it.

The problem was that I deleted the old cookie file at the beginning and then logged back in, however, logging back in reported the error I mentioned at the beginning, so I then tried to use the old cookie file, but instead logged in successfully, and I think all of this, including what happened afterwards, I've described before.
I don't know if you have encountered the situation I described, but I think the best way to do it, if it is possible, is to add manual authentication via CAPTCHA, what do you think?

[DevilXD]

I'm kinda lost on what you're saying. It doesn't matter though, because as me and you said said, you've got an error and thus can't use the miner anymore. It doesn't matter what you did, it's dead now and there's no way to bring it back. Deal with it.

Also, please just be patient, I happen to have some real life to take care of as well. If I happen to implement anything useful that helps resolve this issue, I'll let you know.

[EffortlessFury]

I then tried to use the old cookie file, but instead logged in successfully

Logged in successfully without the integrity check error? Because if you're still getting the integrity check error you're not really logging in successfully. But yes, as DevilXD said, you would've needed to login using a newer version, but now logging in with this application is no longer possible whatsoever, so you missed your window to get valid cookies.

[DevilXD]

I'm currently in the process of testing a solution that appears to be working for now, as in, I'm getting an authentication token - hopefully a usable and workable one. The solution involves using https://github.com/ultrafunkamsterdam/undetected-chromedriver and a local Chrome installation, to access the https://twitch.tv/login page, and let the user enter their credentials (and optional 2FA) as they normally would. The script would then intercept the intermediate /protected_login request and replace it's web browser Client ID with the andoird app Client ID, which yields an authentication token that ultimately isn't protected by the integrity system, bypassing the need to deal with it, as the current v14 does already.

At least in theory, in practice, I'd have to code all of that in first. It also introduces an existing Chrome installation as a dependency, which isn't exactly ideal (I'm a Firefox person myself, for example, had to specifically install Chrome just for this), but it's necessary to effectively deal with this issue. Hopefully it won't be too inconvenient for most people.

[DevilXD]

I had a day off and it looked promising, and 6 hours of binging later: 73a2a61

I took care of fixing the freeze dependencies, so an executable is buildable and works (at least so far for me). It's too early for a release yet, and I have exactly no idea what will happen if you'd have no Chrome installed (there's no handling for that case either). If you'd happen to get any kind of an error out of this one, or have any other issue while logging in, please just reply below.

Anyone here willing to do so, please test it and tell me how it went for you.

EDIT: Got a first successful drop claim off this method, so there should be no issues. Please give it a try and report regardless.

[scryptio]

@DevilXD Gave it a try for you. Was Working for a bit over an hour

(env) C:\Users\\Desktop\TwitchDropsMiner [master ≡]> python .\main.py

Fatal error encountered:
Traceback (most recent call last):
  File "C:\Users\\Desktop\TwitchDropsMiner\main.py", line 175, in <module>
    loop.run_until_complete(client.run())
  File "C:\Python39\lib\asyncio\base_events.py", line 642, in run_until_complete
    return future.result()
  File "C:\Users\\Desktop\TwitchDropsMiner\twitch.py", line 351, in run
    await self._run()
  File "C:\Users\\Desktop\TwitchDropsMiner\twitch.py", line 485, in _run
    await asyncio.gather(*(channel.check_online() for channel in acl_channels))
  File "C:\Users\\Desktop\TwitchDropsMiner\channel.py", line 252, in check_online
    self._stream = await self.get_stream()
  File "C:\Users\\Desktop\TwitchDropsMiner\channel.py", line 238, in get_stream
    response: JsonType = await self._twitch.gql_request(
  File "C:\Users\\Desktop\TwitchDropsMiner\twitch.py", line 987, in gql_request
    raise MinerException(f"GQL error: {response_json['errors']}")
exceptions.MinerException: GQL error: [{'message': 'service timeout', 'path': ['user', 'stream']}]

Exiting...
Cannot connect to Twitch, retrying in 1 seconds...

Trying to reconnect in a loop since. Restarting results in the same error. Guess the token got invalidated? Had to remove the cookies.jar and login again.

[DevilXD]

Had to remove the cookies.jar and login again.

Did it start working back up afterwards?

[scryptio]

Yes, after removing and a new login it was working for another 3.5 hours. Came back from a walk, it started watching another game as it seems and now tried to switch to a priority one and is stuck now (but still claiming channel points as it seems). Progress bars in the app are stuck at 0:00:00 and a campaign from another game, too. Weird stuff :-)

EDIT: It says it is mining, but it isn't. Looks like it can't anymore. I will try deleting the cookie again.

[DevilXD]

There's a lot of confusing information here.

It says it is mining, but it isn't.

Are you sure of that though? Twitch has a bug with displaying progress of some drops, where it'll "appear stuck" (no drop progress updates are received) but actually count properly in the background, and then automatically get unstuck at some point later on. You could've ran into that instead. If the watch event is sent, but there's no drop progress response within 10 seconds afterwards it's the watch loop's job to "manually" trigger a minute countdown, so that the timer continues to tick down even when the bug occurs. If no watch events would be sent, the timer would stop counting down and get stuck at the closest whole minute.

but still claiming channel points as it seems

This is also an indication of the watch events still being sent properly, as each event yields +10 points every minute, and +50 claimable points are yielded every ~15 minutes. No watch events being sent would mean no points events being received. Unless you were watching something else on Twitch in the meantime - the miner will actually receive those events as well.

Speaking of watching something on Twitch with the miner in the background - the miner may stop progressing a drop if you're watching something in the browser, on the same account. The reason is that, your browser will also be sending those watch events to Twitch, and those may stop the miner ones from registering completely. It'll assume it ran into the bug and continue ticking minutes down manually, but no actual progress will be made, and once you stop watching in the browser, the first event that'll register will update the drop's progress to the actual one it's supposed to be at. I've personally experienced some funny things happening if you watch the same channel the miner is watching, or at least try to mine the same drop with the browser and miner simultaneously. I remember turning my PC off and going to sleep with 20% progress, only to wake up to 60% progress the next day. Also weird stuff.

tried to switch to a priority one and is stuck now

Always try the Reload option from the Settings tab when this happens, it could be a case where it could miss a "channel going offline" event and it's sending the watch events to an offline channel now (that appears online in the miner). When that happens, no points nor drop progress will be earned, until you either switch the channel manually or it gets to switch on it's own. Or a reload happens of course.

Overall, I don't think it's the cookie problem, seems more like Twitch just struggling to work properly. Or it's your internet connection dropping crucial events for some unknown reason. Either way, please watch out for any traceback being printed out to the output, as having that will help me understand what the problem is and eliminate any bugs, if there would be any. Hope all of this description helps =)

[DevilXD]

@pree This is known, I've reported the issue to the undetected-chromedriver team: ultrafunkamsterdam/undetected-chromedriver#897

[Windows200000]

Why does this have the "Solved for the next release" label, when the undetected chrome driver doesn't pass the twitch integrity system (as @pree pointed out and I experienced myself with current master)?

[DevilXD]

Well, the general gist of it was that it was solved for the next release, but someone else threw a wrench into the gears, and now it's kinda "pending" instead. However, I'm working on an additional something that might turn this back into being solved, even without the chromedriver - turns out Twitch pulled off the same thing as they did last time, and the standard login endpoint may work like how it used to again, without what the 3rd comment in this issue says (5023: client is not supported for this feature). The same thing happened previously as well, although didn't get us anywhere since the generated cookie was protected by the integrity system later down the line. This one still isn't, and it's just a matter of bringing back the old login flow, with Chrome login flow as a backup, and it should be okay.

The only problem is little available time I have every day, with all the usual life things happening around me. There's the weekend coming, where I'll hopefully be able to push something useful out for testing. Please have patience and stay tuned.

[Windows200000]

Don't feel pressured to hurry. You are doing this for free, which I appreciate, but you have no obligation to even keep this going, so if other stuff gets in the way don't feel too stressed about it... <3

[matarife123]

i also got this error, after closing and starting several times the program, magically it worked again

[JourneyOver]

i also got this error, after closing and starting several times the program, magically it worked again

yea haha, the past few days I've had troubles getting the application to login.. just tried again a minute ago and magically it logged in just fine and is now back to working order from what I can tell (will have to wait a while and see if anything else happens).

edit: Should mention I'm still running v14 since I can't get the master branch to work for me due to me not having normal chrome (I'm using a browser called catsxp which is chrome based and it's portable) and I don't really feel like downloading normal chrome for just one application..

edit2: looks like it worked up until the point that it grabbed a drop, and then it threw an error:

Claimed drop: Pearl SMG (3/4)
Fatal error encountered:

Traceback (most recent call last):
  File "main.py", line 169, in <module>
  File "asyncio\base_events.py", line 616, in run_until_complete
  File "twitch.py", line 423, in run
  File "twitch.py", line 470, in _run
  File "twitch.py", line 1084, in fetch_inventory
  File "twitch.py", line 1059, in gql_request
exceptions.MinerException: GQL error: [{'message': 'service timeout', 'path': ['currentUser', 'inventory', 'dropCampaignsInProgress']}]

Exiting...

Application Terminated.
Close the window to exit the application.

closing the application and opening it back up I'm still able to login and it continues to work, will keep an eye on things.

[DevilXD]

Hmm, well... Yes, this is actually possible, since as I've mentioned, it looks like Twitch has gone back to using the normal login procedure (that v14 uses), and I'm currently working on reimplementing it in v15, together with Chrome login and additional login that'll try the standard login first, with Chrome as a fallback in case Captcha or any other problem happens.

@JourneyOver That looks like a simple GQL timeout due to Twitch having a temporary hiccup, I can add a retry to it when that'd happen again in v15. Cheers for the traceback ^^

[JourneyOver]

@DevilXD

Hmm, well... Yes, this is actually possible, since as I've mentioned, it looks like Twitch has gone back to using the normal login procedure (that v14 uses), and I'm currently working on reimplementing it in v15, together with Chrome login and additional login that'll try the standard login first, with Chrome as a fallback in case Captcha or any other problem happens.

will the v15 allow us to use a portable version of chrome (or custom chrome browser (like said previously I'm using a browser called catsxp which chromium based and has a portable version as well but the exe is called catsxp and not chrome)) when the fallback happens? or will it be required that normal chrome be installed or?

That looks like a simple GQL timeout due to Twitch having a temporary hiccup, I can add a retry to it when that'd happen again in v15. Cheers for the traceback ^^

Yea I figured that's probably what had happened (haven't had anymore issues so far since after I restarted it after I got that error) but wanted to throw up the traceback regardless so it's not a problem :p

[DevilXD]

will the v15 allow us to use a portable version of chrome (...) when the fallback happens?

I'm using this for the browser login: https://github.com/ultrafunkamsterdam/undetected-chromedriver

One of the bullet points says:

Works also on Brave Browser and many other Chromium based browsers, some tweaking

I'm also using it's automated Chrome executable lookup, code of which is here: https://github.com/ultrafunkamsterdam/undetected-chromedriver/blob/7cb068d977a062a1e82c8dd676918089df74bbca/undetected_chromedriver/__init__.py#L675-L715

Anything it manages to find under these paths, should work. Can't really say much more than this as this is my first time using this thing, and I know as much as I can read from the README and the code above, so everything I just shared.

[DevilXD]

I've just finished reimplementing the old login flow in the current master via: 504c6b0

Chrome is kept as backup, but still doesn't work. No further info under ultrafunkamsterdam/undetected-chromedriver#897 yet either. It may work in the future, so I'm leaving it in for now. We're back to testing v15 before it's release.

[Mega349]

@DevilXD i tried your commit 577a428 (~2h ago) to build by myself.
And i cant logon with the message "Please log in to continue" (thats what i try..)

What do i wrong or what can i do?

[DevilXD]

@Mega349 It looks like it's my bad, I haven't thought hard enough when making that commit and forgot a not in one place. It should work again now.

[Mega349]

@DevilXD works thx for the fast answer ^^

[DevilXD]

This is now fixed in v15.