Problem with download from Storytel

[fedeD84]

Until 3-4 days ago i haven't got problem with the download from storytel. In the last day every audiobook i'm trying to download get me:
ERROR: User not authorized
User and password are correct, i've recreate the cookie files but nothing change.

[podusowski]

I can confirm that.

[L2501]

try the old version

[fedeD84]

I’ve try to update after have the error the first time

[burghy86]

I can confirm error

[jo1gi]

The site api has been updated. The Storytel support has to be reimplemented.

[burghy86]

The site api has been updated. The Storytel support has to be reimplemented.

Is there anything we can do to help you reimplement it?

[freddy36]

The site api has been updated. The Storytel support has to be reimplemented.

Is there anything we can do to help you reimplement it?

I did the last rewrite but currently don't have any active storytel subscription. If someone can share his login credentials I can have a look on what needs to be adjusted.

[fedeD84]

The site api has been updated. The Storytel support has to be reimplemented.

Is there anything we can do to help you reimplement it?

I did the last rewrite but currently don't have any active storytel subscription. If someone can share his login credentials I can have a look on what needs to be adjusted.

Hi, i cant provide credential, how can i send it to you?

[freddy36]

You should be able to see my email in my profile now.

[fedeD84]

Mail send

[freddy36]

Got them, thank you.

API didn't change but it seems like they added cloudflare as proxy and it's detecting and blocking (CAPTCHA page) requests made by python/requests. Detection seems to be more sophisticated (suspecting TLS fingerprinting, header order checking or something simmilar). IP Reputation seems to be relevant too (blocked from my server, working fine when proxing it via my Desktop). Not sure how well the current cloudflare solver proxies are working, I'll try to fix it without having to use one.

[fedeD84]

Got them, thank you.

API didn't change but it seems like they added cloudflare as proxy and it's detecting and blocking (CAPTCHA page) requests made by python/requests. Detection seems to be more sophisticated (suspecting TLS fingerprinting, header order checking or something simmilar). IP Reputation seems to be relevant too (blocked from my server, working fine when proxing it via my Desktop). Not sure how well the current cloudflare solver proxies are working, I'll try to fix it without having to use one.

Thanks a lot, we wait for your update and if needed make some tests

[freddy36]

You can try the following change to make sure you've the same problem as me (blocked by cloudflare).
https://github.com/jo1gi/audiobook-dl/pull/108/files
It probably won't fix the blocking, it's just giving a better error message.

[nianhbg]

You can try the following change to make sure you've the same problem as me (blocked by cloudflare). https://github.com/jo1gi/audiobook-dl/pull/108/files It probably won't fix the blocking, it's just giving a better error message.

Hi I tried your code and yes i got nicer error message :-) I will se if I can help

[bonzoskill]

No problem actually for me

[freddy36]

I belive I tracked down the problem. It's indeed TLS fingerprinting related. The urllib3 default SSL context will set the SSL_OP_TLSEXT_PADDING option which will result in a padding extension in the ClientHello in many OpenSSL configurations. When unsetting the option (no padding extension) it seems to work fine.

I'll update the code tomorrow.

[freddy36]

I've updated the storytell_cf branch. Please test if it's working for you. If you're also using other sources, please test them too (disabled the padding extension for all HTTPS requests).

[nianhbg]

I will test it later to night

[nianhbg]

I got this error:

requests.exceptions.SSLError: HTTPSConnectionPool(host='www.storytel.com', port=443): Max retries exceeded with url: /api/login.action?m=1&token=guestsv&userid=-1&version=24.22&terminal=android&locale=sv&deviceId=995f2562-0e44-4410-b1b9-8d08261f33c4&kidsMode=false (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1007)')))

[burghy86]

not use https.

[jonasdavidpenner]

This version is working perfectly for me, thanks!

[freddy36]

@nianhbg the change should have no impact on the certificate validation. If you just use the master version you should get the same error.

[nianhbg]

Ok thanks I got it working :-D

[zaourzag]

I've updated the storytell_cf branch. Please test if it's working for you. If you're also using other sources, please test them too (disabled the padding extension for all HTTPS requests).

im sorry if im stupid, but how do i install that branch?

[freddy36]

Clone the repo and use git checkout storytell_cf to switch to the branch.

[zaourzag]

i have gotten that far, but python3 setup.py install on root does nothing

[jonasdavidpenner]

i have gotten that far, but python3 setup.py install on root does nothing

Remove the previous installation with pip uninstall audiobook-dl

Then

pip install "git+https://github.com/jo1gi/audiobook-dl.git@storytell_cf"

[thewh1teagle]

Few bugs I needed to fix (Windows):
1.

pip uninstall audiobook-dl
pip install git+https://github.com/jo1gi/audiobook-dl.git@storytell_cf

https://github.com/jo1gi/audiobook-dl/blob/storytell_cf/audiobookdl/output/output.py#L113 not working on windows, changed to 256.

https://github.com/jo1gi/audiobook-dl/blob/storytell_cf/audiobookdl/sources/source/__init__.py#L180

ssl_context.check_hostname = False

https://github.com/jo1gi/audiobook-dl/blob/storytell_cf/audiobookdl/sources/source/__init__.py#L188

session.verify = False

[freddy36]

@thewh1teagle thank you for the feedback.

What python and windows version are you running?

ssl_context.check_hostname = False and session.verify = False should not be necessary and disables HTTPS security.
Any chance that you have some fancy security software installed which intercepts web traffic?

[currently-off-my-rocker]

I get the same SSL errors as #106 (comment) and the ones I guess #106 (comment) tries to avoid with turning off verifications.

requests changed how certificates are loaded in 2.32.3. Not clear if it is a bug or not.
psf/requests#6730
If using requests 2.32.2 everything works.

With 2.32.3 it worked for me (Mac OS Sonoma, pyenv, 3.11.9) after adding either a
ssl_context.load_default_certs()
or

import certifi
ssl_context.load_verify_locations(certifi.where())

or something similar in the create_session function

[freddy36]

Thank you @currently-off-my-rocker
You're absolutly right.
I looked into this, and requests introduced a regression with the 2.32.0 release which breaks the usage of custom SSLContext's (Which we need for the cloudflare). It's fixed in 2.32.3, but breaks certifiate loading. I added the ssl_context.load_default_certs() workaround in #113.

Also fixed Windows support via #114
Please test the current master branch or the https://github.com/jo1gi/audiobook-dl/releases/tag/v0.7.4 pre release.

[zaourzag]

i installed it using pip install "git+https://github.com/jo1gi/audiobook-dl.git".
audiobook-dl https://www.storytel.com/nl/nl/books/book
Traceback (most recent call last):
File "C:\Python39\lib\runpy.py", line 197, in _run_module_as_main
return run_code(code, main_globals, None,
File "C:\Python39\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Python39\Scripts\audiobook-dl.exe_main.py", line 4, in
File "C:\Python39\lib\site-packages\audiobookdl_init.py", line 1, in
from .utils.audiobook import AudiobookFile, Chapter, AudiobookMetadata, Cover, Audiobook, Result, Series, BookId
File "C:\Python39\lib\site-packages\audiobookdl\utils\audiobook.py", line 33, in
class AudiobookFile:
File "C:\Python39\lib\site-packages\audiobookdl\utils\audiobook.py", line 41, in AudiobookFile
headers: MutableMapping[str, str | bytes] = Factory(dict)
TypeError: unsupported operand type(s) for |: 'type' and 'type'

[freddy36]

@zaourzag please try again, made some adjustments for older python versions

[zaourzag]

@zaourzag please try again, made some adjustments for older python versions

audiobook-dl https://www.storytel.com/nl/nl/books/book
Finding compatible source
Traceback (most recent call last):
File "C:\Python39\lib\runpy.py", line 197, in run_module_as_main
return run_code(code, main_globals, None,
File "C:\Python39\lib\runpy.py", line 87, in run_code
exec(code, run_globals)
File "C:\Python39\Scripts\audiobook-dl.exe_main.py", line 7, in
File "C:\Python39\lib\site-packages\audiobookdl_main.py", line 33, in main
process_url(url, options, config)
File "C:\Python39\lib\site-packages\audiobookdl_main.py", line 51, in process_url
source = source_class(options)
File "C:\Python39\lib\site-packages\audiobookdl\sources\storytel.py", line 107, in init
super().init(options)
File "C:\Python39\lib\site-packages\audiobookdl\sources\source_init_.py", line 42, in init
self.session: requests.Session = self.create_session(options)
File "C:\Python39\lib\site-packages\audiobookdl\sources\source_init.py", line 196, in create_session
ssl_context: SSLContext = self.create_ssl_context(options)
File "C:\Python39\lib\site-packages\audiobookdl\sources\source_init_.py", line 182, in create_ssl_context
ssl_context: SSLContext = urllib3.util.create_urllib3_context() # type: ignore[attr-defined]
AttributeError: module 'urllib3.util' has no attribute 'create_urllib3_context'.

[zaourzag]

perhaps i should just use a newer python version

[currently-off-my-rocker]

Please test the current master branch or the https://github.com/jo1gi/audiobook-dl/releases/tag/v0.7.4 pre release.

tag v0.7.4 works on my setup.

[freddy36]

perhaps i should just use a newer python version

Yes, that would help, can you check which exact python and urlib3 version you're running currently?

[zaourzag]

im running python v3.9.13 and urllib3 1.26.19.
edit it was the urllib3 version. i did pip install --upgrade urllib3 and it fixed it

[zaourzag]

is it maybe an idea to add an optional flareresolver "proxy"? it essentially bypasses cloudflare blocks.

[freddy36]

is it maybe an idea to add an optional flareresolver "proxy"? it essentially bypasses cloudflare blocks.

If it ever becomes mandatory, that's probably the logical solution. For now it shouldn't be necessary.

[zaourzag]

so an update, I have been using my laptop. it now throws a blocked by cloudflare error. so i moved to my desktop. it can download just fine. both devices are on the same network and on python 3.9. so i dont know the trickery cloudflare employs here

[freddy36]

Made urrlib3 version > 2 mandatory on the latest master branch.
@zaourzag maybe you can check which packets are different by running "pip freeze" in your two virtaulenvs.

[zaourzag]

Made urrlib3 version > 2 mandatory on the latest master branch. @zaourzag maybe you can check which packets are different by running "pip freeze" in your two virtaulenvs.

i dont understand what this would help. it worked on both devices one is now just blocked by cloudflare for whatever reason

[freddy36]

Made urrlib3 version > 2 mandatory on the latest master branch. @zaourzag maybe you can check which packets are different by running "pip freeze" in your two virtaulenvs.

i dont understand what this would help. it worked on both devices one is now just blocked by cloudflare for whatever reason

Just to double check that both are really running the same package versions. Maybe your laptop is running a buggy version of requests.

[zaourzag]

Just to double check that both are really running the same package versions. Maybe your laptop is running a buggy version of requests.

my laptop has requests 2.32.1 and my desktop has 2.31.0. dont think requests made any breaking changes in that time

[zaourzag]

Just to double check that both are really running the same package versions. Maybe your laptop is running a buggy version of requests.

my laptop has requests 2.32.1 and my desktop has 2.31.0. dont think requests made any breaking changes in that time

i just installed 2.31.0 on my laptop and it works again. what sorcery is this i wonder

[freddy36]

2.32.0/2.32.1/2.32.2 are all buggy.
Try 2.32.3 (latest), it contains a bug fix which is important for the cloudflare check.
It introduced another bug, but the master branch contains a workaround for that bug :)

[zaourzag]

2.32.0/2.32.1/2.32.2 are all buggy. Try 2.32.3 (latest), it contains a bug fix which is important for the cloudflare check. It introduced another bug, but the master branch contains a workaround for that bug :)

will do, still dont understand how 2.32.1 worked for a bit then didnt and 2.31.0 just did.
edit: tried it, works with 2.32.3

[zaourzag]

also the issue with urllib3 was that it was not getting installed by this project (not in the pyproject.toml file) so glad you fixed that one. took me a while to figure out lol

[freddy36]

Release v0.7.5 as a stale release to fix this issue