[4chanarchives] 403 Forbidden Bug

[cheese529]

gallery-dl was working perfectly to download from 4chanarchives ever since the support for it was added back in #4012 but unfortunately when trying to download from a board now you just get 403 forbidden errors. Here is an example board to test with along with my verbose log.

https://4chanarchives.com/board/hr/thread/2693240

https://gist.github.com/cheese529/abd49f5abb737bb16f3181cd37a6838c

[Hrxn]

Ugh.. the images are hosted on Imgur, I guess this is why this fails.

Although it does not make sense to me, at the moment, because some links are working here in the browser (at least sometimes?)

This gives me also a HTTP 403 in the browser (incognito mode)

https://i.imgur.com/9FQY2Xy.jpg

This one works..

https://i.imgur.com/9FQY2Xy.jpg

😄

Doesn't really make any sense, not sure what triggers those 403s, I can't reproduce at the moment..

[mikf]

This happens because of the referer option added in v1.26.0.
Disable it with -o referer= or in your config with "referer": false and it'll work again.

Imgur doesn't like its files referred to from a non-imgur site, it seems.

[cheese529]

Ugh.. the images are hosted on Imgur, I guess this is why this fails.

The images that are still available on imgur still fail tho. This board is mostly SFW stuff and therefore a lot these are still available.

Although it does not make sense to me, at the moment, because some links are working here in the browser (at least sometimes?)

Yep I am having the exact same problem. It seems that if you click the link you get a 403 but if you click the actual image it will work.

[cheese529]

This happens because of the referer option added in v1.26.0. Disable it with -o referer= or in your config with "referer": false and it'll work again.

Imgur doesn't like its files referred to from a non-imgur site, it seems.

Ahhh okay that explains everything, thank you for this. Does this also mean we need disable the referrer option on reddit too since that is a non-imgur site?

[mikf]

Does this also mean we need disable the referrer option on reddit too since that is a non-imgur site?

I guess it wouldn't hurt, but reddit handles imgur links indirectly (it spawns an ImgurExtractor that does the "right" thing and uses its own headers and options), while 4chanarchives downloads from these links directly.

[cheese529]

it spawns an ImgurExtractor that does the "right" thing and uses its own headers and options

Ahhh well that makes perfect sense. I guess i'll test with both to see if it changes anything.

BTW is {num} not supported for the filename? I currently have it set as "{filename}_{num}.{extension}", so in case it runs into a duplicate filename it will download with a number at the end but instead I am just getting "None" at the end for all images.

[Hrxn]

Ah yeah, I see it now...

<a target="_blank" href="https://i.imgur.com/8dfUO96.jpg">TaylorSwift-2008MTV-VMA-Arrivals_Vettri.Net-38.jpg</a>

<a target="_blank" href="https://i.imgur.com/8dfUO96.jpg" class="fileThumb" rel="noreferrer">
                     
</a>

(I removed the content inside of the second anchor tag)

Yep I am having the exact same problem. It seems that if you click the link you get a 403 but if you click the actual image it will work.

That is why.

rel can be used just like referrerpolicy

[Hrxn]

it spawns an ImgurExtractor that does the "right" thing and uses its own headers and options

Ahhh well that makes perfect sense. I guess i'll test with both to see if it changes anything.

BTW is {num} not supported for the filename? I currently have it set as "{filename}_{num}.{extension}", so in case it runs into a duplicate filename it will download with a number at the end but instead I am just getting "None" at the end for all images.

Use {no} instead of {num} here...

Shows the age of this extractor, I think..

[mikf]

Shows the age of this extractor, I think..

Support for this site was added this year :) (1406f71)
It's just that it tries to replicate the metadata names used by all the other *chan extractors.

[Hrxn]

Should have taken a look before posting :D

[cheese529]

Use {no} instead of {num} here...

This seems to work. Thank you both once again very much for all your help :)