[Site Request] archive.4plebs

[cheese529]

Hello there Mikf, hope you are doing well my man! It's been a while since i've posted here, been super busy with school and stuff. Anyways was hoping if I could get this site https://archive.4plebs.org/ added in the same format that you added 4chan Archives last year #4012.

It was super useful to just paste the link of a board like https://archive.4plebs.org/hr/page/1/ for example and have it download every single page from the board in order. Would also be nice to have post processors as well to get metadata and such.

Hopefully we can get this done soon before the owner of the site decides to add hard anti scraping measures here as well.

[mikf]

All 4chan archive sites based on Fuuka/Foolfuuka are already supported:
https://github.com/mikf/gallery-dl/blob/master/docs/supportedsites.md

[cheese529]

All 4chan archive sites based on Fuuka/Foolfuuka are already supported: https://github.com/mikf/gallery-dl/blob/master/docs/supportedsites.md

Ah so that must mean there is no support for board extractors for those sites, hence why it keeps saying unsupported URL when I try to do https://archive.4plebs.org/hr/page/1

[Hrxn]

@cheese529 But thread extraction works, right?
Why would you want to grab entire boards, though? Some boards have threads which are, let's say, thematically very different..

[cheese529]

But thread extraction works, right?

It does but some boards have thousands and thousands of threads, it would take hours for me to manually go through each page of threads and get the URL for each one to download. The reason for wanting to grab an entire board is mainly for boards such has /hr/ or /s/ or /aco/ where the themes are pretty similar.

[Hrxn]

I fail to see how /hr/ is thematically similar, to be honest.

[cheese529]

I believe the majority of the threads are made about celebs on /hr/ so I guess you could count that.

[Hrxn]

Let's see..

Based on recently updated threads on https://boards.4chan.org/hr/, as of right now, the first 15 threads include..

• Tarot Cards
• A South Korean Girl K-Pop group (although I admit that could count as celeb pics)
• Pictures about plants in spring
• Pictures about angels in art
• Dragon Ball
• An Anime
• Spiritual Infographics (yes, lol)

So that's about a half. Which would not fit automatically, content-wise

[mikf]

Boards are supported (#1044, fb64183). That's why it lists "Boards" in docs/supportedsites for Foolfuuka sites.

Their accepted URL pattern is currently rather picky, though. It needs to be https://domain/hr/ or https://domain/hr/N. Anything else won't get matched.

[cheese529]

So that's about a half. Which would not fit automatically, content-wise

Those are pretty random haha but I believe it's still the minority. If you go here https://boards.4chan.org/hr/catalog and https://boards.4chan.org/hr/archive you will see it's mostly just celebs posted. I will let you know exactly how much was unrelated stuff once I download the entire board

[cheese529]

Their accepted URL pattern is currently rather picky

Tried the different URL pattern but unfortunately it just gives me a cloud fare 403 error. Opening the board in browser works fine, no captcha or verification required to browser. Visiting API url such as https://archive.4plebs.org/_/api/chan/index/?board=hr&page=1 does not give a verification check either. My user agent is added to config and I am using cookies. Guess this is related to #5399

D:\4plebs> gallery-new.exe --cookies-from-browser firefox https://archive.4plebs.org/hr/
[cookies][info] Extracted 734 cookies from Firefox
[4plebs][warning] Cloudflare challenge
[4plebs][error] HttpError: '403 Forbidden' for 'https://archive.4plebs.org/_/api/chan/index/?board=hr&page=1'

[stubkan]

once I download the entire board

Do you intend to scrape absolutely everything? That kind of stuff is why they have been blocking robots, since people aggressively scrape vast amounts of data. 4plebs is one of the few archives that has a generous api call limit compared to the others, if this is abused - then that can change quickly.

Instead of getting the entire board by hammering 4plebs, why not get it from the provided dumps?

• https://archive.org/details/4plebsimagedump
• https://archive.org/details/imageboard_datasets
• https://archive.org/details/datasets

If you just want the recent stuff - everything in the last 3 days is still on 4chan, and you can scrape 4chan itself which has a very high bandwidth and is more able to handle scraping.

• https://boards.4chan.org/hr/archive

it would take hours for me to manually go through each page of threads and get the URL for each one to download.

If you want specific things, there is a search function provided by 4plebs and other archives to only search for keywords such as "celebrity name" and parse only that for relevant threads, which is much better for the archive site. Grabbing everything doesn't make sense, because you still have to search "everything" you grabbed don't you? Do the search first, before you grab. It's not hard to scan search result page for threads and grab the thread numbers. This is how I did it.

echo "Extracting threads from $1 ..."
grep -Po '(?<=No.).+?(?=Replies)' $1 > extracted-threads.txt
echo "Written output to extracted-threads.txt."