9Gag can fetch only 10 posts

[molotow11]

Hello,

Thanks for great tool.

I have an issue and 9gag parser fetch only 10 posts each time, different paths checked with -n 20 and --limit 20 it is no matter.

E.g.

./skraper ninegag /hot -t log -n 20
./skraper ninegag /coronavirus -t log -n 30
./skraper ninegag /coronavirus -t log --limit 50

All returns:
Fetched 10 posts. Saved to: /home/developer/skraper-master/ninegag/coronavirus_0 4062020_123420.log

[sokomishalov]

Hey, thanks, appreciate that :)
Well, the reason is very simple - 9gag has only 10 posts on the initial page.
Some providers are able to provide much more on one request, and some of them even provide the possibility to request the exact number of posts, but unfortunately, 9gag is not one of them.
That's why this param named the 'limit' and not the 'size'.

[molotow11]

Yes, but this loads next posts with javascript and some providers also using such behavior for make harder parsing. I imagined, that your magic tool also get get such posts.

[sokomishalov]

Well, I'll try to find it out somehow :)
I'll let you know if I have some good news

[subhead]

Are there any updates on this? Facing the same problem.

[sokomishalov]

I've released the 0.6.3 version which contains a fix for this issue.
FYI It will recursively request another batch of data (10 posts) until the limit reached or topic data is over.
So be careful with the limits you are setting.