Skip to content

Commit

Permalink
[gelbooru] workaround pagination limits
Browse files Browse the repository at this point in the history
Gelbooru only allows to retrieve the latest 20k posts for a tag search.
Add 'id:<N' to the search tags to work around that limitation, where N
is the ID of the last retrieved post.

http://gelbooru.me/index.php?page=forum&s=view&id=1467
  • Loading branch information
mikf committed Nov 26, 2021
1 parent f2ae179 commit 93cef78
Showing 1 changed file with 14 additions and 1 deletion.
15 changes: 14 additions & 1 deletion gallery_dl/extractor/gelbooru_v02.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,21 @@ def _pagination(self, params):
params["pid"] = self.page_start
params["limit"] = self.per_page

post = None
while True:
root = self._api_request(params)
try:
root = self._api_request(params)
except ElementTree.ParseError:
if "tags" not in params or post is None:
raise
taglist = [tag for tag in params["tags"].split()
if not tag.startswith("id:<")]
taglist.append("id:<" + str(post.attrib["id"]))
params["tags"] = " ".join(taglist)
params["pid"] = 0
continue

post = None
for post in root:
yield post.attrib

Expand Down

0 comments on commit 93cef78

Please sign in to comment.