Support for extracting URLs in sitemaps#262

Merged

anjackson merged 7 commits intointernetarchive:masterfrom kris-sigur:sitemaps

May 20, 2021

+314-1

Commits on Jul 5, 2019

Add crawler-commons dependency.
kristinn
committed
Add support for new hopptype, (M)anifest
kristinn
committed
Add extractor to get sitemap url from robots.txt
kristinn
committed
Add extractor that handles sitemaps
kristinn
committed
Add sitemap extraction to default profile
kristinn
committed

Commits on Apr 20, 2021

Only copy source tag if not null.
Kristinn Sigurdsson
committed

Commits on May 20, 2021

Merge branch 'master' into sitemaps
anjackson
authored