Support for extracting URLs in sitemaps #262
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 5 commits
July 5, 2019 11:27
Bump commons-io dependency version to match crawler-commons.
|
BTW, taking on |
anjackson
reviewed
Jul 13, 2019
modules/src/main/java/org/archive/modules/extractor/ExtractorRobotsTxt.java
Outdated
Show resolved
Hide resolved
|
Hm, looks like this needs some more work before we can go ahead, so I'll defer this for now. |
Closed
|
Okay, this looks like a sensible port of the UKWA code with the revisit logic taken out. I'll look at shifting our modules to be subclasses of these later on. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This builds on work by @anjackson . I'm submitting it for discussion. It requires more testing before considering merging.
I've modified the two extractors he wrote, eliminating the BL specific, recrawl, elements. I renamed them to match Heritrix extractor naming conventions. I also added a new hop type, M (Manifest) for the links extracted by them. Other than that, they are mostly as Andy wrote them, barring a few minor tweaks.
As the sitemap extractor requires a new dependency (crawler-commons), I added that to the 'modules' sub-project. This creates a dependency conflict as crawler-commons requires a newer version of commons-io. I upgraded the commons-io (from 1.4 to 2.4) in the 'commons' sub-project. As far as I can tell this doesn't cause any regression.