Add lastmod timestamp comparison in JsonldSpider

[iannesbitt]

I strongly suspect that the timestamp comparison is not working in its current state, and because of that the entire sitemap is being crawled on each harvest. Ensuring that the timestamp comparison function is working may cut down significantly on unneccessary traffic.

Related:

• Scraping Harvard Dataverse HTML rather than requesting JSON-LD directly? #23
• Create simple webserver system for testing #34

[iannesbitt]

Relevant docs are here and describe filtering based on a static value. This could be useful if we want to set the lastmod_filter to the last time we successfully harvested the site.

It would be much harder and more resource intensive, but perhaps more useful to be able to get this value from the database. However, in the case of redirects, the database would have to store the pre-redirect URL which it does not at the moment. The pre-redirect URL could be used in a query to potentially find the last time it was updated and whether the lastmod value has changed since it was stored.

[iannesbitt]

At the very least, lastmod_filter should be set to the last successful harvest date and time.

[iannesbitt]

Tested and working. lastmod_filter can be set to any date and time handled by dateparser. I set "lastmod_filter": "two years ago" in instance/nodes/mnTestOPENTOPO/settings.json and it requested only 71 of the 790 or so datasets.