Make backscraper interval size a caller keyword argument

[grossir]

A bunch of backscrapers use the date_utils.make_date_range_tuples function to create the back_scrape_iterable, which takes a gap value for the size in days of each interval

juriscraper/juriscraper/lib/date_utils.py

Lines 123 to 152 in 01b0309

	def make_date_range_tuples(start, end, gap):
	"""Make an iterable of date tuples for use in iterating forms
	For example, a form might allow start and end dates and you want to iterate
	it one week at a time starting on Jan 1 and ending on Feb 3:
	>>> make_date_range_tuples(date(2017, 1, 1), date(2017, 2, 3), 7)
	[(Jan 1, Jan 7), (Jan 8, Jan 14), (Jan 15, Jan 21), (Jan 22, Jan 28),
	(Jan 29, Feb 3)]
	:param start: date when the query should start.
	:param end: date when the query should end.
	:param gap: the number of days, inclusive, that a query should span at a
	time.
	:rtype list(tuple)
	:returns: list of start, end tuples
	"""
	# We create a list of start dates and a list of end dates, then zip them
	# together. If end_dates is shorter than start_dates, fill the last value
	# with the original end date.
	start_dates = [
	d.date() for d in rrule(DAILY, interval=gap, dtstart=start, until=end)
	]
	end_start = start + datetime.timedelta(days=gap - 1)
	end_dates = [
	d.date()
	for d in rrule(DAILY, interval=gap, dtstart=end_start, until=end)
	]
	return list(zip_longest(start_dates, end_dates, fillvalue=end))

As of now, we hard code the gap value. However, we could make this a dynamic variable from the caller keyword arguments, with a sensible default in case it is not passed

juriscraper/juriscraper/opinions/united_states/state/colo.py

Lines 139 to 161 in 01b0309

	def make_backscrape_iterable(self, kwargs: dict) -> None:
	"""Checks if backscrape start and end arguments have been passed
	by caller, and parses them accordingly
	:param kwargs: passed when initializing the scraper, may or
	may not contain backscrape controlling arguments
	:return None
	"""
	start = kwargs.get("backscrape_start")
	end = kwargs.get("backscrape_end")
	if start:
	start = datetime.strptime(start, "%m/%d/%Y")
	else:
	start = self.first_opinion_date
	if end:
	end = datetime.strptime(end, "%m/%d/%Y")
	else:
	end = datetime.now()
	self.back_scrape_iterable = make_date_range_tuples(
	start, end, self.days_interval
	)

When running backscraper, I have found that the self.days_interval I defined was to big in some scrapers for some time periods, and the backscraper is not getting all documents due to page size. This would be easily solved by a dynamic argument

Also, we could take this opportunity to refactor the most common case of creating the back_scrape_iterable, which takes 2 datetime.date as start and end dates, and days_interval: int, and save it as a function to be reused. This same pattern is being used in 14 scrapers
https://github.com/search?q=repo%3Afreelawproject%2Fjuriscraper%20self.back_scrape_iterable%20%3D%20make_date_range_tuples&type=code