Refactor scheduler/polling out of scheduled-feed/main.go

[Qinusty]

scheduled-feed/main.go was becoming quite cluttered as functionality has been added, this PR remedies this by shifting many of the responsibilities to the scheduler with additional structs being introduced to handle the polling of feed collections.

[tom--pollard]

Other than my previous comments, this looks a lot cleaner! It also opens up a smoother path to introducing per feed polling rates, as mentioned in #85

[tom--pollard]

Running the binary locally, and triggering via a remote curl, at the end of the logs I get:

time="2021-05-20T11:17:11Z" level=info msg="Packages successfully processed" feed=pypi num_processed=0
time="2021-05-20T11:17:11Z" level=info msg="Packages successfully processed" feed=goproxy num_processed=0
time="2021-05-20T11:17:11Z" level=info msg="Packages successfully processed" feed=packagist num_processed=0
time="2021-05-20T11:17:12Z" level=info msg="Packages successfully processed" feed=crates num_processed=0
time="2021-05-20T11:17:13Z" level=info msg="Packages successfully processed" feed=nuget num_processed=0
time="2021-05-20T11:17:13Z" level=info msg="Packages successfully processed" feed=npm num_processed=0
time="2021-05-20T11:17:13Z" level=info msg="Packages successfully processed" feed=rubygems num_processed=0
time="2021-05-20T11:17:13Z" level=info msg="0 packages processed"

I've noticed before these changes that the num_processed logging can sometimes be odd, but we should probably look to address this in the PR.

At a glance, I think it's due to WithField("num_processed", len(result.packages)) expecting an interface, not an int.

[Qinusty]

I've noticed before these changes that the num_processed logging can sometimes be odd, but we should probably look to address this in the PR.

num_processed=0 is due to no packages being processed within the cutoff window I believe. I'm not witnessing this with longer pollRate values.

[tom--pollard]

I've noticed before these changes that the num_processed logging can sometimes be odd, but we should probably look to address this in the PR.

num_processed=0 is due to no packages being processed within the cutoff window I believe. I'm not witnessing this with longer pollRate values.

I'm not sure that's the case, on a clean execution with the default time, I get multiple sending package upstream logs, and a return of n packages processed on the http request. This log happens at polltime, and it does seem to correctly report per feed, but then at the end of the logs again as zero... which would suggest something is triggering another poll immediately?

• I can't now replicate this with curl, but I can with a web browser which is seemingly making 2 requests, as such the second requests gets zero hits due to the preceding cutoff being applied.

[tom--pollard]

LGTM!