re-think how we calculate "since" info for published lints

[pq]

Context: SpinlockLabs/github.dart#244 (comment).

We currently use a very naive method to fetch all (and increasingly many) SDK repo tags to calculate when a lint was first included in an SDK and this is turning out to no longer scale. (Our caching scheme doesn't play nice with the frequent releases either.)

We should really re-think it.

[robrbecker]

I gave this a test myself and yes indeed it takes 57 pages of tags for about 38s.
I think if I were to make this https://github.com/SpinlockLabs/github.dart/blob/master/lib/src/common/repos_service.dart#L245 look like
github.repositories.listTags(slug, page: 57, pages: 1, perPage: 30)
(ie allow passing in the starting page, how many pages to get, and the page size)

Then that would allow you to save the list of tags up to a certain page, and then next time it is run just request starting at the last page. You'd still have to resolve duplicates. Or I suppose if you did a page size of 1 you could just start where you left of and only get new tags since you last checked.

One thing to note is that the API returns the tags in the same order, but I don't know if that's guaranteed. Also changing the per_page means github calculates the starting point as per_page * page so asking for page 5 is not the same with different per_page requests. That means if you start caching tags you should always use the same page size. (or recalculate your starting point correctly)

Would it be helpful if I added the proposed changes?

[robrbecker]

Well, I published 8.0.0 with the pagination additions. https://github.com/SpinlockLabs/github.dart/blob/master/lib/src/common/repos_service.dart#L245
That should allow you to rework caching if you need.

[pq]

Thanks so much @robrbecker!