Custom enumerators #944
Custom enumerators #944
Conversation
| def enumerator_builder(cursor:) | ||
| collection = self.collection | ||
|
|
||
| job_iteration_builder = JobIteration::EnumeratorBuilder.new(nil) |
There was a problem hiding this comment.
It's a bit icky to pass in nil here - it's for the job argument.
@job is only referenced by EnumeratorBuilder#build_throttle_enumerator, which isn't called here, so there's no immediate issue with it being nil.
Alternatively, job could be provided as an additional argument to enumerator_builder, but leaking the job instance into the Task interface feels worse than passing in nil.
|
I have signed the CLA! |
rianmcguire
force-pushed
the
enumerator_builder
branch
from
3a9a44c
to
fb0033e
Compare
rianmcguire
force-pushed
the
enumerator_builder
branch
from
fb0033e
to
f1ba5f5
Compare
There was a problem hiding this comment.
This makes sense to me. The API seems straightforward and the example in the README is clear 👍 Could we add @sambostock as a co-author since some of his original contributions were ported over to this PR?
cc @Shopify/rails for review on this please.
|
Please squash your commits and let's give Sam co-author please. Thank you! |
Co-authored-by: Sam Bostock <sam.bostock@shopify.com> Co-authored-by: Gannon McGibbon <gannon@hey.com>
rianmcguire
force-pushed
the
enumerator_builder
branch
from
4127af5
to
c0f8220
Compare
Done! Thanks for the feedback and reviews everyone. |
This is based on the work in #326 and the design suggested by @adrianna-chang-shopify in this comment.
It allows tasks to define their own custom enumerators, in order to support use cases like iteration over external API resources, or alternative database querying strategies.
It does this by exposing an overridable
enumerator_buildermethod onTask, whose default implementation is the existingcollectiontype inspection logic.Fixes #207