feat(batchprocessor): support partial consumption of batch entries #6203
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bisakhmondal
changed the title
bisakhmondal
force-pushed
the
feat/batchprocessor/partialconsume
branch
from
52d90a7
to
82646a7
Compare
spacewander
reviewed
Jan 26, 2022
apisix/plugins/datadog.lua
Outdated
| end | ||
| -- request counter | ||
| ok, err = sock:send(format("%s:%s|%s%s", prefix .. | ||
| "request.counter", 1, "c", suffix)) |
apisix/utils/batch-processor.lua
Outdated
| local function slice_batch(batch, n) | ||
| local slice = {} | ||
| for i = n or 1, #batch, 1 do | ||
| slice[#slice+1] = batch[i] |
There was a problem hiding this comment.
We can use offset to avoid counting len?
There was a problem hiding this comment.
Oh, I see. I thought len is O(1) operation but it seems, during each invocation it gets recomputed at O(logn).
Nice nit picking 👍
apisix/utils/batch-processor.lua
Outdated
| function execute_func(premature, self, batch) | ||
| if premature then | ||
| return | ||
| end | ||
|
|
||
| local ok, err = self.func(batch.entries, self.batch_max_size) | ||
| --- In case of "err" and a valid "n" batch processor considers, all n-1 entries have been |
There was a problem hiding this comment.
Would be first_fail better than n as the var name?
|
Don't forget to mention it in the doc. |
spacewander
approved these changes
Jan 27, 2022
apisix/plugins/datadog.lua
Outdated
| end | ||
| -- request counter | ||
| ok, err = sock:send(format("%s:%s|%s%s", prefix .. | ||
| "request.counter", 1, "c", suffix)) |
There was a problem hiding this comment.
My bad! Actually, I mean the alignment like:
Lines 274 to 275 in 615ee41
tzssangglass
approved these changes
Jan 28, 2022
Signed-off-by: Bisakh Mondal <bisakhmondal00@gmail.com>
bisakhmondal
force-pushed
the
feat/batchprocessor/partialconsume
branch
from
7aee653
to
2fbe081
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Signed-off-by: Bisakh Mondal bisakhmondal00@gmail.com
What this PR does / why we need it:
Helpful where the batch processor entries can't be processed in one go (transactional) and an error occurs during the middle of the consumption. With this PR, the batch processor takes a hint from the consumer so that the already consumed entries don't get retried in the next run.
Use case: #6113 (comment), datadog plugin etc (where entries can't be processed as a bulk)
Pre-submission checklist: