Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Develop a concurrent memory-limited batch processor #80

Closed
jmacd opened this issue Nov 3, 2023 · 0 comments · Fixed by #93
Closed

Develop a concurrent memory-limited batch processor #80

jmacd opened this issue Nov 3, 2023 · 0 comments · Fixed by #93
Assignees

Comments

@jmacd
Copy link
Contributor

jmacd commented Nov 3, 2023

Briefly: we have had trouble tuning the configuration of an OTel-Arrow pipeline so that data does not drop. The root cause of this is the exporterhelper configuration and concurrency behavior combined with the core batchprocessor configuration and concurrency behavior.

We will develop an alternative based on the the core batchprocessor by forking the code, and then:

  1. Modify the batchprocessor to block requests until the batch response is received
  2. Modify the batchprocessor to return error or no-errors to the producer
  3. Modify the batchprocessor to send >1 request in parallel, whenever data is available, using background goroutines for each batch
  4. Establish an in-flight-bytes limit that prevents more than N bytes from entering its internal queue, to limit the amount of concurrent work and overall resource usage by the rest of the pipeline.
jmacd pushed a commit that referenced this issue Nov 3, 2023
Splitting up #71 into 2
PR's. This first PR is just simply forking the batchprocessor into this
repository before applying further changes on top.

Part of #80.
jmacd pushed a commit that referenced this issue Nov 3, 2023
Part 2 for #80

This PR adds the batchprocessor to otel-arrow with a couple
enhancements:
- block requests until the batch response is received
- errors reported to producers; timeout will now report an error

The main files to review are `batch_processor.go` and
`batch_processor_test.go`
jmacd added a commit that referenced this issue Nov 8, 2023
Part 3 of #80

This PR:
- renames component to `concurrentbatchprocessor`
- when items are sent, `export` is called in a goroutine

---------

Co-authored-by: Joshua MacDonald <jmacd@users.noreply.github.com>
@jmacd jmacd closed this as completed in #93 Nov 17, 2023
jmacd added a commit that referenced this issue Nov 17, 2023
#93)

This PR adds a new config option in the `concurrentbatchprocessor` named
`max_in_flight_bytes`. This config option will be used by processor to
control admission (using a semaphore) into the producer queue based on
the size of requests.

Fixes #80.
Fixes #11.

---------

Co-authored-by: Joshua MacDonald <jmacd@users.noreply.github.com>
jmacd added a commit that referenced this issue Nov 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants