-
Notifications
You must be signed in to change notification settings - Fork 16
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Develop a concurrent memory-limited batch processor #80
Comments
jmacd
pushed a commit
that referenced
this issue
Nov 3, 2023
jmacd
pushed a commit
that referenced
this issue
Nov 3, 2023
Part 2 for #80 This PR adds the batchprocessor to otel-arrow with a couple enhancements: - block requests until the batch response is received - errors reported to producers; timeout will now report an error The main files to review are `batch_processor.go` and `batch_processor_test.go`
jmacd
added a commit
that referenced
this issue
Nov 8, 2023
Part 3 of #80 This PR: - renames component to `concurrentbatchprocessor` - when items are sent, `export` is called in a goroutine --------- Co-authored-by: Joshua MacDonald <jmacd@users.noreply.github.com>
jmacd
added a commit
that referenced
this issue
Nov 17, 2023
#93) This PR adds a new config option in the `concurrentbatchprocessor` named `max_in_flight_bytes`. This config option will be used by processor to control admission (using a semaphore) into the producer queue based on the size of requests. Fixes #80. Fixes #11. --------- Co-authored-by: Joshua MacDonald <jmacd@users.noreply.github.com>
jmacd
added a commit
that referenced
this issue
Nov 17, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Briefly: we have had trouble tuning the configuration of an OTel-Arrow pipeline so that data does not drop. The root cause of this is the
exporterhelper
configuration and concurrency behavior combined with the corebatchprocessor
configuration and concurrency behavior.We will develop an alternative based on the the core
batchprocessor
by forking the code, and then:The text was updated successfully, but these errors were encountered: