-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce new exporter helper with batching option #8122
Comments
Introduce a new exporter helper that operates over client-provided requests instead of pdata. The helper user now has to provide `Converter` - an interface with a function implementing translation of pdata Metrics/Traces/Logs into a user-defined `Request`. `Request` is an interface with only one required function `Export`. It opens a door for moving batching to the exporter, where batches will be built from client data format, instead of pdata. The batches can be properly sized by custom request size, which can be different from OTLP. The same custom request sizing will be applied to the sending queue. It will also improve the performance of the sending queue retries for non-OTLP exporters, they don't need to translate pdata on every retry. This is an implementation alternative to #7874 as suggested in #7874 (comment) Tracking Issue: #8122 --------- Co-authored-by: Alex Boten <alex@boten.ca>
This change adds collector's internal metrics and tracing to the new request-based exporter helpers. Only those metrics and traces are added that are already adopted by the existing exporter helpers for backward compatibility. The new exporter helpers can and should expose more metrics in the future, e.g. for tracking converter errors. Tracking Issue: #8122
Some feedback about the interface:
|
Thanks for the feedback!
That's what I wanted to do after #8248 is merged 👍 |
…on (#8764) As proposed in #8122 (comment) If we need backward conversion, we will use an optional argument to the helper function instead of an optional interface.
#9164) Introduce an option to limit the queue size by the number of items instead of number of requests. This is preliminary step for having the exporter helper v2 with a batcher sender placed after the queue sender. Otherwise, it'll be hard for the users to estimate the queue size based on the number of requests without batch processor in front of it. This change doesn't effect the existing functionality and the items based queue limiting cannot be utilized yet. Updates #8122 Alternative to #9147
Introduce a way to enable queue in the new exporter helper with a developer interface suggested in #8248 (comment). The new configuration interface for the end users provides a new `queue_size_items` option to limit the queue by a number of spans, log records, or metric data points. The previous way to limit the queue by number of requests is preserved under the same field, `queue_size,` which will later be deprecated through a longer transition process. Tracking issue: #8122
This change introduces new experimental batching functionality to the exporter helper. The batch sender is fully concurrent and synchronous. It's set after the queue sender, which, if enabled, introduces the asynchronous behavior and ensures no data loss with the permanent queue. Follow-up TODO list: - Add pre-built merge funcs for pdata - Handle partial errors - A missing part compared to the batch processor is the ability to shard the batches by context value. Updates #8122
Introduce default batching functionality based on the internal data type (pdata). This makes the exporter batching capability available to the regular exporter helpers without using custom requests. Updates #8122
@verejoel, the batch sharding will be added for parity with the batch processor. But there are no plans to have separate sender pipelines. The sharding will ensure that one batch won't have data with different metadata values but the batches will be processed by one pipeline. |
To properly support multi-tenant pipeline, having a sender queue per batch will be critical. Otherwise, we can still have batches from tenants with back-pressure blocking the pipeline for all tenants attempting to ship data. Can we work in sharded queues as part of this feature? Or should I write it down as a separate feature request? |
@dmitryax Thanks for the batch sender! I've been experimenting with it lately. In my experiments, the batch sender is producing inconsistent batch sizes which could be lower than desired due to goroutine scheduling even after #9761 . The scenario I usually run into is that given queue sender concurrency = batch sender concurrency limit = N, and they are all blocked on I tried to fix it by resetting On the performance side of things, I haven't benchmarked it, but I'm not sure how performant it is when queue consumer concurrency is high (e.g. 1000+) given the number of running goroutines. I understand that spawning goroutines and blocking them may be the only way to avoid deleting the entry in persistent queue before request export. I wish I could give some useful suggestions here, but I don't have a good idea within the current paradigm. Have you considered the limitations I mentioned, and are there bigger changes planned ahead? edit: I'm starting to think, will it be fine if the batch sender waits up to FlushTimeout even when concurrency limit is reached when not all goroutines are in the same batch? |
batch size & concurrency
I've drafted a PR to do so, and feedback is welcome: #9891 The issue is captured in #9952 batching based on bytes
I see that batching based on size rather than item count is planned but apparently not implemented yet. Is that still on the roadmap? |
The counting of batch sizes in bytes is something that would be super useful for us, since some of our backends limit us on payload size in bytes. @dmitryax let me know if there's anything I can do to help push that forward, would be happy to help out. |
Add opt-in support for the experimental batcher helper - open-telemetry/opentelemetry-collector#8122 When opting into this functionality the exporter's Consume* methods will make synchronous bulk requests to Elasticsearch, without additional batching/buffering in the exporter.
**Description:** Add opt-in support for the experimental batch sender (open-telemetry/opentelemetry-collector#8122). When opting into this functionality the exporter's `Consume*` methods will make synchronous bulk requests to Elasticsearch, without additional batching/buffering in the exporter. By default the exporter continues to use its own buffering, which supports thresholds for time, number of documents, and size of encoded documents in bytes. The batch sender does not currently support a bytes-based threshold, and is experimental, hence why we are not yet making it the default for the Elasticsearch exporter. This PR is based on #32632, but made to be non-breaking. **Link to tracking Issue:** #32377 **Testing:** Added unit and integration tests. Manually tested with Elasticsearch, using the following config to highlight that client metadata can now flow through all the way: ```yaml receivers: otlp: protocols: http: endpoint: 0.0.0.0:4318 include_metadata: true exporters: elasticsearch: endpoint: "http://localhost:9200" auth: authenticator: headers_setter batcher: enabled: false extensions: headers_setter: headers: - action: insert key: Authorization from_context: authorization service: extensions: [headers_setter] pipelines: traces: receivers: [otlp] processors: [] exporters: [elasticsearch] ``` I have Elasticsearch running locally, with an "admin" user with the password "changeme". Sending OTLP/HTTP to the collector with `telemetrygen traces --otlp-http --otlp-insecure http://localhost:4318 --otlp-header "Authorization=\"Basic YWRtaW46Y2hhbmdlbWU=\""`, I observe the following: - Without the `batcher` config, the exporter fails to index data into Elasticsearch due to an auth error. That's because the exporter is buffering and dropping the context with client metadata, so there's no Authorization header attached to the requests going out. - With `batcher: {enabled: true}`, same behaviour as above. Unlike the [`batch` processor](https://github.com/open-telemetry/opentelemetry-collector/tree/main/processor/batchprocessor), the batch sender does not maintain client metadata. - With `batcher: {enabled: false}`, the exporter successfully indexes data into Elasticsearch. **Documentation:** Updated the README. --------- Co-authored-by: Carson Ip <carsonip@users.noreply.github.com>
As discussed in today's Collector SIG meeting, we (Elastic) have the same requirement as @verejoel described in #8122 (comment). We intend to support a multi-tenant collector that receives data from different tenants, and sends the data to a tenant-specific Elasticsearch cluster. For performance reasons we should batch the data before exporting to Elasticsearch, which must necessarily be done per tenant. Although partitioning may be done by the batch sender (#10825), each backend instance (in our case, Elasticsearch cluster) may have different performance (different load, different scaling, ...) or availability; with a single queue, this may lead to head-of-line blocking. |
#### Description This PR adds opt-in support to oltp exporter for the experimental batch sender (#8122). By default batch sender is not enabled. Similar: * open-telemetry/opentelemetry-collector-contrib#34238 * open-telemetry/opentelemetry-collector-contrib#32563 #### Link to tracking issue Resolves #10834 #### Testing `exporter/otlpexporter/config_test.go` #### Documentation Updated the `oltpexporter` README to point to `exporterhelper` README for batching. --------- Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>
This is a tracking issue for introducing the new exporter helper and migrating the existing exporters to use it.
The primary reason for introducing the new exporter helper is to move the batching to the exporter side and deprecate the batch processor as part of making the delivery pipeline reliable, as reported in #7460. More details about moving batching to the exporter helper can be found in #4646.
Shifting batching to the exporter side grants us the opportunity to leverage the exporter's data model instead of relying on OTLP. As a result, we can achieve the following benefits:
Adapting to the new exporter helper requires exporter developers to implement at least two functions:
Design document: https://docs.google.com/document/d/1uxnn5rMHhCBLP1s8K0Pg_1mAs4gCeny8OWaYvWcuibs
Essential sub-issues to mark this as complete:
Additional sub-issues to get feature parity with the batch processor:
The text was updated successfully, but these errors were encountered: