New component: OpenSearch Exporter #23611

MitchellGale · 2023-06-21T17:59:14Z

The purpose and use-cases of the new component

OpenSearch is used for real-time application monitoring, log analytics, website search and more. The OpenSearch exporter would convert OpenTelemetry data directly into OpenSearch allowing logs, trace, metrics to be ingested and searched using OpenSearch. Exporting OpenTelemetry data into OpenSearch will help improve observability.

Many users are currently using OpenSearch as store repository for logs and need a common way and a standard protocol to consolidate both logs and trace signals.

Example configuration for the component

exporters:
  opensearch/trace:
    keyspace: "otel"
    endpoints: [https://localhost:9200]
    traces_index: trace_index
  opensearch/log:
    endpoints: [http://localhost:9200]
    logs_index: my_log_index
    sending_queue:
      enabled: true
      num_consumers: 20
      queue_size: 1000
······
service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [opensearch/log]
    traces:
      receivers: [otlp]
      exporters: [opensearch/trace]
      processors: [batch]

Telemetry data types supported

Traces
Logs

Is this a vendor-specific component?

This is a vendor-specific component
If this is a vendor-specific component, I am proposing to contribute this as a representative of the vendor.

Sponsor (optional)

Anthony Mirabella (Aneurysm9)

Additional context

Will start contributing with traces initially then follow up with logs.

OpenSearch has recently forked and showed the usage of OpenSearch as a backend store and exploration tools on top of the open telemetry astronomer shop demo using simple schema as the mapping physical store of the OTEL signals

https://github.com/opensearch-project/opentelemetry-demo
https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability
https://github.com/opensearch-project/opensearch-catalog/tree/main/docs/schema

I will be directly contributing to this. I am open to becoming a member of OpenTelemetry as progress on this exporter progresses.

The text was updated successfully, but these errors were encountered:

atoulme · 2023-06-27T15:43:42Z

@Aneurysm9 please comment if you are accepting to sponsor, and we can then label this issue as "Accepted component".

Aneurysm9 · 2023-06-27T22:51:10Z

Yes, I am sponsoring this component.

Adding initial set-up for OpenSearch exporter addition. Future PRs will include adding functionality to the exporter, code coverage and e2e tests. Broken up for easier review. [New component proposal.](#23611) Start of resolution of #7905. Will come in future PR to 80+% coverage. Break-up of #23045 --------- Signed-off-by: Max Ksyunz <max.ksyunz@improving.com>

github-actions · 2023-08-28T03:32:59Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

jaehnri · 2023-08-31T22:30:12Z

@MitchellGale @Aneurysm9, is log support already in progress?

If not, in rancher/opni, we have been using a forked opensearch log exporter for a while now. I'd be glad to contribute it (given the necessary changes, of course).

## Description: <Describe what has changed.>  Implementation of exporter to OpenSearch using opensearch-go library. As of now, this PR was heavily inspired by https://github.com/dbason/opentelemetry-collector-contrib/tree/opensearch-exporter/exporter/opensearchexporter. By default, requests sent adhere to the OpenSearch Catalog [schema for logs](https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability/logs), but allows users to export using the Elastic Common Schema as well. This PR also: - enables users to define the `bulk_action` between `create` and `index` - enables users to define the logs index without necessarily adhering to the new [index naming conventions](opensearch-project/observability#1405) through the `LogsIndex` config. ## Tracking Issue: [23611](#23611) ## Testing: <Describe what testing was performed and which tests were added.> ### Integration - Successful round-trip to HTTP endpoint, - Permanent error during round-trip, - Retryable error response for first request, followed by successful response on retry, - Two retriable error responses, followed by successful response on second retry. ### Manual - Authentication using `configtls.TLSSetting` (`ca_file`, `cert_file`, `key_file`) - Tested in EKS and K3s clusters running [opni](https://github.com/rancher/opni). --------- Signed-off-by: João Henri <joao.rocha@suse.com> Signed-off-by: João Henri <joao.henri@suse.com>

github-actions · 2023-11-01T03:30:14Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

## Description: <Describe what has changed.>  Implementation of exporter to OpenSearch using opensearch-go library. As of now, this PR was heavily inspired by https://github.com/dbason/opentelemetry-collector-contrib/tree/opensearch-exporter/exporter/opensearchexporter. By default, requests sent adhere to the OpenSearch Catalog [schema for logs](https://github.com/opensearch-project/opensearch-catalog/tree/main/schema/observability/logs), but allows users to export using the Elastic Common Schema as well. This PR also: - enables users to define the `bulk_action` between `create` and `index` - enables users to define the logs index without necessarily adhering to the new [index naming conventions](opensearch-project/observability#1405) through the `LogsIndex` config. ## Tracking Issue: [23611](open-telemetry#23611) ## Testing: <Describe what testing was performed and which tests were added.> ### Integration - Successful round-trip to HTTP endpoint, - Permanent error during round-trip, - Retryable error response for first request, followed by successful response on retry, - Two retriable error responses, followed by successful response on second retry. ### Manual - Authentication using `configtls.TLSSetting` (`ca_file`, `cert_file`, `key_file`) - Tested in EKS and K3s clusters running [opni](https://github.com/rancher/opni). --------- Signed-off-by: João Henri <joao.rocha@suse.com> Signed-off-by: João Henri <joao.henri@suse.com>

github-actions · 2024-01-01T03:30:18Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

atoulme · 2024-01-02T06:08:38Z

Can we close this as done? I see the component code in contrib and it was added to the contrib distro with open-telemetry/opentelemetry-collector-releases#434

MitchellGale · 2024-01-02T16:20:06Z

Can we close this as done? I see the component code in contrib and it was added to the contrib distro with open-telemetry/opentelemetry-collector-releases#434

Yes @atoulme, I think it's reasonable to close as completed :). Thanks!

MitchellGale added the needs triage New item requiring triage label Jun 21, 2023

codeboten added Sponsor Needed New component seeking sponsor and removed Sponsor Needed New component seeking sponsor labels Jun 21, 2023

atoulme added Sponsor Needed New component seeking sponsor and removed needs triage New item requiring triage labels Jun 27, 2023

Aneurysm9 added Accepted Component New component has been sponsored and removed Sponsor Needed New component seeking sponsor labels Jun 27, 2023

MitchellGale mentioned this issue Jun 28, 2023

[Exporter/OpenSearch] OpenSearch exporter setup #23819

Merged

jaehnri mentioned this issue Aug 17, 2023

Opni logging should collect traces rancher/opni#1605

Closed

github-actions bot added the Stale label Aug 28, 2023

github-actions bot removed the Stale label Sep 1, 2023

This was referenced Sep 1, 2023

[exporter/opensearch] Schema fix in sso_model #26372

Closed

[chore] Add docker demo to opensearchexporter Bit-Quill/opentelemetry-collector-contrib#26

Draft

jaehnri mentioned this issue Sep 5, 2023

[exporter/opensearch] Send logs to Opensearch #26475

Merged

confiq mentioned this issue Sep 11, 2023

can't decode "opensearch" component #26560

Closed

github-actions bot added the Stale label Nov 1, 2023

atoulme removed the Stale label Nov 1, 2023

github-actions bot added the Stale label Jan 1, 2024

MitchellGale closed this as completed Jan 2, 2024

edmondsiu0 mentioned this issue Jan 15, 2024

Exporter Feature: OpenSearch Metrics #30556

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New component: OpenSearch Exporter #23611

New component: OpenSearch Exporter #23611

MitchellGale commented Jun 21, 2023 •

edited by Aneurysm9

Loading

atoulme commented Jun 27, 2023

Aneurysm9 commented Jun 27, 2023

github-actions bot commented Aug 28, 2023

jaehnri commented Aug 31, 2023 •

edited

Loading

github-actions bot commented Nov 1, 2023

github-actions bot commented Jan 1, 2024

atoulme commented Jan 2, 2024

MitchellGale commented Jan 2, 2024

New component: OpenSearch Exporter #23611

New component: OpenSearch Exporter #23611

Comments

MitchellGale commented Jun 21, 2023 • edited by Aneurysm9 Loading

The purpose and use-cases of the new component

Example configuration for the component

Telemetry data types supported

Is this a vendor-specific component?

Sponsor (optional)

Additional context

atoulme commented Jun 27, 2023

Aneurysm9 commented Jun 27, 2023

github-actions bot commented Aug 28, 2023

jaehnri commented Aug 31, 2023 • edited Loading

github-actions bot commented Nov 1, 2023

github-actions bot commented Jan 1, 2024

atoulme commented Jan 2, 2024

MitchellGale commented Jan 2, 2024

MitchellGale commented Jun 21, 2023 •

edited by Aneurysm9

Loading

jaehnri commented Aug 31, 2023 •

edited

Loading