Unable to find complete exported spans in OpenSearch Backend

[charan906]

I have generated 20000 spans , spans are received by Open telemetry collector, but during export when i checked logs of collector pod every times it got struct at particular span number , with thisIi cannot see complete span data in the backend

this is the configuration i am using , with otel_version="v0.107.0" and its a Customized binary its contains both core and contrib repository plugins

configuration:
exporters:
debug:
verbosity: detailed
opensearch:
http:
endpoint: ${env:SE_SERVER_URLS}
tls:
ca_file: ${env:ROOT_CA_CERT}
cert_file: ${env:CLIENT_CRT}
key_file: ${env:CLIENT_KEY}
extensions:
memory_ballast: {}
health_check:
endpoint: ${env:MY_POD_IP}:13133
jaegerremotesampling:
source:
reload_interval: 0s
file: /etc/sampling/samplingstrategies.json
processors:
batch: {}
memory_limiter:
# Check_interval is the time between measurements of memory usage.
check_interval: 5s
# By default limit_mib is set to 80% of ".Values.resources.limits.memory"
limit_percentage: 80
# By default spike_limit_mib is set to 25% of ".Values.resources.limits.memory"
spike_limit_percentage: 25
probabilistic_sampler:
sampling_percentage: 100
receivers:
otlp:
protocols:
grpc:
endpoint: ${env:MY_POD_IP}:4317
tls:
cert_file: ${env:SERVER_CRT}
key_file: ${env:SERVER_KEY}
http:
endpoint: ${env:MY_POD_IP}:4318
tls:
cert_file: ${env:SERVER_CRT}
key_file: ${env:SERVER_KEY}
service:
extensions:
- memory_ballast
- health_check
- jaegerremotesampling
pipelines:
traces:
exporters:
- debug
- opensearch
processors:
- memory_limiter
- batch
- probabilistic_sampler
receivers:
- otlp

Kubernetes resources specifications:

resources:
telemetry-collector:
requests:
memory: 64Mi
cpu: 250m
limits:
memory: 128Mi
cpu: 500m

Server starting logs:

2024-10-30T12:15:48.331Z info memorylimiter/memorylimiter.go:151 Using percentage memory limiter {"kind": "processor", "name": "memory_limiter", "pipeline": "traces", "total_memory_mib": 128, "limit_percentage": 80, "spike_limit_percentage": 25}
2024-10-30T12:15:48.331Z info memorylimiter/memorylimiter.go:75 Memory limiter configured {"kind": "processor", "name": "memory_limiter", "pipeline": "traces", "limit_mib": 102, "spike_limit_mib": 32, "check_interval": 5}
2024-10-30T12:15:48.333Z info service@v0.107.0/service.go:195 Starting otelcol-custom... {"Version": "1.0.0", "NumCPU": 8}

[codeboten]

This appears to possibly be an issue with the opensearch exporter, transferring to the contrib repo where that exporter lives

[github-actions]

Pinging code owners for exporter/opensearch: @Aneurysm9 @MitchellGale @MaxKsyunz @YANG-DB. See Adding Labels via Comments if you do not have permissions to add labels yourself.

[YANG-DB]

@codeboten we are reviewing this issue and will respond shortly

[charan906]

Hello @codeboten any findings on this issue .

[ps48]

Hi @charan906,

We’re currently unable to replicate this issue on our side. We tested using the latest build, but it doesn’t seem like there have been recent changes to the OpenSearch exporter. Could you please help us by providing more details? Specifically:

• You mentioned that the pod gets stuck at a particular span number—could you let us know what that number is?
• Are there any errors in the OpenSearch logs or in the Collector pod’s error logs that might help us debug the issue?

I’m not entirely familiar with the chunking mechanism used by the OpenSearch exporter for bulk requests, but it seems like a potential factor. Additionally, have you observed the same issue when the following processor configuration is removed?

memory_limiter:
  # Interval between memory usage checks
  check_interval: 5s
  # Default limit is 80% of ".Values.resources.limits.memory"
  limit_percentage: 80
  # Default spike limit is 25% of ".Values.resources.limits.memory"
  spike_limit_percentage: 25

probabilistic_sampler:
  sampling_percentage: 100

Thanks!