request help: opentelemetry-plugin report no data to otel-collecotor

[xuminwlt]

Issue description

I see the opentelemetry-plugin is merge to branch master, and I try to test it.
use otel-collecotor 4317 port, when i make run with plugin config like this, is somewhere a bug?
BTW, the java service send trace data success, so it is not the collector question.

plugins:                          # plugin list (sorted by priority)
  - opentelemetry                 # priority: -1200

plugin_attr:
  opentelemetry:
    trace_id_source: x-request-id
    resource:
      service.name: APISIX
    collector:
      address: 127.0.0.1:4317
      request_timeout: 3
    batch_span_processor:
      drop_on_queue_full: false
      max_queue_size: 1024
      batch_timeout: 2
      inactive_timeout: 1
      max_export_batch_size: 2

In error.log, I have got error in error.log.

2022/02/15 22:11:23 [error] 73677#10524709: *337752 [lua] http_client.lua:41: do_request(): request failed: closed, context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:9080
2022/02/15 22:11:23 [error] 73677#10524709: *337752 lua entry thread aborted: runtime error: ...deps/share/lua/5.1/opentelemetry/trace/exporter/otlp.lua:24: assertion failed!
stack traceback:
coroutine 0:
	[C]: in function 'assert'
	...deps/share/lua/5.1/opentelemetry/trace/exporter/otlp.lua:24: in function 'export_spans'
	...are/lua/5.1/opentelemetry/trace/batch_span_processor.lua:18: in function 'process_batches'
	...are/lua/5.1/opentelemetry/trace/batch_span_processor.lua:46: in function <...are/lua/5.1/opentelemetry/trace/batch_span_processor.lua:29>, context: ngx.timer, client: 127.0.0.1, server: 0.0.0.0:9080

this plugin fail send trace data to otel-collector.

Environment

• apisix version (cmd: apisix version): 2.12.0
• OS (cmd: uname -a): centos
• OpenResty / Nginx version (cmd: nginx -V or openresty -V):
• etcd version, if have (cmd: run curl http://127.0.0.1:9090/v1/server_info to get the info from server-info API):
• apisix-dashboard version, if have:
• the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner):
• luarocks version, if the issue is about installation (cmd: luarocks --version):

@yangxikun please have a look, brother.

[yangxikun]

The error log seems maybe there is a bug at: https://github.com/yangxikun/opentelemetry-lua/blob/6c4a5112de5f70bb94a87e39656121f9c7407d95/lib/opentelemetry/trace/batch_span_processor.lua#L37

Could you try replace /usr/local/apisix/deps/share/lua/5.1/opentelemetry/trace/batch_span_processor.lua#L37 to:

if now() - self.first_queue_t >= self.batch_timeout and #self.queue > 0 then

then restart apisix container and test if the error log still appear.

BTW, the above bug shouldn't affect sending spans. Does your otel-collecotor 4317 port use http protocol? otel-collector-config.yaml just like:

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4317

[xuminwlt]

Thanks @yangxikun , i'll try these right now.

[xuminwlt]

Success, trace data is send to otel-collector.

But, I have a question, the default config of the otel-collector port like this:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

default config

 otlp:
    protocols:
      grpc: # on port 4317
      http: # on port 4318

So i change the port like this:

plugin_attr:
  opentelemetry:
    trace_id_source: x-request-id
    resource:
      service.name: APISIX
    collector:
      address: 127.0.0.1:4318  # ← success
      request_timeout: 3
    batch_span_processor:
      drop_on_queue_full: false
      max_queue_size: 1024
      batch_timeout: 2
      inactive_timeout: 1
      max_export_batch_size: 2

no other question, bang bang da.

[yangxikun]

Great, I will make a PR to fix the error log.