Skip to content

Logs with nested elements not received via OpenTelemetry HTTP #10673

@mkuzdzal

Description

@mkuzdzal

Bug Report

Describe the bug
Logs with nested elements fail to be received when using OpenTelemetry over HTTP, but they are received correctly when using OpenTelemetry over gRPC.

To Reproduce
When we send the following log using OpenTelemetry gRPC (protos directory contains OTLP .proto files):

grpcurl -plaintext \
-import-path './protos' \
-proto './protos/opentelemetry/proto/collector/v1/logs/v1/logs_service.proto' \
-d '{
    "resourceLogs": [
        {
            "resource": {},
            "scopeLogs": [
                {
                    "scope": {},
                    "logRecords": [
                        {
                            "observedTimeUnixNano": "1754059282910920618",
                            "body": {
                                "kvlistValue": {
                                    "values": [
                                        {
                                            "key": "a",
                                            "value": {
                                                "kvlistValue": {
                                                    "values": [
                                                        {
                                                            "key": "b",
                                                            "value": {
                                                                "kvlistValue": {
                                                                    "values": [
                                                                        {
                                                                            "key": "c",
                                                                            "value": {
                                                                                "kvlistValue": {
                                                                                    "values": [
                                                                                        {
                                                                                            "key": "d",
                                                                                            "value": {
                                                                                                "arrayValue": {
                                                                                                    "values": [
                                                                                                        {
                                                                                                            "kvlistValue": {
                                                                                                                "values": [
                                                                                                                    {
                                                                                                                        "key": "e",
                                                                                                                        "value": {
                                                                                                                            "kvlistValue": {
                                                                                                                                "values": [
                                                                                                                                    {
                                                                                                                                        "key": "f",
                                                                                                                                        "value": {
                                                                                                                                            "stringValue": "g"
                                                                                                                                        }
                                                                                                                                    }
                                                                                                                                ]
                                                                                                                            }
                                                                                                                        }
                                                                                                                    }
                                                                                                                ]
                                                                                                            }
                                                                                                        }
                                                                                                    ]
                                                                                                }
                                                                                            }
                                                                                        }
                                                                                    ]
                                                                                }
                                                                            }
                                                                        }
                                                                    ]
                                                                }
                                                            }
                                                        }
                                                    ]
                                                }
                                            }
                                        }
                                    ]
                                }
                            }
                        }
                    ]
                }
            ]
        }
    ]
}' \
localhost:4317 opentelemetry.proto.collector.logs.v1.LogsService/Export

Everything looks good and we successfully receive the message:

Fluent Bit v4.0.5
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___  _____
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/


[2025/08/01 17:01:49] [ info] Configuration:
[2025/08/01 17:01:49] [ info]  flush time     | 1.000000 seconds
[2025/08/01 17:01:49] [ info]  grace          | 5 seconds
[2025/08/01 17:01:49] [ info]  daemon         | 0
[2025/08/01 17:01:49] [ info] ___________
[2025/08/01 17:01:49] [ info]  inputs:
[2025/08/01 17:01:49] [ info]      opentelemetry
[2025/08/01 17:01:49] [ info] ___________
[2025/08/01 17:01:49] [ info]  filters:
[2025/08/01 17:01:49] [ info] ___________
[2025/08/01 17:01:49] [ info]  outputs:
[2025/08/01 17:01:49] [ info]      stdout.0
[2025/08/01 17:01:49] [ info] ___________
[2025/08/01 17:01:49] [ info]  collectors:
[2025/08/01 17:01:49] [ info] [fluent bit] version=4.0.5, commit=d9be919eab, pid=1
[2025/08/01 17:01:49] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2025/08/01 17:01:49] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/08/01 17:01:49] [ info] [simd    ] NEON
[2025/08/01 17:01:49] [ info] [cmetrics] version=1.0.4
[2025/08/01 17:01:49] [ info] [ctraces ] version=0.6.6
[2025/08/01 17:01:49] [ info] [input:opentelemetry:opentelemetry.0] initializing
[2025/08/01 17:01:49] [ info] [input:opentelemetry:opentelemetry.0] storage_strategy='memory' (memory only)
[2025/08/01 17:01:49] [debug] [opentelemetry:opentelemetry.0] created event channels: read=25 write=26
[2025/08/01 17:01:49] [debug] [downstream] listening on 0.0.0.0:4317
[2025/08/01 17:01:49] [ info] [input:opentelemetry:opentelemetry.0] listening on 0.0.0.0:4317
[2025/08/01 17:01:49] [debug] [stdout:stdout.0] created event channels: read=28 write=29
[2025/08/01 17:01:49] [ info] [sp] stream processor started
[2025/08/01 17:01:49] [ info] [output:stdout:stdout.0] worker #0 started
[2025/08/01 17:01:49] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2025/08/01 17:02:01] [debug] [task] created task=0xe35bdee23640 id=0 OK
[2025/08/01 17:02:01] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
GROUP METADATA :

{"schema"=>"otlp", "resource_id"=>0, "scope_id"=>0}

GROUP ATTRIBUTES :

{"resource"=>{"schema_url"=>""}, "scope"=>{}}

[0] v1_logs: [[1754067721.821162193, {"otlp"=>{"observed_timestamp"=>1754059282910920618, "trace_flags"=>0}}], {"a"=>{"b"=>{"c"=>{"d"=>[{"e"=>{"f"=>"g"}}]}}}}]
[2025/08/01 17:02:01] [debug] [out flush] cb_destroy coro_id=0
[2025/08/01 17:02:01] [debug] [task] destroy task=0xe35bdee23640 (task_id=0)

But when we send the same log using OpenTelemetry HTTP:

curl --location 'http://localhost:4317/v1/logs' \
--header 'Content-Type: application/json' \
--data '{
    "resourceLogs": [
        {
            "resource": {},
            "scopeLogs": [
                {
                    "scope": {},
                    "logRecords": [
                        {
                            "observedTimeUnixNano": "1754059282910920618",
                            "body": {
                                "kvlistValue": {
                                    "values": [
                                        {
                                            "key": "a",
                                            "value": {
                                                "kvlistValue": {
                                                    "values": [
                                                        {
                                                            "key": "b",
                                                            "value": {
                                                                "kvlistValue": {
                                                                    "values": [
                                                                        {
                                                                            "key": "c",
                                                                            "value": {
                                                                                "kvlistValue": {
                                                                                    "values": [
                                                                                        {
                                                                                            "key": "d",
                                                                                            "value": {
                                                                                                "arrayValue": {
                                                                                                    "values": [
                                                                                                        {
                                                                                                            "kvlistValue": {
                                                                                                                "values": [
                                                                                                                    {
                                                                                                                        "key": "e",
                                                                                                                        "value": {
                                                                                                                            "kvlistValue": {
                                                                                                                                "values": [
                                                                                                                                    {
                                                                                                                                        "key": "f",
                                                                                                                                        "value": {
                                                                                                                                            "stringValue": "g"
                                                                                                                                        }
                                                                                                                                    }
                                                                                                                                ]
                                                                                                                            }
                                                                                                                        }
                                                                                                                    }
                                                                                                                ]
                                                                                                            }
                                                                                                        }
                                                                                                    ]
                                                                                                }
                                                                                            }
                                                                                        }
                                                                                    ]
                                                                                }
                                                                            }
                                                                        }
                                                                    ]
                                                                }
                                                            }
                                                        }
                                                    ]
                                                }
                                            }
                                        }
                                    ]
                                }
                            }
                        }
                    ]
                }
            ]
        }
    ]
}'

We receive HTTP 400 and see the following error:

Fluent Bit v4.0.5
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___  _____
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/


[2025/08/01 17:07:40] [ info] Configuration:
[2025/08/01 17:07:40] [ info]  flush time     | 1.000000 seconds
[2025/08/01 17:07:40] [ info]  grace          | 5 seconds
[2025/08/01 17:07:40] [ info]  daemon         | 0
[2025/08/01 17:07:40] [ info] ___________
[2025/08/01 17:07:40] [ info]  inputs:
[2025/08/01 17:07:40] [ info]      opentelemetry
[2025/08/01 17:07:40] [ info] ___________
[2025/08/01 17:07:40] [ info]  filters:
[2025/08/01 17:07:40] [ info] ___________
[2025/08/01 17:07:40] [ info]  outputs:
[2025/08/01 17:07:40] [ info]      stdout.0
[2025/08/01 17:07:40] [ info] ___________
[2025/08/01 17:07:40] [ info]  collectors:
[2025/08/01 17:07:40] [ info] [fluent bit] version=4.0.5, commit=d9be919eab, pid=1
[2025/08/01 17:07:40] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2025/08/01 17:07:40] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/08/01 17:07:40] [ info] [simd    ] NEON
[2025/08/01 17:07:40] [ info] [cmetrics] version=1.0.4
[2025/08/01 17:07:40] [ info] [ctraces ] version=0.6.6
[2025/08/01 17:07:40] [ info] [input:opentelemetry:opentelemetry.0] initializing
[2025/08/01 17:07:40] [ info] [input:opentelemetry:opentelemetry.0] storage_strategy='memory' (memory only)
[2025/08/01 17:07:40] [debug] [opentelemetry:opentelemetry.0] created event channels: read=25 write=26
[2025/08/01 17:07:40] [debug] [downstream] listening on 0.0.0.0:4317
[2025/08/01 17:07:40] [ info] [input:opentelemetry:opentelemetry.0] listening on 0.0.0.0:4317
[2025/08/01 17:07:40] [debug] [stdout:stdout.0] created event channels: read=28 write=29
[2025/08/01 17:07:40] [ info] [sp] stream processor started
[2025/08/01 17:07:40] [ info] [output:stdout:stdout.0] worker #0 started
[2025/08/01 17:07:40] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2025/08/01 17:07:47] [error] [input:opentelemetry:opentelemetry.0] failed to process logs from JSON payload (2) FLB_OTEL_LOGS_ERR_UNEXPECTED_ROOT_OBJECT_TYPE

Expected behavior
Logs should be handled correctly over HTTP, just as they are over gRPC.

Screenshots
N/A

Your Environment

  • Version used: v4.0.5
  • Configuration:
service:
  log_level: trace

pipeline:
  inputs:
    - name: opentelemetry
      port: 4317

  outputs:
    - name: stdout
      match: '*'
  • Environment name and version (e.g. Kubernetes? What version?): Docker
  • Server type and version: N/A
  • Operating System and version: fluent/fluent-bit:4.0.5
  • Filters and plugins: built-in OpenTelemetry input plugin

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions