-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Closed
Labels
Description
Bug Report
Describe the bug
Logs with nested elements fail to be received when using OpenTelemetry over HTTP, but they are received correctly when using OpenTelemetry over gRPC.
To Reproduce
When we send the following log using OpenTelemetry gRPC (protos directory contains OTLP .proto files):
grpcurl -plaintext \
-import-path './protos' \
-proto './protos/opentelemetry/proto/collector/v1/logs/v1/logs_service.proto' \
-d '{
"resourceLogs": [
{
"resource": {},
"scopeLogs": [
{
"scope": {},
"logRecords": [
{
"observedTimeUnixNano": "1754059282910920618",
"body": {
"kvlistValue": {
"values": [
{
"key": "a",
"value": {
"kvlistValue": {
"values": [
{
"key": "b",
"value": {
"kvlistValue": {
"values": [
{
"key": "c",
"value": {
"kvlistValue": {
"values": [
{
"key": "d",
"value": {
"arrayValue": {
"values": [
{
"kvlistValue": {
"values": [
{
"key": "e",
"value": {
"kvlistValue": {
"values": [
{
"key": "f",
"value": {
"stringValue": "g"
}
}
]
}
}
}
]
}
}
]
}
}
}
]
}
}
}
]
}
}
}
]
}
}
}
]
}
}
}
]
}
]
}
]
}' \
localhost:4317 opentelemetry.proto.collector.logs.v1.LogsService/ExportEverything looks good and we successfully receive the message:
Fluent Bit v4.0.5
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
______ _ _ ______ _ _ ___ _____
| ___| | | | | ___ (_) | / || _ |
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' |
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| |
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ /
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/
[2025/08/01 17:01:49] [ info] Configuration:
[2025/08/01 17:01:49] [ info] flush time | 1.000000 seconds
[2025/08/01 17:01:49] [ info] grace | 5 seconds
[2025/08/01 17:01:49] [ info] daemon | 0
[2025/08/01 17:01:49] [ info] ___________
[2025/08/01 17:01:49] [ info] inputs:
[2025/08/01 17:01:49] [ info] opentelemetry
[2025/08/01 17:01:49] [ info] ___________
[2025/08/01 17:01:49] [ info] filters:
[2025/08/01 17:01:49] [ info] ___________
[2025/08/01 17:01:49] [ info] outputs:
[2025/08/01 17:01:49] [ info] stdout.0
[2025/08/01 17:01:49] [ info] ___________
[2025/08/01 17:01:49] [ info] collectors:
[2025/08/01 17:01:49] [ info] [fluent bit] version=4.0.5, commit=d9be919eab, pid=1
[2025/08/01 17:01:49] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2025/08/01 17:01:49] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/08/01 17:01:49] [ info] [simd ] NEON
[2025/08/01 17:01:49] [ info] [cmetrics] version=1.0.4
[2025/08/01 17:01:49] [ info] [ctraces ] version=0.6.6
[2025/08/01 17:01:49] [ info] [input:opentelemetry:opentelemetry.0] initializing
[2025/08/01 17:01:49] [ info] [input:opentelemetry:opentelemetry.0] storage_strategy='memory' (memory only)
[2025/08/01 17:01:49] [debug] [opentelemetry:opentelemetry.0] created event channels: read=25 write=26
[2025/08/01 17:01:49] [debug] [downstream] listening on 0.0.0.0:4317
[2025/08/01 17:01:49] [ info] [input:opentelemetry:opentelemetry.0] listening on 0.0.0.0:4317
[2025/08/01 17:01:49] [debug] [stdout:stdout.0] created event channels: read=28 write=29
[2025/08/01 17:01:49] [ info] [sp] stream processor started
[2025/08/01 17:01:49] [ info] [output:stdout:stdout.0] worker #0 started
[2025/08/01 17:01:49] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2025/08/01 17:02:01] [debug] [task] created task=0xe35bdee23640 id=0 OK
[2025/08/01 17:02:01] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
GROUP METADATA :
{"schema"=>"otlp", "resource_id"=>0, "scope_id"=>0}
GROUP ATTRIBUTES :
{"resource"=>{"schema_url"=>""}, "scope"=>{}}
[0] v1_logs: [[1754067721.821162193, {"otlp"=>{"observed_timestamp"=>1754059282910920618, "trace_flags"=>0}}], {"a"=>{"b"=>{"c"=>{"d"=>[{"e"=>{"f"=>"g"}}]}}}}]
[2025/08/01 17:02:01] [debug] [out flush] cb_destroy coro_id=0
[2025/08/01 17:02:01] [debug] [task] destroy task=0xe35bdee23640 (task_id=0)
But when we send the same log using OpenTelemetry HTTP:
curl --location 'http://localhost:4317/v1/logs' \
--header 'Content-Type: application/json' \
--data '{
"resourceLogs": [
{
"resource": {},
"scopeLogs": [
{
"scope": {},
"logRecords": [
{
"observedTimeUnixNano": "1754059282910920618",
"body": {
"kvlistValue": {
"values": [
{
"key": "a",
"value": {
"kvlistValue": {
"values": [
{
"key": "b",
"value": {
"kvlistValue": {
"values": [
{
"key": "c",
"value": {
"kvlistValue": {
"values": [
{
"key": "d",
"value": {
"arrayValue": {
"values": [
{
"kvlistValue": {
"values": [
{
"key": "e",
"value": {
"kvlistValue": {
"values": [
{
"key": "f",
"value": {
"stringValue": "g"
}
}
]
}
}
}
]
}
}
]
}
}
}
]
}
}
}
]
}
}
}
]
}
}
}
]
}
}
}
]
}
]
}
]
}'We receive HTTP 400 and see the following error:
Fluent Bit v4.0.5
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
______ _ _ ______ _ _ ___ _____
| ___| | | | | ___ (_) | / || _ |
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' |
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| |
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ /
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/
[2025/08/01 17:07:40] [ info] Configuration:
[2025/08/01 17:07:40] [ info] flush time | 1.000000 seconds
[2025/08/01 17:07:40] [ info] grace | 5 seconds
[2025/08/01 17:07:40] [ info] daemon | 0
[2025/08/01 17:07:40] [ info] ___________
[2025/08/01 17:07:40] [ info] inputs:
[2025/08/01 17:07:40] [ info] opentelemetry
[2025/08/01 17:07:40] [ info] ___________
[2025/08/01 17:07:40] [ info] filters:
[2025/08/01 17:07:40] [ info] ___________
[2025/08/01 17:07:40] [ info] outputs:
[2025/08/01 17:07:40] [ info] stdout.0
[2025/08/01 17:07:40] [ info] ___________
[2025/08/01 17:07:40] [ info] collectors:
[2025/08/01 17:07:40] [ info] [fluent bit] version=4.0.5, commit=d9be919eab, pid=1
[2025/08/01 17:07:40] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2025/08/01 17:07:40] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/08/01 17:07:40] [ info] [simd ] NEON
[2025/08/01 17:07:40] [ info] [cmetrics] version=1.0.4
[2025/08/01 17:07:40] [ info] [ctraces ] version=0.6.6
[2025/08/01 17:07:40] [ info] [input:opentelemetry:opentelemetry.0] initializing
[2025/08/01 17:07:40] [ info] [input:opentelemetry:opentelemetry.0] storage_strategy='memory' (memory only)
[2025/08/01 17:07:40] [debug] [opentelemetry:opentelemetry.0] created event channels: read=25 write=26
[2025/08/01 17:07:40] [debug] [downstream] listening on 0.0.0.0:4317
[2025/08/01 17:07:40] [ info] [input:opentelemetry:opentelemetry.0] listening on 0.0.0.0:4317
[2025/08/01 17:07:40] [debug] [stdout:stdout.0] created event channels: read=28 write=29
[2025/08/01 17:07:40] [ info] [sp] stream processor started
[2025/08/01 17:07:40] [ info] [output:stdout:stdout.0] worker #0 started
[2025/08/01 17:07:40] [ info] [engine] Shutdown Grace Period=5, Shutdown Input Grace Period=2
[2025/08/01 17:07:47] [error] [input:opentelemetry:opentelemetry.0] failed to process logs from JSON payload (2) FLB_OTEL_LOGS_ERR_UNEXPECTED_ROOT_OBJECT_TYPE
Expected behavior
Logs should be handled correctly over HTTP, just as they are over gRPC.
Screenshots
N/A
Your Environment
- Version used: v4.0.5
- Configuration:
service:
log_level: trace
pipeline:
inputs:
- name: opentelemetry
port: 4317
outputs:
- name: stdout
match: '*'- Environment name and version (e.g. Kubernetes? What version?): Docker
- Server type and version: N/A
- Operating System and version: fluent/fluent-bit:4.0.5
- Filters and plugins: built-in OpenTelemetry input plugin
Reactions are currently unavailable