Release 3.2 of fluent-bit crashes on azure-blob support

[GottfriedGanssauge]

Bug Report

Describe the bug
Since version 3.2.0 fluent-bit crashes with SIGSEGV on output to azure blob storage and thus is going into a crash-loop.
With Version 3.1.10 fluent-bit behaves normal.

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

Fluent Bit v3.2.2
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____ 
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /  
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2024/12/02 17:25:00] [ info] [fluent bit] version=3.2.2, commit=a59c867924, pid=1
[2024/12/02 17:25:00] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/12/02 17:25:00] [ info] [simd    ] disabled
[2024/12/02 17:25:00] [ info] [cmetrics] version=0.9.9
[2024/12/02 17:25:00] [ info] [ctraces ] version=0.5.7
[2024/12/02 17:25:00] [ info] [input:tail:tail.0] initializing
[2024/12/02 17:25:00] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/12/02 17:25:00] [ info] [input:tail:tail.0] db: delete unmonitored stale inodes from the database: count=0
[2024/12/02 17:25:00] [ info] [input:tail:tail.1] initializing
[2024/12/02 17:25:00] [ info] [input:tail:tail.1] storage_strategy='memory' (memory only)
[2024/12/02 17:25:00] [ info] [input:tail:tail.1] db: delete unmonitored stale inodes from the database: count=0
[2024/12/02 17:25:00] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc.cluster.local port=443
[2024/12/02 17:25:00] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2024/12/02 17:25:00] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2024/12/02 17:25:00] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2024/12/02 17:25:00] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2024/12/02 17:25:00] [ info] [output:es:es.0] worker #0 started
[2024/12/02 17:25:00] [ info] [output:azure_blob:azure_blob.1] account_name=logstideskp2, container_name=access-logs, blob_type=blockblob, emulator_mode=no, endpoint=logstideskp2.blob.core.windows.net, auth_type=key
[2024/12/02 17:25:00] [ info] [output:es:es.0] worker #1 started
[2024/12/02 17:25:00] [ info] [output:azure_blob:azure_blob.1] initializing worker
[2024/12/02 17:25:00] [ info] [output:azure_blob:azure_blob.1] worker #0 started
[2024/12/02 17:25:00] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/12/02 17:25:00] [ info] [sp] stream processor started
[2024/12/02 17:25:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=585645 watch_fd=1 name=/var/log/containers/aurora-letsencrypt-cert-manager-cainjector-77547cd645-d5rs5_cert-manager_cert-manager-cainjector-86b5ec5a32f3b6bb3279539443bc6d1d1a162f397b75aed1b2a691e85e60c841.log
[2024/12/02 17:25:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=579291 watch_fd=2 name=/var/log/containers/crs-559445b4f5-m9988_default_init-crs-ba7349072cf9474905a8d3eecf12637f07a877233e98287bb367933e275e28e0.log
[2024/12/02 17:25:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=586171 watch_fd=3 name=/var/log/containers/elastic-operator-0_elastic-system_manager-c5c443982a1d231b8b933293ad431f5b2fc13868ed17f44563762f303be2fcf3.log
[2024/12/02 17:25:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=585776 watch_fd=4 name=/var/log/containers/outputmanager-primary-558d5d778d-52z4c_default_outputmanager-b5085ae3687ef05226fa1e23d301c2212f5a769786f51f37ccb1c3d35327d7ed.log
[2024/12/02 17:25:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=586402 watch_fd=5 name=/var/log/containers/productconfig-db-prometheus-exporter-5bd89dd994-lshxj_default_mongodb-exporter-bf432bfdaa56e9ae91a92a1ae19b41b7f1771f83536286516e0c92ba87a6c23e.log
[2024/12/02 17:25:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=585842 watch_fd=6 name=/var/log/containers/static-content-aurora-static-content-deploytest_default_integration-test-40104681518b35c26f87bdceb089236cba991f403694860177ab81d14c9612f4.log
[2024/12/02 17:25:00] [ info] [input:tail:tail.1] inotify_fs_add(): inode=586553 watch_fd=1 name=/var/log/containers/aurora-controller-ingress-nginx-controller-677859486-5mdqn_ingress_istio-init-7f5970a12e83f8736de2926e51ecc71fe1b51c815a31504eafcf7753c912ec6b.log
[2024/12/02 17:25:00] [ info] [input:tail:tail.1] inotify_fs_add(): inode=586582 watch_fd=2 name=/var/log/containers/aurora-controller-ingress-nginx-controller-677859486-5mdqn_ingress_istio-proxy-4ac987779d51964574c8ac298b86d153b7ae03d985944474511481f8b47070e2.log
[2024/12/02 17:25:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=566543 watch_fd=7 name=/var/log/containers/crs-559445b4f5-m9988_default_crs-server-e584a852c412d682d688c42b670a40caa814541fc24f1a446fdd2fd574361c35.log
[2024/12/02 17:25:00] [ info] [input:tail:tail.0] inotify_fs_add(): inode=566547 watch_fd=8 name=/var/log/containers/aurora-controller-ingress-nginx-controller-677859486-5mdqn_ingress_controller-3c1cc39100ea9594e28bc30b903375cfa60cac2af3e153a226c1ed362dcbe39f.log
[2024/12/02 17:25:00] [ info] [input:tail:tail.1] inotify_fs_add(): inode=566547 watch_fd=3 name=/var/log/containers/aurora-controller-ingress-nginx-controller-677859486-5mdqn_ingress_controller-3c1cc39100ea9594e28bc30b903375cfa60cac2af3e153a226c1ed362dcbe39f.log
[2024/12/02 17:25:04] [ info] [output:azure_blob:azure_blob.1] container 'access-logs' already exists
[2024/12/02 17:25:05] [ info] [output:azure_blob:azure_blob.1] content uploaded successfully: file=kube-ingress-controller.1733160304970
[2024/12/02 17:25:05] [ info] [output:azure_blob:azure_blob.1] blob id ZmxiLTE3MzMxNjAzMDQuOTcwMS5pZA== committed successfully
[2024/12/02 17:25:09] [error] [tls] error: error:0A0C0103:SSL routines::internal error
[2024/12/02 17:25:09] [error] [output:azure_blob:azure_blob.1] error requesting container properties
[2024/12/02 17:25:09] [ warn] [engine] failed to flush chunk '1-1733160305.509332912.flb', retry in 8 seconds: task_id=2, input=tail.1 > output=azure_blob.1 (out_id=1)
[2024/12/02 17:25:14] [engine] caught signal (SIGSEGV)
#0  0x7f109a80d45b      in  ???() at ???:0
#1  0x7f109a8111b7      in  ???() at ???:0
#2  0x7f109a811bb3      in  ???() at ???:0
#3  0x7f109a81d06b      in  ???() at ???:0
#4  0x7f109a81e431      in  ???() at ???:0
#5  0x7f109a81eb47      in  ???() at ???:0
#6  0x7f109a820335      in  ???() at ???:0
#7  0x7f109a82140b      in  ???() at ???:0
#8  0x7f109a9f08ff      in  ???() at ???:0#9  0x7f109aa2e48d      in  ???() at ???:0
#10 0x7f109aa2a6c4      in  ???() at ???:0
#11 0x563bc9ddf50c      in  tls_net_handshake() at src/tls/openssl.c:981
#12 0x563bc9de0214      in  flb_tls_session_create() at src/tls/flb_tls.c:556
#13 0x563bc9deee82      in  flb_io_net_connect() at src/flb_io.c:175
#14 0x563bc9dc6c40      in  create_conn() at src/flb_upstream.c:610
#15 0x563bc9dc6c40      in  flb_upstream_conn_get() at src/flb_upstream.c:799
#16 0x563bc9e7e1a7      in  ensure_container() at plugins/out_azure_blob/azure_blob.c:546
#17 0x563bc9e7e1a7      in  cb_azure_blob_flush() at plugins/out_azure_blob/azure_blob.c:1046
#18 0x563bca2f8c26      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#19 0xffffffffffffffff  in  ???() at ???:0

Expected behavior

Fluent-bit should write the access log entry to the blob container

Screenshots

Your Environment

• Version used: 3.2.0 and 3.2.2
• Configuration:

[SERVICE]
    Flush                 5
    Log_Level             info
    Daemon                off
    Parsers_File          parsers.conf
    Parsers_File          custom_parsers.conf
    # required for readiness and liveness probes
    HTTP_Server           On
    HTTP_Listen           0.0.0.0
    HTTP_PORT             2020

[INPUT]
    Name                  tail
    Tag                   kube.*
    Path                  /var/log/containers/*.log
    Exclude_Path          *kube-*,*azure-*,*blackbox-exporter-*,*blobfuse-flexvol-*,*coredns-*,*istio-*,*node-exporter-*,*keyvault-flexvolume-*,*kiali-*,*fluent-bit*
    Parser                cri_o
    DB                    /var/log/flb_kube.db
    Mem_Buf_Limit         10MB
    Skip_Long_Lines       On
    Refresh_Interval      10
    Inotify_Watcher       ${INOTIFY_WATCHER_ENABLED}

[INPUT]
    Name                  tail
    Tag                   kube-ingress-controller
    Path                  /var/log/containers/*ingress-nginx-controller*.log
    Parser                cri_o
    DB                    /var/log/flb_kube_ingress_controller.db
    Mem_Buf_Limit         10MB
    Skip_Long_Lines       On
    Refresh_Interval      10
    Inotify_Watcher       ${INOTIFY_WATCHER_ENABLED}

# note the order of the filters.
# the lua filters must come after the kubernetes filter, otherwise the log field isn't yet decoded
# when reaching the lua code.
[FILTER]
    Name                  kubernetes
    Match                 kube.*
    Kube_URL              https://kubernetes.default.svc.cluster.local:443
    Merge_Log             On
    Merge_Log_Trim        On
    Keep_Log              Off
    K8S-Logging.Parser    On
    K8S-Logging.Exclude   On
    Buffer_Size           1Mb

[FILTER]
    Name                  lua
    Match                 *
    script                ../scripts/extract_tag.lua
    call                  extract_tag

[FILTER]
    Name                  lua
    Match                 kube*crs-server*
    script                ../scripts/crs.lua
    call                  augment_crs

[FILTER]
    Name                  record_modifier
    Match                 *
    Record                resource_group prod2402130227

[OUTPUT]
    Name                  es
    Match                 kube*
    Host                  ${ELASTICSEARCH_HOST_PREFIX}-es-http
    Port                  9200
    HTTP_User             ${ELASTIC_USERNAME}
    HTTP_Passwd           ${ELASTIC_PASSWORD}
    Buffer_Size           1Mb
    Logstash_Format       On
    Logstash_Prefix       fluent-bit
    Type                  _doc
    Replace_Dots          On
    Time_Key              @ts
    Time_Key_Format       %Y-%m-%dT%H:%M:%S
    tls                   On
    tls.verify            Off
    Suppress_Type_Name    On
    Trace_Error           On

[OUTPUT]
    name                  azure_blob
    match                 kube-ingress-controller
    account_name          ${LOG_STORAGE_ACCOUNT}
    shared_key            ${LOG_STORAGE_ACCOUNT_KEY}
    container_name        access-logs
    blob_type             blockblob
    auto_create_container on
    tls                   on

• Environment name and version (e.g. Kubernetes? What version?):
  Kubernetes-1.30.5
• Server type and version:
  Azure Standard_E4as_v5
• Operating System and version:
  Ubuntu-22.04.5 LTS with kernel 5.15.0-1074-azure
• Filters and plugins:
  (see config)

Additional context

We are running our application on an AKS cluster with kubernetes version 1.30.5.
There are 14 Nodes running currently, having 32GB of memory and 4 CPUs each.
The idea for using the azure_blob plugin is to output a copy of the ingress access logs to a blob storage
On clusters where we don't use the plugin fluent-bit runs just fine.
Likewise, version 3.1.10 runs fine on the same system with the exact same configuration
Fluent-bit is deployed using helm through an umbrella chart with release 0.48.3 from https://fluent.github.io/helm-charts as a dependency.

[tanmaya-panda1]

@GottfriedGanssauge @edsiper @leonardo-albertovich this particularly happens with tls On in the blob output config.

[GottfriedGanssauge]

Maybe???

I'm required to use tls so I can't check without

[patrick-stephens]

TLS off seems to work ok, can we try 3.2.7?

Can we provide a simple reproducer with Azurite or similar in a docker compose stack?

[GottfriedGanssauge]

With 3.2.7 the pod starts fine but on the first output to the storage account it crashes:

[2025/03/10 18:19:22] [ info] [output:azure_blob:azure_blob.1] container 'access-logs' already exists
[2025/03/10 18:19:22] [ info] [output:azure_blob:azure_blob.1] content uploaded successfully: file=kube-ingress-controller.1741630762467
[2025/03/10 18:19:22] [ info] [output:azure_blob:azure_blob.1] blob id ZmxiLTE3NDE2MzA3NjIuNDY3Ni5pZA== committed successfully
[2025/03/10 18:19:27] [error] [tls] error: error:0A0C0103:SSL routines::internal error
[2025/03/10 18:19:27] [error] [output:azure_blob:azure_blob.1] error requesting container properties
[2025/03/10 18:19:27] [ warn] [engine] failed to flush chunk '1-1741630762.424224375.flb', retry in 7 seconds: task_id=2, input=tail.1 > output=azure_blob.1 (out_id=1)
[2025/03/10 18:19:32] [engine] caught signal (SIGSEGV)
#0  0x7f76bf39345b      in  ???() at ???:0
#1  0x7f76bf3971b7      in  ???() at ???:0
#2  0x7f76bf397bb3      in  ???() at ???:0
#3  0x7f76bf3a306b      in  ???() at ???:0
#4  0x7f76bf3a4431      in  ???() at ???:0
#5  0x7f76bf3a4b47      in  ???() at ???:0
#6  0x7f76bf3a6335      in  ???() at ???:0
#7  0x7f76bf3a740b      in  ???() at ???:0
#8  0x7f76bf5768ff      in  ???() at ???:0
#9  0x7f76bf5b448d      in  ???() at ???:0
#10 0x7f76bf5b06c4      in  ???() at ???:0
#11 0x5651c5d75e5c      in  tls_net_handshake() at src/tls/openssl.c:981
#12 0x5651c5d76b64      in  flb_tls_session_create() at src/tls/flb_tls.c:556
#13 0x5651c5d85a52      in  flb_io_net_connect() at src/flb_io.c:175
#14 0x5651c5d5d530      in  create_conn() at src/flb_upstream.c:610
#15 0x5651c5d5d530      in  flb_upstream_conn_get() at src/flb_upstream.c:799
#16 0x5651c5e19b97      in  ensure_container() at plugins/out_azure_blob/azure_blob.c:546
#17 0x5651c5e19b97      in  cb_azure_blob_flush() at plugins/out_azure_blob/azure_blob.c:1046
#18 0x5651c63319c6      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#19 0xffffffffffffffff  in  ???() at ???:0

[GottfriedGanssauge]

I was able to make a docker compose application to demonstrate the problem - but not with azurite :(

With a real storage account the demo crashed reliably with fluent-bit 3.2.x (x in [0,7,8] and it ran stable with fluent-bit 3.1.10.

The azurite version would refuse to create the blob container and thus didn't come into the failure mode.

The demo is contained in https://github.com/GottfriedGanssauge/fluent-bit-crash/tree/main.

[tanmaya-panda1]

@leonardo-albertovich @patrick-stephens @edsiper Do we have an update on this issue?

[tanmaya-panda1]

I am attaching the fluentbit logs with -DFLB_TRACE=On

[2025/03/05 10:42:27] [debug] [input:tail:tail.0] inode=2148333, /var/log/containers/fluent-bit-demo-9rbqb_default_fluent-bit-3a4c5c712dc98d0e645ae63b50afeb902d2a18a01aa4698fcbfa71937c89cc0f.log, events: IN_MODIFY
[2025/03/05 10:42:27] [ warn] [input] tail.0 paused (mem buf overlimit)
[2025/03/05 10:42:27] [ info] [input] pausing tail.0
[2025/03/05 10:42:27] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:28] [trace] [engine] resuming coroutine=0x7f6683414000
[2025/03/05 10:42:28] [trace] [io coro=0x7f6683414000] [net_read] ret=359
[2025/03/05 10:42:28] [debug] [upstream] KA connection #103 to xxxxxx.blob.core.windows.net:443 is now available
[2025/03/05 10:42:28] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE3NDExNzEzNDcuODk4Mi5pZA== committed successfully
[2025/03/05 10:42:28] [debug] [out flush] cb_destroy coro_id=0
[2025/03/05 10:42:28] [trace] [coro] destroy coroutine=0x7f6683414000 data=0x7f6683414018
[2025/03/05 10:42:28] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:28] [trace] [engine] [task event] task_id=0 out_id=0 return=OK
[2025/03/05 10:42:28] [debug] [task] destroy task=0x7f66851a8c00 (task_id=0)
[2025/03/05 10:42:28] [ info] [input] resume tail.0
[2025/03/05 10:42:28] [ info] [input] tail.0 resume (mem buf overlimit)
[2025/03/05 10:42:28] [debug] [input:tail:tail.0] inode=2148333, /var/log/containers/fluent-bit-demo-9rbqb_default_fluent-bit-3a4c5c712dc98d0e645ae63b50afeb902d2a18a01aa4698fcbfa71937c89cc0f.log, events: IN_MODIFY
[2025/03/05 10:42:28] [trace] [input chunk] update output instances with new chunk size diff=12583, records=14, input=tail.0
[2025/03/05 10:42:28] [ warn] [input] tail.0 paused (mem buf overlimit)
[2025/03/05 10:42:28] [ info] [input] pausing tail.0
[2025/03/05 10:42:28] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:28] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:28] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:28] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:28] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:28] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:28] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_addon-token-adapter-033d28350e5d3c334e0c5a364f8a02405828b0fa8777f8c1cdf5e31534201da2.log, inode 2146691
[2025/03/05 10:42:28] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-76f5583ed0c9eeaf86485a8a830e81db6406f20c1b62bd953cf133c248cf1302.log, inode 2146758
[2025/03/05 10:42:28] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-prometheus-f1c55ba141dc1d7deff9dbaa3e1b0959b4a577d0d773ba4e32ed2fd2d94fff3d.log, inode 2146762
[2025/03/05 10:42:28] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-metrics-node-2fwbt_kube-system_addon-token-adapter-fdc41b4ad0b16fa232553a8ee9c1a50ef67819e7e976200456ed3961b1c39a88.log, inode 2146769
[2025/03/05 10:42:28] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/coredns-c6f8cd479-df79c_kube-system_coredns-20ca46fca92111a50d36a4485689adde305d767da1db2b2eda6b840ed8244b18.log, inode 2064667
[2025/03/05 10:42:28] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/csi-azuredisk-node-wwtz2_kube-system_azuredisk-5c8304cca3b71494f0a09af2f423640477fad2dc08b9f3df546467828a993273.log, inode 2146950
[2025/03/05 10:42:28] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/csi-azuredisk-node-wwtz2_kube-system_liveness-probe-6a91ffa22806450e1648cdfc8b463e9ac55763d613d8e68ab95a2e22bceccc24.log, inode 2146940
[2025/03/05 10:42:28] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/metrics-server-6549b48546-pbm6x_kube-system_metrics-server-4b48c4fc694840ea75be4c61ab476252efb68d09dba96400a12e9a3b7375c37f.log, inode 2146955
[2025/03/05 10:42:28] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/metrics-server-6549b48546-pbm6x_kube-system_metrics-server-vpa-4144d4e21f90f5d76154bd6638a157691f5e2b1726f567b5342f79f3dba867d8.log, inode 2146938
[2025/03/05 10:42:28] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:28] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:29] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:29] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:29] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:29] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:29] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:29] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_addon-token-adapter-033d28350e5d3c334e0c5a364f8a02405828b0fa8777f8c1cdf5e31534201da2.log, inode 2146691
[2025/03/05 10:42:29] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-76f5583ed0c9eeaf86485a8a830e81db6406f20c1b62bd953cf133c248cf1302.log, inode 2146758

[2025/03/05 10:42:29] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/kube-proxy-ftzzd_kube-system_kube-proxy-bootstrap-2e32958fc54cf6d2f4920ac1ff110ac85e08cbe3b41fd168be097c25ae4aee3e.log, inode 5434104
[2025/03/05 10:42:29] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/metrics-server-6549b48546-pbm6x_kube-system_metrics-server-4b48c4fc694840ea75be4c61ab476252efb68d09dba96400a12e9a3b7375c37f.log, inode 2146955
[2025/03/05 10:42:29] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/metrics-server-6549b48546-pbm6x_kube-system_metrics-server-vpa-4144d4e21f90f5d76154bd6638a157691f5e2b1726f567b5342f79f3dba867d8.log, inode 2146938
[2025/03/05 10:42:29] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:29] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:30] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:30] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:30] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:30] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:30] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_addon-token-adapter-033d28350e5d3c334e0c5a364f8a02405828b0fa8777f8c1cdf5e31534201da2.log, inode 2146691
[2025/03/05 10:42:30] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-76f5583ed0c9eeaf86485a8a830e81db6406f20c1b62bd953cf133c248cf1302.log, inode 2146758
[2025/03/05 10:42:30] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/csi-azuredisk-node-wwtz2_kube-system_liveness-probe-6a91ffa22806450e1648cdfc8b463e9ac55763d613d8e68ab95a2e22bceccc24.log, inode 2146940
[2025/03/05 10:42:30] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/csi-azuredisk-node-wwtz2_kube-system_node-driver-registrar-7e5ac51876fe3cdee7026ccf0d544cbded479ebb65504501e70ed89b450f2ecf.log, inode 2146944
[2025/03/05 10:42:30] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/csi-azurefile-node-l5sp5_kube-system_azurefile-757b7be3a560968848e74a3a7e3fa62f336721c5abb377bfc5670676fc00e163.log, inode 2073142
[2025/03/05 10:42:30] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/csi-azurefile-node-l5sp5_kube-system_azurefile-e809eedc37e63b90d1b9ba9fce79582be6d9050f4299dc64e557801f37c8ba4b.log, inode 2066598
[2025/03/05 10:42:30] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/csi-azurefile-node-l5sp5_kube-system_liveness-probe-3352e59413e726f7fa5b6aca6f61e8f796a1f314cc447df1d96556198fbed0b0.log, inode 2073107
[2025/03/05 10:42:30] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/metrics-server-6549b48546-pbm6x_kube-system_metrics-server-vpa-4144d4e21f90f5d76154bd6638a157691f5e2b1726f567b5342f79f3dba867d8.log, inode 2146938
[2025/03/05 10:42:30] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:30] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:31] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:31] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:31] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:31] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:31] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:31] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_addon-token-adapter-033d28350e5d3c334e0c5a364f8a02405828b0fa8777f8c1cdf5e31534201da2.log 5435833
[2025/03/05 10:42:31] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/kube-proxy-ftzzd_kube-system_kube-proxy-971f6a0fefbac88dac12d50adeb11119d7ac830e5537ed022e7c2fca05fc5498.log, inode 5434095
[2025/03/05 10:42:31] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/kube-proxy-ftzzd_kube-system_kube-proxy-bootstrap-2e32958fc54cf6d2f4920ac1ff110ac85e08cbe3b41fd168be097c25ae4aee3e.log, inode 5434104
[2025/03/05 10:42:31] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/metrics-server-6549b48546-pbm6x_kube-system_metrics-server-4b48c4fc694840ea75be4c61ab476252efb68d09dba96400a12e9a3b7375c37f.log, inode 2146955
[2025/03/05 10:42:31] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/metrics-server-6549b48546-pbm6x_kube-system_metrics-server-vpa-4144d4e21f90f5d76154bd6638a157691f5e2b1726f567b5342f79f3dba867d8.log, inode 2146938
[2025/03/05 10:42:31] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:31] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:32] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:32] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:32] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:32] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:32] [trace] [task 0x7f66851a8c00] created (id=0)
[2025/03/05 10:42:32] [debug] [task] created task=0x7f66851a8c00 id=0 OK
[2025/03/05 10:42:32] [trace] [task 0x7f66851a8520] created (id=1)
[2025/03/05 10:42:32] [debug] [task] created task=0x7f66851a8520 id=1 OK
[2025/03/05 10:42:32] [trace] [task 0x7f66851a7ee0] created (id=2)
[2025/03/05 10:42:32] [debug] [task] created task=0x7f66851a7ee0 id=2 OK
[2025/03/05 10:42:32] [debug] [output:azure_blob:azure_blob.0] task_id=0 assigned to thread #0
[2025/03/05 10:42:32] [debug] [output:azure_blob:azure_blob.0] task_id=1 assigned to thread #0
[2025/03/05 10:42:32] [debug] [output:azure_blob:azure_blob.0] task_id=2 assigned to thread #0
[2025/03/05 10:42:32] [trace] [upstream] get new connection for xxxxxxx.blob.core.windows.net:443, net setup:
net.connect_timeout        = 10 seconds
net.source_address         = any
net.keepalive              = enabled
net.keepalive_idle_timeout = 30 seconds
net.max_worker_connections = 0
[2025/03/05 10:42:32] [debug] [upstream] KA connection #105 to xxxxxx.blob.core.windows.net:443 has been assigned (recycled)
[2025/03/05 10:42:32] [debug] [http_client] not using http_proxy for header
[2025/03/05 10:42:32] [trace] [io coro=0x7f6683414000] [net_write] trying 381 bytes
[2025/03/05 10:42:32] [error] [tls] error: error:0A0C0103:SSL routines::internal error
[2025/03/05 10:42:32] [trace] [io coro=0x7f6683414000] [net_write] ret=-1 total=0/381
[2025/03/05 10:42:32] [error] [output:azure_blob:azure_blob.0] error requesting container properties
[2025/03/05 10:42:32] [debug] [upstream] KA connection #105 to xxxxxx.blob.core.windows.net:443 is now available
[2025/03/05 10:42:32] [trace] [upstream] get new connection for xxxxxx.blob.core.windows.net:443, net setup:
net.connect_timeout        = 10 seconds
net.source_address         = any
net.keepalive              = enabled
net.keepalive_idle_timeout = 30 seconds
net.max_worker_connections = 0
[2025/03/05 10:42:32] [debug] [upstream] KA connection #110 to xxxxxx.blob.core.windows.net:443 has been assigned (recycled)
[2025/03/05 10:42:32] [debug] [http_client] not using http_proxy for header
[2025/03/05 10:42:32] [trace] [io coro=0x7f66834147e0] [net_write] trying 381 bytes
[2025/03/05 10:42:32] [error] [tls] error: error:0A0C0103:SSL routines::internal error
[2025/03/05 10:42:32] [trace] [io coro=0x7f66834147e0] [net_write] ret=-1 total=0/381
[2025/03/05 10:42:32] [error] [output:azure_blob:azure_blob.0] error requesting container properties
[2025/03/05 10:42:32] [debug] [upstream] KA connection #110 to xxxxxxx.blob.core.windows.net:443 is now available
[2025/03/05 10:42:32] [debug] [out flush] cb_destroy coro_id=9
[2025/03/05 10:42:32] [trace] [coro] destroy coroutine=0x7f6683414000 data=0x7f6683414018
[2025/03/05 10:42:32] [debug] [out flush] cb_destroy coro_id=10
[2025/03/05 10:42:32] [trace] [coro] destroy coroutine=0x7f66834147e0 data=0x7f66834147f8
[2025/03/05 10:42:32] [trace] [upstream] get new connection for xxxxx.blob.core.windows.net:443, net setup:
net.connect_timeout        = 10 seconds
net.source_address         = any
net.keepalive              = enabled
net.keepalive_idle_timeout = 30 seconds
net.max_worker_connections = 0
[2025/03/05 10:42:32] [debug] [upstream] KA connection #102 to xxxxxxxx.blob.core.windows.net:443 has been assigned (recycled)
[2025/03/05 10:42:32] [debug] [http_client] not using http_proxy for header
[2025/03/05 10:42:32] [trace] [io coro=0x7f66834147e0] [net_write] trying 381 bytes
[2025/03/05 10:42:32] [error] [tls] error: error:0A0C0103:SSL routines::internal error
[2025/03/05 10:42:32] [trace] [io coro=0x7f66834147e0] [net_write] ret=-1 total=0/381
[2025/03/05 10:42:32] [error] [output:azure_blob:azure_blob.0] error requesting container properties
[2025/03/05 10:42:32] [debug] [upstream] KA connection #102 to xxxxxx.blob.core.windows.net:443 is now available
[2025/03/05 10:42:32] [debug] [out flush] cb_destroy coro_id=11
[2025/03/05 10:42:32] [trace] [coro] destroy coroutine=0x7f66834147e0 data=0x7f66834147f8
[2025/03/05 10:42:32] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:32] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:32] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_addon-token-adapter-033d28350e5d3c334e0c5a364f8a02405828b0fa8777f8c1cdf5e31534201da2.log, inode 2146691
[2025/03/05 10:42:32] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-76f5583ed0c9eeaf86485a8a830e81db6406f20c1b62bd953cf133c248cf1302.log, inode 2146758
[2025/03/05 10:42:32] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-prometheus-f1c55ba141dc1d7deff9dbaa3e1b0959b4a577d0d773ba4e32ed2fd2d94fff3d.log, inode 2146762
[2025/03/05 10:42:32] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-metrics-node-2fwbt_kube-system_addon-token-adapter-fdc41b4ad0b16fa232553a8ee9c1a50ef67819e7e976200456ed3961b1c39a88.log, inode 2146769
[2025/03/05 10:42:32] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/metrics-server-6549b48546-pbm6x_kube-system_metrics-server-vpa-4144d4e21f90f5d76154bd6638a157691f5e2b1726f567b5342f79f3dba867d8.log, inode 2146938
[2025/03/05 10:42:32] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:32] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:32] [trace] [engine] [task event] task_id=0 out_id=0 return=RETRY
[2025/03/05 10:42:32] [debug] [retry] new retry created for task_id=0 attempts=1
[2025/03/05 10:42:32] [ warn] [engine] failed to flush chunk '1-1741171347.974103724.flb', retry in 6 seconds: task_id=0, input=tail.0 > output=azure_blob.0 (out_id=0)
[2025/03/05 10:42:32] [trace] [engine] [task event] task_id=1 out_id=0 return=RETRY
[2025/03/05 10:42:32] [debug] [retry] new retry created for task_id=1 attempts=1
[2025/03/05 10:42:32] [ warn] [engine] failed to flush chunk '1-1741171347.974259546.flb', retry in 6 seconds: task_id=1, input=tail.0 > output=azure_blob.0 (out_id=0)
[2025/03/05 10:42:32] [trace] [engine] [task event] task_id=2 out_id=0 return=RETRY
[2025/03/05 10:42:32] [debug] [retry] new retry created for task_id=2 attempts=1
[2025/03/05 10:42:32] [ warn] [engine] failed to flush chunk '1-1741171347.977629682.flb', retry in 11 seconds: task_id=2, input=tail.0 > output=azure_blob.0 (out_id=0)
[2025/03/05 10:42:32] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:33] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:33] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:33] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:33] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:33] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_addon-token-adapter-033d28350e5d3c334e0c5a364f8a02405828b0fa8777f8c1cdf5e31534201da2.log, inode 2146691
[2025/03/05 10:42:33] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:33] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:34] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:34] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:34] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:34] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:34] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:34] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_addon-token-adapter-033d28350e5d3c334e0c5a364f8a02405828b0fa8777f8c1cdf5e31534201da2.log, inode 2146691
[2025/03/05 10:42:34] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-76f5583ed0c9eeaf86485a8a830e81db6406f20c1b62bd953cf133c248cf1302.log, inode 2146758
[2025/03/05 10:42:34] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-prometheus-f1c55ba141dc1d7deff9dbaa3e1b0959b4a577d0d773ba4e32ed2fd2d94fff3d.log, inode 2146762
[2025/03/05 10:42:34] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:34] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:35] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:35] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:35] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:35] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:35] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:35] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_addon-token-adapter-033d28350e5d3c334e0c5a364f8a02405828b0fa8777f8c1cdf5e31534201da2.log, inode 2146691
[2025/03/05 10:42:35] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-76f5583ed0c9eeaf86485a8a830e81db6406f20c1b62bd953cf133c248cf1302.log, inode 2146758
[2025/03/05 10:42:35] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-prometheus-f1c55ba141dc1d7deff9dbaa3e1b0959b4a577d0d773ba4e32ed2fd2d94fff3d.log, inode 2146762
[2025/03/05 10:42:35] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-metrics-node-2fwbt_kube-system_addon-token-adapter-fdc41b4ad0b16fa232553a8ee9c1a50ef67819e7e976200456ed3961b1c39a88.log, inode 2146769
[2025/03/05 10:42:35] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-metrics-node-2fwbt_kube-system_prometheus-collector-c91e69b3636b7d940e4cc857b183ebfe73ff4772519149be83c0cf5306a657e3.log, inode 2146765
[2025/03/05 10:42:35] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:35] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:36] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:36] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:36] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:36] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:36] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_addon-token-adapter-033d28350e5d3c334e0c5a364f8a02405828b0fa8777f8c1cdf5e31534201da2.log, inode 2146691
[2025/03/05 10:42:36] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:36] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:37] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:37] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:37] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:37] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:37] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:37] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_addon-token-adapter-033d28350e5d3c334e0c5a364f8a02405828b0fa8777f8c1cdf5e31534201da2.log, inode 2146691
[2025/03/05 10:42:37] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-76f5583ed0c9eeaf86485a8a830e81db6406f20c1b62bd953cf133c248cf1302.log, inode 2146758
[2025/03/05 10:42:37] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:37] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:38] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:38] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:38] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:38] [debug] [output:azure_blob:azure_blob.0] task_id=0 assigned to thread #0
[2025/03/05 10:42:38] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:38] [debug] [output:azure_blob:azure_blob.0] task_id=1 assigned to thread #0
[2025/03/05 10:42:38] [trace] [upstream] get new connection for xxxxx.blob.core.windows.net:443, net setup:
net.connect_timeout        = 10 seconds
net.source_address         = any
net.keepalive              = enabled
net.keepalive_idle_timeout = 30 seconds
net.max_worker_connections = 0
[2025/03/05 10:42:38] [debug] [upstream] KA connection #106 to xxxxxxx.blob.core.windows.net:443 has been assigned (recycled)
[2025/03/05 10:42:38] [debug] [http_client] not using http_proxy for header
[2025/03/05 10:42:38] [trace] [io coro=0x7f66834147e0] [net_write] trying 381 bytes
[2025/03/05 10:42:38] [error] [tls] error: error:0A0C0103:SSL routines::internal error
[2025/03/05 10:42:38] [trace] [io coro=0x7f66834147e0] [net_write] ret=-1 total=0/381
[2025/03/05 10:42:38] [error] [output:azure_blob:azure_blob.0] error requesting container properties
[2025/03/05 10:42:38] [debug] [upstream] KA connection #106 to xxxxxx.blob.core.windows.net:443 is now available
[2025/03/05 10:42:38] [trace] [upstream] get new connection for xxxxxx.blob.core.windows.net:443, net setup:
net.connect_timeout        = 10 seconds
net.source_address         = any
net.keepalive              = enabled
net.keepalive_idle_timeout = 30 seconds
net.max_worker_connections = 0
[2025/03/05 10:42:38] [debug] [upstream] KA connection #104 to xxxxxx.blob.core.windows.net:443 has been assigned (recycled)
[2025/03/05 10:42:38] [debug] [http_client] not using http_proxy for header
[2025/03/05 10:42:38] [trace] [io coro=0x7f66834147c8] [net_write] trying 381 bytes
[2025/03/05 10:42:38] [error] [tls] error: error:0A0C0103:SSL routines::internal error
[2025/03/05 10:42:38] [trace] [io coro=0x7f66834147c8] [net_write] ret=-1 total=0/381
[2025/03/05 10:42:38] [error] [output:azure_blob:azure_blob.0] error requesting container properties
[2025/03/05 10:42:38] [debug] [upstream] KA connection #104 to xxxxx.blob.core.windows.net:443 is now available
[2025/03/05 10:42:38] [debug] [out flush] cb_destroy coro_id=12
[2025/03/05 10:42:38] [trace] [coro] destroy coroutine=0x7f66834147e0 data=0x7f66834147f8
[2025/03/05 10:42:38] [debug] [out flush] cb_destroy coro_id=13
[2025/03/05 10:42:38] [trace] [coro] destroy coroutine=0x7f66834147c8 data=0x7f66834147e0
[2025/03/05 10:42:38] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:38] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:38] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_addon-token-adapter-033d28350e5d3c334e0c5a364f8a02405828b0fa8777f8c1cdf5e31534201da2.log, inode 2146691
[2025/03/05 10:42:38] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-76f5583ed0c9eeaf86485a8a830e81db6406f20c1b62bd953cf133c248cf1302.log, inode 2146758
[2025/03/05 10:42:38] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:38] [trace] [engine] [task event] task_id=0 out_id=0 return=RETRY
[2025/03/05 10:42:38] [debug] [task] task_id=0 reached retry-attempts limit 1/1
[2025/03/05 10:42:38] [error] [engine] chunk '1-1741171347.974103724.flb' cannot be retried: task_id=0, input=tail.0 > output=azure_blob.0
[2025/03/05 10:42:38] [debug] [task] destroy task=0x7f66851a8c00 (task_id=0)
[2025/03/05 10:42:38] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:38] [trace] [engine] [task event] task_id=1 out_id=0 return=RETRY
[2025/03/05 10:42:38] [debug] [task] task_id=1 reached retry-attempts limit 1/1
[2025/03/05 10:42:38] [error] [engine] chunk '1-1741171347.974259546.flb' cannot be retried: task_id=1, input=tail.0 > output=azure_blob.0
[2025/03/05 10:42:38] [debug] [task] destroy task=0x7f66851a8520 (task_id=1)
[2025/03/05 10:42:38] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:39] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:39] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:39] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:39] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:39] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_addon-token-adapter-033d28350e5d3c334e0c5a364f8a02405828b0fa8777f8c1cdf5e31534201da2.log, inode 2146691
[2025/03/05 10:42:39] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-76f5583ed0c9eeaf86485a8a830e81db6406f20c1b62bd953cf133c248cf1302.log, inode 2146758
[2025/03/05 10:42:39] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-logs-5vst7_kube-system_ama-logs-prometheus-f1c55ba141dc1d7deff9dbaa3e1b0959b4a577d0d773ba4e32ed2fd2d94fff3d.log, inode 2146762
[2025/03/05 10:42:39] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-metrics-node-2fwbt_kube-system_addon-token-adapter-fdc41b4ad0b16fa232553a8ee9c1a50ef67819e7e976200456ed3961b1c39a88.log, inode 2146769
[2025/03/05 10:42:39] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/ama-metrics-node-2fwbt_kube-system_prometheus-collector-c91e69b3636b7d940e4cc857b183ebfe73ff4772519149be83c0cf5306a657e3.log, inode 2146765
[2025/03/05 10:42:39] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/azure-ip-masq-agent-6mrq8_kube-system_azure-ip-masq-agent-4b7bdb8f854eb8c8a445fc3bed368e8d9b2e0b37f8c974d3f7a68c067cb7d200.log, inode 2064684
[2025/03/05 10:42:39] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:39] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:40] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:40] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:40] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:40] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:40] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2025/03/05 10:42:40] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2025/03/05 10:42:40] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:41] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:41] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:41] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:42:41] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:43:04] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:43:04] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:43:04] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:43:04] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:43:04] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:43:04] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:43:04] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:43:04] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:43:04] [trace] [net] connection #102 in process to xxxxx.blob.core.windows.net:443
[2025/03/05 10:43:04] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:43:04] [trace] [engine] resuming coroutine=0x7f66834147f8
[2025/03/05 10:43:04] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:43:04] [trace] [engine] resuming coroutine=0x7f66834147f8
[2025/03/05 10:43:04] [trace] [sched] 0 timer coroutines destroyed
[2025/03/05 10:43:04] [trace] [engine] resuming coroutine=0x7f66834147f8
[2025/03/05 10:43:04] [engine] caught signal (SIGSEGV)
[2025/03/05 10:43:05] [trace] [sched] 0 timer coroutines destroyed
#0  0x7f66883a345b      in  ???() at ???:0
#1  0x7f66883a71b7      in  ???() at ???:0
#2  0x7f66883a7bb3      in  ???() at ???:0
#3  0x7f66883b306b      in  ???() at ???:0
#4  0x7f66883b4431      in  ???() at ???:0
#5  0x7f66883b4b47      in  ???() at ???:0
#6  0x7f66883b6335      in  ???() at ???:0
#7  0x7f66883b740b      in  ???() at ???:0
#8  0x7f66885868ff      in  ???() at ???:0
#9  0x7f66885c448d      in  ???() at ???:0
#10 0x7f66885c06c4      in  ???() at ???:0
#11 0x5593208ad53c      in  tls_net_handshake() at src/tls/openssl.c:981
#12 0x5593208ae2d4      in  flb_tls_session_create() at src/tls/flb_tls.c:556
#13 0x5593208bdc62      in  flb_io_net_connect() at src/flb_io.c:175
#14 0x559320894a00      in  create_conn() at src/flb_upstream.c:610
#15 0x559320894a00      in  flb_upstream_conn_get() at src/flb_upstream.c:799
#16 0x559320952cd7      in  ensure_container() at plugins/out_azure_blob/azure_blob.c:546
#17 0x559320952cd7      in  cb_azure_blob_flush() at plugins/out_azure_blob/azure_blob.c:1046
#18 0x559320e77e06      in  co_init() at lib/monkey/deps/flb_libco/amd64.c:117
#19 0xffffffffffffffff  in  ???() at ???:0

[GottfriedGanssauge]

I checked 4.0.0 and the issue persists

[kustonaut]

@patrick-stephens @leonardo-albertovich request your kind help.

[leonardo-albertovich]

I'm currently working on this issue, I'll send an update as soon as I'm able to reproduce it. If anyone wants to get in touch to assist during the process feel free to talk to me in the fluent slack server.

[leonardo-albertovich]

I was able to reproduce the crash locally using azurite in combination with socat, however, the crash I see does not resemble the one you've shared @GottfriedGanssauge. I'm still waiting to get access to azure to try to reproduce the crash and validate the PR but I'd appreciate it if you could give PR #10235 a go.

[brablc]

Tested latest master (with PR #10235 merged) and it works for me. The logging seems little off with message like blob id (null) committed successfully, but it does not crash and it uploads logs.

Fluent Bit v4.0.1
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___  _____
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/


[2025/04/22 14:53:22] [ info] [fluent bit] version=4.0.1, commit=aac988327b, pid=159787
[2025/04/22 14:53:22] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/04/22 14:53:22] [ info] [simd    ] disabled
[2025/04/22 14:53:22] [ info] [cmetrics] version=0.9.9
[2025/04/22 14:53:22] [ info] [ctraces ] version=0.6.2
[2025/04/22 14:53:22] [ info] [input:systemd:systemd.0] initializing
[2025/04/22 14:53:22] [ info] [input:systemd:systemd.0] storage_strategy='memory' (memory only)
[2025/04/22 14:53:22] [ info] [input:systemd:systemd.0] seek_cursor=s=9c2d5f62fc22425bbd962cde03e2b6de;i=315... OK
[2025/04/22 14:53:22] [ info] [input:node_exporter_metrics:node_exporter_metrics.1] initializing
[2025/04/22 14:53:22] [ info] [input:node_exporter_metrics:node_exporter_metrics.1] storage_strategy='memory' (memory only)
[2025/04/22 14:53:22] [ info] [input:node_exporter_metrics:node_exporter_metrics.1] path.procfs = /proc
[2025/04/22 14:53:22] [ info] [input:node_exporter_metrics:node_exporter_metrics.1] path.sysfs  = /sys
[2025/04/22 14:53:22] [ info] [input:node_exporter_metrics:node_exporter_metrics.1] thread instance initialized
[2025/04/22 14:53:23] [ info] [input:emitter:emitter_for_rewrite_tag.1] initializing
[2025/04/22 14:53:23] [ info] [input:emitter:emitter_for_rewrite_tag.1] storage_strategy='memory' (memory only)
[2025/04/22 14:53:23] [ info] [output:azure_blob:azure_blob.0] account_name=xxxdebug, container_name=logs, blob_type=appendblob, emulator_mode=no, endpoint=xxxdebug.blob.core.windows.net, auth_type=sas
[2025/04/22 14:53:23] [ info] [output:azure_blob:azure_blob.0] initializing worker
[2025/04/22 14:53:23] [ info] [output:azure_blob:azure_blob.0] worker #0 started
[2025/04/22 14:53:23] [ info] [output:prometheus_exporter:prometheus_exporter.1] listening iface=0.0.0.0 tcp_port=2021
[2025/04/22 14:53:23] [ info] [sp] stream processor started
[2025/04/22 14:54:22] [ info] [output:azure_blob:azure_blob.0] container 'logs' already exists
[2025/04/22 14:54:22] [ info] [output:azure_blob:azure_blob.0] content uploaded successfully:
[2025/04/22 14:54:22] [ info] [output:azure_blob:azure_blob.0] blob id (null) committed successfully
[2025/04/22 14:55:22] [ info] [output:azure_blob:azure_blob.0] container 'logs' already exists
[2025/04/22 14:55:22] [ info] [output:azure_blob:azure_blob.0] content uploaded successfully:
[2025/04/22 14:55:22] [ info] [output:azure_blob:azure_blob.0] blob id (null) committed successfully
[2025/04/22 14:55:52] [ info] [output:azure_blob:azure_blob.0] container 'logs' already exists
[2025/04/22 14:55:52] [ info] [output:azure_blob:azure_blob.0] content uploaded successfully:
[2025/04/22 14:55:52] [ info] [output:azure_blob:azure_blob.0] blob id (null) committed successfully