Consistent CPU Spikes every 24 hours

[PettitWesley]

Bug Report

Describe the bug
Consistent CPU spikes every 24 hours that last for 1 - 2 hours. No spike in memory at the same time.

To Reproduce

I ran this on ECS EC2 with a pre-release build which is similar to the 1.9.10 tag which was just cut; config that just collects json-file log driver container log files:

[SERVICE]
        Flush                     5
        Log_Level                 info
        Daemon                    off
        Parsers_File              parsers.conf
        HTTP_Server               On
        HTTP_Listen               0.0.0.0
        HTTP_Port                 2020
        storage.path              /var/fluent-bit/state/flb-storage/
        storage.sync              normal
        storage.checksum          off
        storage.backlog.mem_limit 5M


[INPUT]
        Name                tail
        Tag                 ecs.*
        Path                /var/lib/docker/containers/*/*.log
        Docker_Mode         On
        Docker_Mode_Flush   5
        Docker_Mode_Parser  container_firstline
        Parser              docker
        DB                  /var/fluent-bit/state/flb_container.db
        Mem_Buf_Limit       1G
        buffer_max_size     1G
        Skip_Long_Lines     On
        Refresh_Interval    10
        Rotate_Wait         30
        storage.type        filesystem
        Read_From_Head      Off


[FILTER]
        Name ecs
        Match *
        ECS_Tag_Prefix ecs.var.lib.docker.containers.
        ADD container $ContainerName
        ADD ecs_task_id $TaskID
        ADD cluster $ClusterName
        ADD log_stream_name $ClusterName.$TaskDefinitionFamily.$TaskID.$ECSContainerName


[OUTPUT]
        Name cloudwatch_logs
        Match *
        log_group_name daemon-load-test-application
        log_stream_prefix fallback-
        log_stream_template $log_stream_name
        region us-west-1
        auto_create_group On
        workers 1

Expected behavior

I'm mainly just curious if anyone has ideas on what could be causing this and any potential mitigations. Do other users see similar once a day spikes? Is there anything in FB that could cause a once a day spike?

Screenshots

Your Environment

• Version used: 1.9.9 with the ECS filter cherry picked on to it (So close to what we have in the 1.9.10 tag)
• Configuration: see above
• Environment name and version (e.g. Kubernetes? What version?): AWS ECS EC2
• Server type and version:
• Operating System and version: Amazon Linux EC2
• Filters and plugins:

Additional context

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.