out_cloudwatch_logs: Added StorageResolution support for EMF

[devin-andrenam]

The following PR adds support for StorageResolution for CloudWatch metrics. This tunable allows users to indicate to CloudWatch the expected granularity to store the metrics. See the following CloudWatch documentation for details.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

[SERVICE]
    flush        1
    daemon       Off
    log_level    info
    parsers_file parsers.conf
    http_server  Off
    http_listen  0.0.0.0
    http_port    2020

[INPUT]
    name mem
    tag  metrics.mem

[FILTER]
    name record_modifier
    match metrics.*
    record device_id bug-device-test

[OUTPUT]
    name cloudwatch_logs
    match metrics.*
    region us-east-1
    log_format json/emf
    metric_namespace bug-testing
    metric_dimensions device_id
    metric_storage_resolution 1
    log_group_name bug-testing
    log_stream_name memory
    auto_create_group true

• Debug log output from testing the change

Fluent Bit v4.0.2
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___  _____ 
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/ 

[2025/05/28 20:05:20] [ info] Configuration:
[2025/05/28 20:05:20] [ info]  flush time     | 1.000000 seconds
[2025/05/28 20:05:20] [ info]  grace          | 5 seconds
[2025/05/28 20:05:20] [ info]  daemon         | 0
[2025/05/28 20:05:20] [ info] ___________
[2025/05/28 20:05:20] [ info]  inputs:
[2025/05/28 20:05:20] [ info]      mem
[2025/05/28 20:05:20] [ info] ___________
[2025/05/28 20:05:20] [ info]  filters:
[2025/05/28 20:05:20] [ info]      record_modifier.0
[2025/05/28 20:05:20] [ info] ___________
[2025/05/28 20:05:20] [ info]  outputs:
[2025/05/28 20:05:20] [ info]      cloudwatch_logs.0
[2025/05/28 20:05:20] [ info] ___________
[2025/05/28 20:05:20] [ info]  collectors:
[2025/05/28 20:05:20] [ info] [fluent bit] version=4.0.2, commit=f28e61edd8, pid=106499
[2025/05/28 20:05:20] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2025/05/28 20:05:20] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/05/28 20:05:20] [ info] [simd    ] disabled
[2025/05/28 20:05:20] [ info] [cmetrics] version=1.0.2
[2025/05/28 20:05:20] [ info] [ctraces ] version=0.6.6
[2025/05/28 20:05:20] [ info] [input:mem:mem.0] initializing
[2025/05/28 20:05:20] [ info] [input:mem:mem.0] storage_strategy='memory' (memory only)
[2025/05/28 20:05:20] [debug] [mem:mem.0] created event channels: read=25 write=26
[2025/05/28 20:05:20] [debug] [cloudwatch_logs:cloudwatch_logs.0] created event channels: read=27 write=28
[2025/05/28 20:05:20] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Metric Namespace=bug-testing
[2025/05/28 20:05:20] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Metric Dimensions=device_id
[2025/05/28 20:05:20] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Metric Storage Resolution=1
[2025/05/28 20:05:20] [debug] [aws_credentials] Initialized Env Provider in standard chain
[2025/05/28 20:05:20] [debug] [aws_credentials] creating profile (null) provider
[2025/05/28 20:05:20] [debug] [aws_credentials] Initialized AWS Profile Provider in standard chain
[2025/05/28 20:05:20] [debug] [aws_credentials] Not initializing EKS provider because AWS_ROLE_ARN was not set
[2025/05/28 20:05:20] [debug] [aws_credentials] Not initializing ECS/EKS HTTP Provider because AWS_CONTAINER_CREDENTIALS_RELATIVE_URI and AWS_CONTAINER_CREDENTIALS_FULL_URI is not set
[2025/05/28 20:05:20] [debug] [aws_credentials] Initialized EC2 Provider in standard chain
[2025/05/28 20:05:20] [debug] [aws_credentials] Sync called on the EC2 provider
[2025/05/28 20:05:20] [debug] [aws_credentials] Init called on the env provider
[2025/05/28 20:05:20] [debug] [aws_credentials] upstream_set called on the EC2 provider
[2025/05/28 20:05:20] [ info] [sp] stream processor started
[2025/05/28 20:05:20] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] worker #0 started
[2025/05/28 20:05:21] [debug] [task] created task=0x7bda0c085d60 id=0 OK
[2025/05/28 20:05:21] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] task_id=0 assigned to thread #0
[2025/05/28 20:05:21] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=memory, group=bug-testing
[2025/05/28 20:05:21] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream memory in log group bug-testing
[2025/05/28 20:05:21] [debug] [upstream] KA connection #48 to logs.us-east-1.amazonaws.com:443 is connected
[2025/05/28 20:05:21] [debug] [http_client] not using http_proxy for header
[2025/05/28 20:05:21] [debug] [aws_credentials] Requesting credentials from the env provider..
[2025/05/28 20:05:22] [debug] [aws_client] logs.us-east-1.amazonaws.com: http_do=0, HTTP Status: 400
[2025/05/28 20:05:22] [debug] [upstream] KA connection #48 to logs.us-east-1.amazonaws.com:443 is now available
[2025/05/28 20:05:22] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] CreateLogStream http status=400
[2025/05/28 20:05:22] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Stream memory already exists
[2025/05/28 20:05:22] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] cloudwatch:PutLogEvents: events=1, payload=823 bytes
[2025/05/28 20:05:22] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Sending log events to log stream memory
[2025/05/28 20:05:22] [debug] [upstream] KA connection #48 to logs.us-east-1.amazonaws.com:443 has been assigned (recycled)
[2025/05/28 20:05:22] [debug] [http_client] not using http_proxy for header
[2025/05/28 20:05:22] [debug] [aws_credentials] Requesting credentials from the env provider..
[2025/05/28 20:05:22] [debug] [upstream] KA connection #48 to logs.us-east-1.amazonaws.com:443 is now available
[2025/05/28 20:05:22] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http status=200
[2025/05/28 20:05:22] [debug] [out flush] cb_destroy coro_id=0
[2025/05/28 20:05:22] [debug] [task] destroy task=0x7bda0c085d60 (task_id=0)
[2025/05/28 20:05:22] [debug] [task] created task=0x7bda0c086050 id=0 OK
[2025/05/28 20:05:22] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] task_id=0 assigned to thread #0
[2025/05/28 20:05:22] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=memory, group=bug-testing
[2025/05/28 20:05:22] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] cloudwatch:PutLogEvents: events=1, payload=823 bytes
[2025/05/28 20:05:22] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Sending log events to log stream memory
[2025/05/28 20:05:22] [debug] [upstream] KA connection #48 to logs.us-east-1.amazonaws.com:443 has been assigned (recycled)
[2025/05/28 20:05:22] [debug] [http_client] not using http_proxy for header
[2025/05/28 20:05:22] [debug] [aws_credentials] Requesting credentials from the env provider..
[2025/05/28 20:05:22] [debug] [upstream] KA connection #48 to logs.us-east-1.amazonaws.com:443 is now available
[2025/05/28 20:05:22] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http status=200
[2025/05/28 20:05:22] [debug] [out flush] cb_destroy coro_id=1
[2025/05/28 20:05:22] [debug] [task] destroy task=0x7bda0c086050 (task_id=0)
[2025/05/28 20:05:23] [debug] [task] created task=0x7bda0c083e60 id=0 OK
[2025/05/28 20:05:23] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] task_id=0 assigned to thread #0
[2025/05/28 20:05:23] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=memory, group=bug-testing
[2025/05/28 20:05:23] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] cloudwatch:PutLogEvents: events=1, payload=823 bytes
[2025/05/28 20:05:23] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Sending log events to log stream memory
[2025/05/28 20:05:23] [debug] [upstream] KA connection #48 to logs.us-east-1.amazonaws.com:443 has been assigned (recycled)
[2025/05/28 20:05:23] [debug] [http_client] not using http_proxy for header
[2025/05/28 20:05:23] [debug] [aws_credentials] Requesting credentials from the env provider..
[2025/05/28 20:05:23] [debug] [upstream] KA connection #48 to logs.us-east-1.amazonaws.com:443 is now available
[2025/05/28 20:05:23] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http status=200
[2025/05/28 20:05:23] [debug] [out flush] cb_destroy coro_id=2
[2025/05/28 20:05:23] [debug] [task] destroy task=0x7bda0c083e60 (task_id=0)
^C[2025/05/28 20:05:24] [engine] caught signal (SIGINT)
[2025/05/28 20:05:24] [debug] [task] created task=0x7bda0c088140 id=0 OK
[2025/05/28 20:05:24] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] task_id=0 assigned to thread #0
[2025/05/28 20:05:24] [ warn] [engine] service will shutdown in max 5 seconds
[2025/05/28 20:05:24] [debug] [engine] retry=0x7bda115f5d70 for task 0 already scheduled to run, not re-scheduling it.
[2025/05/28 20:05:24] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Using stream=memory, group=bug-testing
[2025/05/28 20:05:24] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] cloudwatch:PutLogEvents: events=1, payload=823 bytes
[2025/05/28 20:05:24] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] Sending log events to log stream memory
[2025/05/28 20:05:24] [debug] [upstream] KA connection #48 to logs.us-east-1.amazonaws.com:443 has been assigned (recycled)
[2025/05/28 20:05:24] [debug] [http_client] not using http_proxy for header
[2025/05/28 20:05:24] [debug] [aws_credentials] Requesting credentials from the env provider..
[2025/05/28 20:05:24] [debug] [upstream] KA connection #48 to logs.us-east-1.amazonaws.com:443 is now available
[2025/05/28 20:05:24] [debug] [output:cloudwatch_logs:cloudwatch_logs.0] PutLogEvents http status=200
[2025/05/28 20:05:24] [debug] [out flush] cb_destroy coro_id=3
[2025/05/28 20:05:24] [debug] [task] destroy task=0x7bda0c088140 (task_id=0)
[2025/05/28 20:05:24] [ info] [engine] service has stopped (0 pending tasks)
[2025/05/28 20:05:24] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopping...
[2025/05/28 20:05:24] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] thread worker #0 stopped

The following images illustrate the correctness of the raw EMF logs as well as its sub 60s resolution at the metric graphing level.

• [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature
  Documentation PR

Backporting

• [N/A] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[cosmo0920]

This StorageResolution parameter is already supported for the metrics type of pipelines. So, it'll be only effective for logs type of pipelines which contain cpu or mem plugin.

[cosmo0920]

These lines could be violated for the coding guide line:
https://github.com/fluent/fluent-bit/blob/master/CONTRIBUTING.md#coding-style

[devin-andrenam]

Updated formatting.

And correct. As cpu and mem are not considered metrics, this will only apply to them. May be outside the scope of this PR but does bring up the issue of whether cpu and mem should be updated to be metrics or kept as logs.

[edsiper]

ping @sparrc @singholt @swapneils