bump 2.31.1

[matthewfala]

2.31.1

This release includes:

• Fluent Bit 1.9.10
• Amazon CloudWatch Logs for Fluent Bit 1.9.1
• Amazon Kinesis Streams for Fluent Bit 1.10.1
• Amazon Kinesis Firehose for Fluent Bit 1.7.1

Compared to 2.31.0, this release adds the following feature that we are working on getting accepted upstream:

• Bug - Resolve Datadog buffer resize segfault by including critical parts of reverted Datadog PR aws-for-fluent-bit:491
• Feature - Add a configuration option for S3 retry attempts fluent-bit:6475

Same as 2.31.0, this release includes the following fixes and features that we are working on getting accepted upstream:

• Feature - Add kinesis_firehose and kinesis_streams support for time_key_format milliseconds with %3N option, and nanoseconds 9N and %L options fluent-bit:2831
• Feature - Support OpenSearch Serverless data ingestion via OpenSearch plugin fluent-bit:6448
• Enhancement - Transition S3 to fully synchronous file uploads to improve plugin stability fluent-bit:6573
• Bug - Mitigate Datadog output plugin issue by reverting recent PR aws-for-fluent-bit:491
• Bug - Format S3 filename with timestamp from the first log in uploaded file, rather than the time the first log was buffered by the s3 output aws-for-fluent-bit:459
• Bug - Resolve S3 logic to display log_key missing warning message if the configured log_key field is missing from log payload fluent-bit:6557
• Bug - ECS Metadata filter gracefuly handle task metadata query errors and cache metadata processing state to improve performance aws-for-fluent-bit:505

Add the following cherrypicks

# Add back Datadog fixes that do not trigger segfault
https://github.com/matthewfala/fluent-bit.git ecs-datadog-sequential-revert acc01a4bf4ce656023d82943bec2683b5b7755e5
https://github.com/matthewfala/fluent-bit.git ecs-datadog-sequential-revert ac30b7c876a95d44c33a406b2b58ce08b978587e

# Datadog Partial Fix Patch
https://github.com/matthewfala/fluent-bit.git datadog-ecs-patch 3c1ad69ada5bb6f2e448c6f39a1a0ea6a6f4ff17

# S3 retry limit configuration option
https://github.com/Claych/fluent-bit.git clay-retry-limit-1.9 26922abb9b970c322d658117df68228dc26df5f9
https://github.com/Claych/fluent-bit.git clay-retry-limit-1.9 a5410e4d3bda21f5072f55c8e6a128654a9b9894
https://github.com/Claych/fluent-bit.git clay-retry-limit-1.9 f0fe0245228330a9200325dcc2d594d4e3d49e89
https://github.com/Claych/fluent-bit.git clay-retry-limit-1.9 96587eebdb91d1f4d37c705e54e1ae91327f3353

Signed-off-by: Matthew Fala falamatt@amazon.com

Issue #, if available:

Description of changes:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[PettitWesley]

does this need to be a bunch of commits? I think the PR could be squashed to 1.

[matthewfala]

Squashed. Thanks Clay

[PettitWesley]

I think this is not the most user friendly message. Can we rephrase it to not be how we accomplished this, but what it means for customers?

[matthewfala]

Updated

[PettitWesley]

remember to run the new Makefile target for validating this file which is in our runbook now.

@matthewfala @rawahars should we make that target run in the release target? that way we fail fast.

make validate-version-file-format

[matthewfala]

Ran the script. It works.

[PettitWesley]

what do you think about just making it part of the release target so we already run it a fail fast?

[matthewfala]

That's a good idea. I can submit a pr for it.

[matthewfala]

.PHONY: validate-version-file-format
validate-version-file-format:
	jq -e . windows.versions && true || false
	jq -e . linux.version && true || false

Actually, it's not clear what the point of this script is,

Is it supposed to fail if the following is not in JSON format?

{
  "linux": {
    "version": "2.31.1",
    "build": "1",
    "fluent-bit": "1.9.10",
    "kinesis-plugin": "v1.10.1",
    "firehose-plugin": "v1.7.1",
    "cloudwatch-plugin": "v1.9.1"
  }
}

the make validate-version-file-format does not, if that is the case.

Meanwhile it appears that the build linux plugins does fail if the version document is not JSON regardless of whether the validate-version-file-format command is run or not

./scripts/build_plugins.sh \
        --KINESIS_PLUGIN_CLONE_URL= \
        --KINESIS_PLUGIN_TAG= \
        --KINESIS_PLUGIN_BRANCH= \
        --FIREHOSE_PLUGIN_CLONE_URL= \
        --FIREHOSE_PLUGIN_TAG= \
        --FIREHOSE_PLUGIN_BRANCH= \
        --CLOUDWATCH_PLUGIN_CLONE_URL= \
        --CLOUDWATCH_PLUGIN_TAG= \
        --CLOUDWATCH_PLUGIN_BRANCH=
latest version not found for linux
make: *** [linux-plugins] Error 1

Our runbook specifies:
The above changes go in the linux.version and windows.versions file. Run make validate-version-file-format to check that the update was successful. If formatted incorrectly, it will lead to a release pipeline failure.

If the JSON file is not formatted correctly, it won't be printed out, so that may be a way to manually check for issues.

[rawahars]

@PettitWesley I think it would be. great if we add it to the release target.

@matthewfala Yes that is correct. The make target would fail if the json files for Windows and Linux are not formatted properly. It would be a fast fail to ensure that our Pipelines do not fail due to invalid JSON.

I am not clear what you mean when you say the make validate-version-file-format does not, if that is the case.

For example-

{
  "linux": {
    "version": "2.31.1",
    "build": "1",
    "fluent-bit": "1.9.10",
    "kinesis-plugin": "v1.10.1",
    "firehose-plugin": "v1.7.1",
    "cloudwatch-plugin": "v1.9.1"
  }
}

This would succeed since the format of json is correct.

{
  "linux": {
    "version": "2.31.1",
    "build": "1",
    "fluent-bit": "1.9.10",
    "kinesis-plugin": "v1.10.1",
    "firehose-plugin": "v1.7.1",
    "cloudwatch-plugin": "v1.9.1",
  }
}

This would fail with the following error-

jq -e . linux.version && true || false
parse error: Expected another key-value pair at line 9, column 3
make: *** [validate-version-file-format] Error 1

This is because the above JSON is not formatted properly and has an extra comma at "cloudwatch-plugin": "v1.9.1",. Such errors can be easily identified before we commit the changes leading to pipeline failures.