Update fluent-bit to v4.0.7. by franciscovalentecastro · Pull Request #2071 · GoogleCloudPlatform/ops-agent

franciscovalentecastro · 2025-09-19T18:56:37Z

Description

Upgrade fluent-bit submodule from v3.1.6 to v4.0.7 to bring the following bug fixes :

in_windows_exporter_metrics: Handle DST more precisely with dynamic timezone verison of functions [Backport to 4.0] fluent/fluent-bit#10701 (Customer Issue)
in_winevtlog: Handle Daylight saving time by using newer functions fluent/fluent-bit#10628 (Customer Issue)
out_stackdriver bug fix: return cached token when current_timestamp is less than cached_expiration fluent/fluent-bit#9652
out_stackdriver: fixed a memory leak (CID 508244) fluent/fluent-bit#9299
out_stackdriver: fixed a wrong data type used for two boolean options fluent/fluent-bit#9222
utf8: enhancements for handling of multibyte sequences fluent/fluent-bit#9687 (seems to fix utf8 log issues from v3.2.2)

Some details about the upgrade :

Revamped Dockerfile.windows to follow : v4.0.7/dockerfiles/Dockerfile.windows.
Setting shutdown Grace period to 0s to avoid any unreliable logs to be sent after Exit_On_Eof.
- Fluent-bit has a long standing bug multiline parser doens't play well with the last record fluent/fluent-bit#8623 .
Created a submodules/go.mod file to encapsulate any Go code from submodules from the top level module.
SLES-12 requires a newer GCC compiler (chose gcc8) to build fluent-bit 4.0.7.
CMake 3.20 or higher is required to build fluent-bit 4.0.7.
DFLB_KAFKA=OFF is a newly added flag that disables all kafka libraries which fail to build and are not needed.

Related issue

b/443678958
b/446168784
b/446655640

How has this been tested?

Checklist:

Unit tests
- Unit tests do not apply.
- Unit tests have been added/modified and passed for this PR.
Integration tests
- Integration tests do not apply.
- Integration tests have been added/modified and passed for this PR.
Documentation
- This PR introduces no user visible changes.
- This PR introduces user visible changes and the corresponding documentation change has been made.
Minor version bump
- This PR introduces no new features.
- This PR introduces new features, and there is a separate PR to bump the minor version since the last release already.
- This PR bumps the version.

franciscovalentecastro · 2025-09-24T20:34:22Z

Current Status - Fluent-bit 4.0.6 has a couple of small updates that break some of our tests :

Ubuntu / Debian packages provide cmake 3.18.x versions, but fluent-bit now requires cmake 3.20.x to build some dependencies (e.g. onigmo for parse_regex).
The following update lib: nghttp2: upgrade to v1.65.0 : fluent/fluent-bit@a237287 adds some Golang code to one of the libraries subfolders that messes up our go test ./... since there is no go.mod to separate the submodule as another codebase.

kokoro/scripts/test/go_test.sh

…om testing.

jefferbrecht · 2025-10-01T13:28:09Z

Dockerfile.windows

    -DFLB_METRICS=On `
    -DFLB_WASM=Off `
    -DFLB_WAMRC=Off `
+    -DFLB_KAFKA=OFF `


Does this mean we don't need -DFLB_(IN|OUT)_KAFKA=Off anymore?

Also, nit: OFF -> Off

Does this mean we don't need -DFLB_(IN|OUT)_KAFKA=Off anymore?

Yes, from the implementation ¹ we could remove IN and OUT from this list. I didn't remove them to be overly descriptive. I'll remove them to see how it looks.

# Kafka support (library in general) if(FLB_KAFKA) if(FLB_IN_KAFKA OR FLB_OUT_KAFKA) ... ...

Footnotes

https://github.com/fluent/fluent-bit/blob/v4.0.7/CMakeLists.txt#L757-L771 ↩

jefferbrecht · 2025-10-01T13:29:57Z

dockerfiles/compile.go

-		build-essential cmake bison flex file libsystemd-dev \
-		devscripts cdbs pkg-config openjdk-${OPENJDK_MAJOR_VERSION}-jdk zip`,
+		build-essential bison flex file libsystemd-dev \
+		devscripts cdbs pkg-config openjdk-${OPENJDK_MAJOR_VERSION}-jdk zip` + installCMake,


Shouldn't we only need installCMake for distros that ship < 3.20.0? Bookworm ships 3.25.1

Yes, we should only use installCmake when we need to. I conflated all Ubuntu/Debian failures to this reason initially. Removed installCmake. Done!

jefferbrecht · 2025-10-01T13:31:08Z

dockerfiles/compile.go

-		build-essential cmake bison flex file libsystemd-dev tzdata \
-		devscripts cdbs pkg-config openjdk-${OPENJDK_MAJOR_VERSION}-jdk zip`,
+		build-essential bison flex file libsystemd-dev tzdata \
+		devscripts cdbs pkg-config openjdk-${OPENJDK_MAJOR_VERSION}-jdk zip` + installCMake,


Likewise, Jammy ships CMake 3.22.1

jefferbrecht · 2025-10-01T13:31:37Z

dockerfiles/compile.go

-		build-essential cmake bison flex file libsystemd-dev tzdata \
-		devscripts cdbs pkg-config openjdk-${OPENJDK_MAJOR_VERSION}-jdk zip debhelper`,
+		build-essential bison flex file libsystemd-dev tzdata \
+		devscripts cdbs pkg-config openjdk-${OPENJDK_MAJOR_VERSION}-jdk zip debhelper` + installCMake,


Noble ships CMake 3.28.3

jefferbrecht · 2025-10-01T13:32:26Z

dockerfiles/compile.go

-		build-essential cmake bison flex file systemd-dev debhelper libsystemd-dev tzdata \
-		devscripts cdbs pkg-config openjdk-${OPENJDK_MAJOR_VERSION}-jdk zip`,
+		build-essential bison flex file systemd-dev debhelper libsystemd-dev tzdata \
+		devscripts cdbs pkg-config openjdk-${OPENJDK_MAJOR_VERSION}-jdk zip` + installCMake,


Plucky ships CMake 3.31.6

franciscovalentecastro · 2025-10-04T01:08:11Z

All tests passed. Merging.

franciscovalentecastro added the kokoro:force-run Forces kokoro to run integration tests on a CL label Sep 19, 2025

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Sep 19, 2025

franciscovalentecastro added the kokoro:force-run Forces kokoro to run integration tests on a CL label Sep 22, 2025

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Sep 22, 2025

franciscovalentecastro changed the title ~~Update fluent-bit to v3.2.2.~~ Update fluent-bit to v3.2.3. Sep 22, 2025

franciscovalentecastro marked this pull request as ready for review September 22, 2025 17:56

franciscovalentecastro requested review from a team, XuechunHou and jefferbrecht and removed request for a team September 22, 2025 19:07

franciscovalentecastro changed the title ~~Update fluent-bit to v3.2.3.~~ Update fluent-bit to v4.0.6. Sep 24, 2025

franciscovalentecastro force-pushed the fcovalente-fluent-bit-v3.2.2 branch from 9cf0195 to 8e52693 Compare September 24, 2025 19:19

franciscovalentecastro force-pushed the fcovalente-fluent-bit-v3.2.2 branch from 2f1d985 to 5373e09 Compare September 29, 2025 19:22

jefferbrecht reviewed Sep 30, 2025

View reviewed changes

kokoro/scripts/test/go_test.sh Outdated Show resolved Hide resolved

franciscovalentecastro added 15 commits September 30, 2025 17:26

Update fluent-bit to v3.2.2.

82b1ee7

Update fluent-bit to v3.2.3.

d552436

Update soak_test/README.

6be4f73

Update fluent-bit to v4.0.0.

6220414

Update fluent-bit to v3.2.10.

5757a51

Update fluent-bit to v4.0.6.

76c4ff5

Reverd doc changes.

60c444d

Set new cmake config option FLB_KAFKA=OFF.

fb42698

Run go mod tidy.

f172f97

Install newer cmake in debian and ubuntu.

63f5cf5

Run go mod tidy.

fb7ae64

Update all uses of go test ./... to exclude submodules/ folder fr…

a002deb

…om testing.

Update ci.yaml.

77eecf8

Fix use of go list.

bc5b341

Revert change in go_test.sh.

eb079fc

jefferbrecht reviewed Oct 1, 2025

View reviewed changes

franciscovalentecastro added 2 commits October 1, 2025 14:38

Remove redundant cmake configs and keep DFLB_KAFKA=Off.

abba5ad

Remove installCMake in Ubuntu.

dc016e3

franciscovalentecastro changed the title ~~Update fluent-bit to v4.0.6.~~ Update fluent-bit to v4.0.7. Oct 1, 2025

franciscovalentecastro added 11 commits October 1, 2025 14:54

Remove installCmake in bookworm.

618b465

Install visual studio from https://aka.ms/vs/17/release`.

d687c7a

Set 2022 instead of 2019.

4e7f5bf

Revamp Dockerfile.windows.

2627942

Add build setup.

d3b804e

Set DOPENSSL_ROOT_DIR and DFLB_LIBYAML_DIR.

c571c57

Follow latest Dockefile.windows from fluent-bit repo.

faf414e

Add comments and fix copy paths.

3e1c272

Set Grace: 0 to avoid flaky transformation tests.

8e34dde

Update comment.

12f7314

Cleanup Dockerfile.windows.

bbbc363

XuechunHou approved these changes Oct 3, 2025

View reviewed changes

franciscovalentecastro added the kokoro:force-run Forces kokoro to run integration tests on a CL label Oct 3, 2025

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Oct 3, 2025

franciscovalentecastro added the kokoro:force-run Forces kokoro to run integration tests on a CL label Oct 3, 2025

stackdriver-instrumentation-release removed the kokoro:force-run Forces kokoro to run integration tests on a CL label Oct 3, 2025

franciscovalentecastro merged commit 656dd8d into master Oct 4, 2025
62 checks passed

franciscovalentecastro deleted the fcovalente-fluent-bit-v3.2.2 branch October 4, 2025 01:08

franciscovalentecastro mentioned this pull request Oct 6, 2025

Update fluent-bit to 4.0.8. #2084

Merged

9 tasks

franciscovalentecastro mentioned this pull request Nov 10, 2025

[Draft] Trying timezone dst issue #2065

Closed

9 tasks

Conversation

franciscovalentecastro commented Sep 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issue

How has this been tested?

Checklist:

Uh oh!

franciscovalentecastro commented Sep 24, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Footnotes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

franciscovalentecastro commented Oct 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Comments

franciscovalentecastro commented Sep 19, 2025 •

edited

Loading

franciscovalentecastro Oct 1, 2025 •

edited

Loading