NO-JIRA: Updates CI images to use RHEL10 for early feedback

[bharath-b-rh]

The PR is for updating the base images used in the CI tests for operator and operand containers, from rhel-9 to rhel-10 for providing early feedback.

[openshift-ci-robot]

@bharath-b-rh: This pull request explicitly references no jira issue.

Details

In response to this:

The PR is for updating the base images used in the CI tests for operator and operand containers, from rhel-9 to rhel-10 for providing early feedback.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

Walkthrough

Updated base images in two CI Dockerfile configurations from UBI9-minimal to OpenShift RHCOS development image (4.21.0-ec.3-x86_64). Build steps and functionality remain unchanged across both files.

Changes

Cohort / File(s)	Summary
Docker base image updates images/ci/Dockerfile, images/ci/operand.Dockerfile	Changed base image from registry.access.redhat.com/ubi9-minimal:9.4 to registry.ci.openshift.org/rhcos-devel/ocp-4.21-10.1:4.21.0-ec.3-x86_64 in both files. No functional changes to build steps or entrypoints. Minor whitespace adjustment in operand.Dockerfile.

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

• Simple, repetitive base image updates with identical changes across two files
• No logic modifications or structural changes
• Straightforward dependency image version bump

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bharath-b-rh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [bharath-b-rh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[coderabbitai]

Actionable comments posted: 2

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to Reviews -> Disable Knowledge Base setting

📥 Commits

Reviewing files that changed from the base of the PR and between 8b0bfce and 754ffe8.

📒 Files selected for processing (2)

• images/ci/Dockerfile (1 hunks)
• images/ci/operand.Dockerfile (1 hunks)

🧰 Additional context used

📓 Path-based instructions (1)

**

⚙️ CodeRabbit configuration file

-Focus on major issues impacting performance, readability, maintainability and security. Avoid nitpicks and avoid verbosity.

Files:

• images/ci/operand.Dockerfile
• images/ci/Dockerfile

🔇 Additional comments (2)

images/ci/Dockerfile (1)

17-17: Verify runtime compatibility between rhel-9 builder and rhel-10 RHCOS final stage.

The builder stage uses a rhel-9 golang image, but the final stage now uses a rhel-10 RHCOS development image. Given that the binary is built with CGO enabled and strictfipsruntime tags, there is a risk of ABI incompatibility or missing runtime dependencies in the new RHCOS image.

Please verify:

1. The new image contains all required runtime libraries for a Go binary built with CGO_ENABLED=1 and strictfipsruntime tags
2. ABI compatibility between rhel-9 (build) and rhel-10 RHCOS (runtime) for the compiled binary

images/ci/operand.Dockerfile (1)

15-15: Confirm stability of early-release candidate image used in CI.

The 4.21.0-ec.3-x86_64 tag is an early candidate release from an internal CI registry. EC tags are non-stable pre-releases that may be replaced or deleted during development. Verify that this image will remain available throughout the testing period or switch to a stable release tag or SHA-based pinning.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, let's examine the Dockerfile and surrounding context
cat -n images/ci/Dockerfile | head -30

Repository: openshift/external-secrets-operator

Length of output: 214

---

🏁 Script executed:

# Check if there are other Dockerfiles using similar versioning patterns
find . -name "Dockerfile*" -type f | head -20

Repository: openshift/external-secrets-operator

Length of output: 214

---

🏁 Script executed:

# Search for other base image references to understand versioning conventions used
rg "registry\.ci\.openshift\.org" --type dockerfile -A 2 -B 2

Repository: openshift/external-secrets-operator

Length of output: 214

---

🏁 Script executed:

# Look for CI configuration or documentation about image stability
find . -type f \( -name "*.md" -o -name "*.txt" -o -name ".gitlab-ci.yml" -o -name "Jenkinsfile" -o -name ".github" \) | grep -E "(README|CI|pipeline|docker)" | head -20

Repository: openshift/external-secrets-operator

Length of output: 214

---

🌐 Web query:

OpenShift 4.21 early candidate ec.3 tag stability registry.ci.openshift.org

💡 Result:

Short answer: The 4.21 candidate tag ec.3 is present as an early-candidate payload on the CI registries, but it is a CI/preview (not GA) payload and can be changed or replaced — treat it as unstable for production. Evidence: the release dashboards show 4.21.0-ec.3 accepted/listed, and the candidate-4.21 channel listing shows candidate-4.21 -> 4.21.0-ec.3 (last modified Nov 28, 2025). [1][2][3]

Sources:

• OpenShift release dashboard (4.21 ec.* listings). [1]
• candidate-4.21 channel listing showing 4.21.0-ec.3. [2]
• multi-release dashboard / registry CI listing for 4.21 ec.* payloads. [3]

---

🌐 Web query:

OpenShift CI pipeline base image versioning ec early candidate tags

💡 Result:

Short answer: In OpenShift CI and release naming, "ec" in tags means an Early‑Candidate / Early‑Access pre‑release build. You’ll see tags like 4.21.0-ec.0 or component tags such as 4.12.0-ec.3 used for testing and preview releases (not final GA). Those builds appear in CI/release mirrors and in component lists and rpm-ostree/base-image version strings; they should be treated as pre-release/test images and referenced by full tag or digest when pinning in pipelines. [1][2][3]

Sources:

• Example release tag 4.21.0-ec.0 (microshift GitHub releases). [1]
• Red Hat blog/console output showing components with 4.12.0-ec.3. [2]
• OpenShift issue discussion showing use of "ec" candidate images in CI flows. [3]

---

Address early-candidate image instability risk in CI pipeline.

The base image tag 4.21.0-ec.3 is an Early-Candidate (pre-release) build, not a GA release. EC tags are explicitly marked as CI/preview payloads and can be changed, replaced, or removed. Using pre-release images for CI pipelines introduces instability risk. Verify whether this is intentional for testing purposes or if a stable release tag should be used instead. If early-candidate images are required, ensure the pipeline tolerates potential changes to the image or has fallback mechanisms.

🤖 Prompt for AI Agents

images/ci/Dockerfile lines 17-17: the Dockerfile uses an Early-Candidate base
image tag (4.21.0-ec.3) which is unstable for CI; either replace it with a
confirmed GA/stable release tag (or preferably pin to a specific stable image
digest) or, if the EC image is intentional for testing, document that choice in
the repo and add robustness: implement a fallback to a known-good image (or
image digest) and/or make the CI tolerate image changes by parameterizing the
base image via a pipeline variable so it can be quickly switched; update the
Dockerfile reference and CI pipeline variables/documentation accordingly.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# First, let's look at the operand.Dockerfile file to see the full context
cat -n images/ci/operand.Dockerfile

Repository: openshift/external-secrets-operator

Length of output: 214

---

🏁 Script executed:

# Also check the referenced images/ci/Dockerfile for comparison
cat -n images/ci/Dockerfile

Repository: openshift/external-secrets-operator

Length of output: 214

---

🏁 Script executed:

# Search for any other uses of these base images in the repo
rg "registry.ci.openshift.org" --type dockerfile -B 1 -A 1

Repository: openshift/external-secrets-operator

Length of output: 214

---

🌐 Web query:

RHEL 9 to RHEL 10 binary compatibility CGO compiled binaries

💡 Result:

Short answer: cgo-built Go binaries are not guaranteed to be 100% ABI‑stable across RHEL major releases because they link to the system C runtime and other native libraries. In practice:

• A binary compiled on RHEL 9 will usually run on RHEL 10 if it only uses symbols/APIs present in RHEL 9’s libraries, because newer glibc versions normally remain backward‑compatible for existing symbols. [1][2]
• RHEL 10 introduces new toolchain and library versions (glibc 2.39, GCC 14.x, zlib‑ng, different TBB, changes to long double ABI, etc.) and some libraries/packages are changed or replaced; those changes can break cgo binaries that depend on specific library ABIs or removed packages. Rebuilds may be required for components that changed ABI. [1][3][5]
• Red Hat’s compatibility guidance classifies ABIs and explicitly documents compatibility windows and when you must rebuild; consult the RHEL 10 Application Compatibility/Adoption notes for specifics. [2][3]

Practical recommendations

• If you need the widest portability, build on the oldest supported platform you intend to run on (build on RHEL 9 / UBI9 to run on RHEL 9 and usually RHEL 10). [2][6]
• Where possible avoid using non‑stable/undocumented C APIs, and prefer linking to stable interfaces.
• Run functional and integration tests on RHEL 10 (and any changed libraries like zlib/tbb) before deploying.
• If you must guarantee runtime independence, consider fully static linking or distributing a compatible runtime (note: static glibc has limitations and is not always practical).
• If you see breakage tied to a specific library (zlib‑ng, TBB, long double ABI, removed packages), rebuild the cgo code on RHEL 10 or vendor the needed library. [1][3][5]

Sources

• RHEL 10 “Considerations in adopting RHEL 10” (lists glibc/GCC versions and incompatible package changes). [1]
• Red Hat “RHEL 10: Application Compatibility / ABI compatibility” guidance. [2]
• RHEL 10 package/change notes (zlib‑ng, TBB, long double ABI notes). [3]
• RHEL container compatibility matrix (practical host/image compatibility guidance). [6]

If you want, tell me (a) whether your cgo binary is built static or dynamically, and (b) which libraries/symbols it depends on (ldd / objdump -T), and I’ll check likely compatibility and suggest next steps.

---

🌐 Web query:

strictfipsruntime OpenSSL RHEL 9 RHEL 10 compatibility

💡 Result:

Short answer

• strictfipsruntime is a Go toolchain/GOEXPERIMENT option (from the golang‑fips downstream) that enables “strict FIPS runtime” checks: a binary built with it will panic at startup if it detects it’s running in a FIPS environment but cannot find/use a compatible OpenSSL/FIPS provider. [1][2]

RHEL 9 vs RHEL 10 compatibility considerations

• RHEL 9 and RHEL 10 ship different OpenSSL/provider packaging and crypto‑policy defaults (both use OpenSSL 3.x in modern releases, but RHEL 10 removes compat-openssl11 and tightens default policies). That means a strictfipsruntime-built Go binary can run on RHEL 9 or 10 only if the target host provides a compatible libcrypto/OpenSSL + FIPS provider and acceptable crypto policy. If the binary cannot dlopen the expected OpenSSL/FIPS provider on that OS, it will crash under strictfipsruntime. [2][3]

Practical guidance

• Test your binaries on the exact RHEL minor you will run (9.x vs 10.x). Confirm presence of a compatible libcrypto.so (OpenSSL 3 FIPS provider) and that /proc/sys/crypto/fips_enabled (or OPENSSL_FORCE_FIPS_MODE/GOLANG_FIPS env vars) reflect the FIPS state you intend. Use GO_OPENSSL_VERSION_OVERRIDE for forced testing if needed. If you must support older ABI/compat libraries, note RHEL 10 removed compat-openssl11. [1][2][3]

Sources

1. golang-fips/go (strictfipsruntime, runtime checks, GOEXPERIMENT). [GitHub / pkg docs].
2. Red Hat Developer: FIPS mode for Red Hat Go Toolset (behavior, OpenSSL selection, env vars).
3. Red Hat docs — RHEL 10 security / considerations (compat-openssl11 removal, crypto‑policy changes, EMS/TLS differences).

---

🌐 Web query:

OpenShift RHCOS early candidate ec image stability production

💡 Result:

Short answer: Early Candidate (EC / candidate) RHCOS images are pre-release/testing builds and are not intended for production use — use GA images from the fast/stable channels for production clusters. See Red Hat’s release-channel guidance (candidate = release-candidate/testing; fast/stable = production-supported) and Red Hat examples that use EC images for testing/labs only. [1][2]

References

• OpenShift release-channel guidance (candidate vs fast/stable — candidate contains release candidates and is not supported for production). [1]
• Red Hat blog demonstrating EC RHCOS usage in lab/early-access scenarios (shows EC used for testing). [2]

---

🌐 Web query:

builder stage runtime OS version mismatch best practices container

💡 Result:

Short guidance and practical best practices when your build (builder) stage OS/version differs from the runtime stage in a container image:

1. Prefer true multi-stage builds and copy only artifacts into the final runtime image (don’t carry build tools). This reduces size and attack surface. [1][5]

2. Pin exact base-image tags for both builder and runtime (no floating "latest") so builds are reproducible. [5]

3. Match ABI/Libc/architecture expectations:

   • Know whether your build artifacts require glibc vs musl (Alpine uses musl; Debian/Ubuntu use glibc). A glibc-linked binary will often fail on musl. Either build for the runtime libc or pick a runtime base that provides the expected libc. [2][3]

4. When possible produce portable binaries:

   • Use static linking (or build with CGO_ENABLED=0 for Go) or build against the same libc used by the runtime. Static or musl-built binaries avoid libc mismatches. [3][7]

5. If you must run glibc-built binaries on musl images, only as a last resort use compatibility packages (gcompat/libc6-compat) or a glibc shim — but be aware of compatibility and security trade-offs. Prefer changing the runtime base instead. [3][2]

6. Use minimal, well-audited runtime images (distroless/scratch/alpine/slim) but only after validating compatibility of native extensions and system libraries. [1][4]

7. Test the final image in an environment that mirrors production (including OS version, libc, locales, timezone, seccomp/AppArmor policies) to catch runtime-only failures early. Automate smoke tests in CI. [1][4]

8. Expose and control build-time vs runtime configuration:

   • Pass needed compile-time flags / build args.
   • Avoid embedding build-time secrets; use build-time secrets features or CI secret handling. [5]

9. Keep runtime surface minimal:

   • Copy only required libraries and files; remove temp/build artefacts. Scan final image for known vulnerabilities before deployment. [5][4]

10. Document and enforce policies:

    • Specify supported base images, libc expectations, and how to produce compatible artifacts in your README/CI. Version-control Dockerfiles and base-image pins to make rollbacks straightforward. [5][1]

If you want, tell me the language/toolchain (Go, Java, Python, Node, C/C++) and the builder/runtime images you currently use and I’ll give concrete Dockerfile patterns and exact flags to avoid ABI/runtime mismatches.

---

Update to stable RHCOS GA image and verify runtime compatibility before deployment.

The operand binary is built with CGO_ENABLED=1 and strictfipsruntime, making it sensitive to RHEL version differences. Two concrete issues:

1. Early candidate image not suitable for production: The -ec.3 tag indicates an early/pre-release RHCOS image, not a supported GA release. Use a stable GA image from the fast or stable release channels instead.

2. strictfipsruntime requires RHEL 10 verification: RHEL 10 removed compat-openssl11 and changed OpenSSL/crypto-policy defaults. The strictfipsruntime runtime check will fail if the FIPS provider or compatible OpenSSL is unavailable. Test the compiled operand binary on the exact RHEL 10 image before deployment to confirm libcrypto.so and the FIPS provider are present and accessible.

Consider aligning the builder stage to RHEL 10 to match the final runtime and follow multi-stage build best practices.

🤖 Prompt for AI Agents

In images/ci/operand.Dockerfile around line 15, the base image uses an early
candidate RHCOS (-ec.3) which is not a GA release and may be incompatible with a
CGO-enabled binary using strictfipsruntime; replace the FROM with a stable GA
RHCOS image from the fast or stable release channel (pick the appropriate 4.21
GA tag), align your builder stage to the same RHEL/OS major (RHEL 10) used at
runtime to avoid ABI/openssl mismatches, and before deployment run the compiled
operand binary inside the chosen RHEL10/RHCOS GA image to verify libcrypto.so
and the FIPS provider are present and accessible (adjust packaging or install
compat-openssl11/polyfill equivalents if needed) so strictfipsruntime checks
succeed.

[openshift-ci]

@bharath-b-rh: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[bharath-b-rh]

no user facing changes, hence adding other required labels
/label docs-approved
/label px-approved
/label qe-approved

[bharath-b-rh]

/hold
The e2e jobs and image builds are going through fine with the RHEL10 base image. Keeping the PR open and not merging for now, until we have clarity on further steps on feedback.