[DO NOT MERGE] Validate Beam 2.69.0rc1 #2892

Amar3tto · 2025-10-17T13:49:36Z

No description provided.

gemini-code-assist · 2025-10-17T13:49:56Z

Summary of Changes

Hello @Amar3tto, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request is a "DO NOT MERGE" validation PR aimed at testing compatibility with the upcoming Apache Beam "2.69.0rc1" release. It involves updating the Beam version in the project's Maven "pom.xml" and all relevant Python dependency files ("default_base_python_requirements.txt", "default_base_yaml_requirements.txt", "base_requirements.txt", and "requirements.txt"). Additionally, the Maven profile activation for "validateCandidate" has been adjusted, and a new Python dependency, "google-cloud-secret-manager", has been introduced. The changes ensure the project can build and run with the new Beam release candidate, facilitating early detection of potential issues.

Highlights

Apache Beam Version Update: The core Apache Beam version has been upgraded to "2.69.0rc1" across both Java (Maven) and Python dependencies.
Maven Repository Configuration: A specific Maven repository URL ("https://repository.apache.org/content/repositories/orgapachebeam-1413") has been added to fetch the Beam release candidate.
Maven Profile Activation Change: The "validateCandidate" Maven profile activation logic in "pom.xml" was modified to activate based on the JDK version ("[1.)") instead of "activeByDefault".
Python Dependency Resolution Updates: The "requirements.txt" file has been extensively updated to reflect the new Beam version, including regenerated dependency hashes and corrected relative paths in comments.
New Python Dependency: "google-cloud-secret-manager" has been added as a new dependency in the Python "requirements.txt".

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

codecov · 2025-10-17T14:05:27Z

Codecov Report

❌ Patch coverage is 14.28571% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 49.97%. Comparing base (4f263f6) to head (0459b07).
⚠️ Report is 3 commits behind head on main.

Files with missing lines	Patch %	Lines
...le/cloud/teleport/spanner/spannerio/SpannerIO.java	14.28%	12 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #2892      +/-   ##
============================================
- Coverage     50.06%   49.97%   -0.09%     
- Complexity     4967     5345     +378     
============================================
  Files           967      967              
  Lines         59298    59312      +14     
  Branches       6441     6442       +1     
============================================
- Hits          29688    29642      -46     
- Misses        27505    27567      +62     
+ Partials       2105     2103       -2

Components	Coverage Δ
spanner-templates	`70.41% <14.28%> (-0.08%)`	⬇️
spanner-import-export	`68.86% <14.28%> (-0.20%)`	⬇️
spanner-live-forward-migration	`79.69% <ø> (ø)`
spanner-live-reverse-replication	`77.42% <ø> (ø)`
spanner-bulk-migration	`88.21% <ø> (ø)`

Files with missing lines	Coverage Δ
.../java/org/apache/beam/it/gcp/TemplateTestBase.java	`0.00% <ø> (ø)`
...le/cloud/teleport/spanner/spannerio/SpannerIO.java	`68.38% <14.28%> (-0.95%)`	⬇️

... and 8 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Amar3tto · 2025-10-18T05:12:02Z

Templates Python Wrapper Module .................... FAILURE [37:08 min]

  CSVToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2530488Z [ERROR]   CSVToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2531928Z [ERROR]   CSVToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2533326Z [ERROR]   CSVToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2534876Z [ERROR]   JdbcToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2536320Z [ERROR]   JdbcToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2538262Z [ERROR]   JdbcToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2539710Z [ERROR]   JdbcToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2541163Z [ERROR]   JdbcToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2542622Z [ERROR]   TextIOToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution

Amar3tto · 2025-10-20T13:19:44Z

Templates Python Wrapper Module .................... FAILURE [37:08 min]

  CSVToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2530488Z [ERROR]   CSVToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2531928Z [ERROR]   CSVToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2533326Z [ERROR]   CSVToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2534876Z [ERROR]   JdbcToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2536320Z [ERROR]   JdbcToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2538262Z [ERROR]   JdbcToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2539710Z [ERROR]   JdbcToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2541163Z [ERROR]   JdbcToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution
2025-10-17T18:53:19.2542622Z [ERROR]   TextIOToBigQueryIT>TemplateTestBase.setUpBase:253->TemplateTestBase.getSpecPath:306 » UncheckedExecution

The reason is customGcsTempLocation parameter, which is bigQueryLoadingTemporaryDirectory in DataflowTemplates. We need to set it during staging

Abacn · 2025-10-20T14:18:11Z

The reason is customGcsTempLocation parameter, which is bigQueryLoadingTemporaryDirectory in DataflowTemplates. We need to set it during staging

This sounds a upstream breaking change. Any clue why "bigQueryLoadingTemporaryDirectory" becomes needed in Beam 2.69.0 ?

checked that the check in Beam has always been there though: https://github.com/apache/beam/blob/af748d07a1a7ca244ff057554d68932c014d5eeb/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L3100

Amar3tto · 2025-10-20T14:36:50Z

The reason is customGcsTempLocation parameter, which is bigQueryLoadingTemporaryDirectory in DataflowTemplates. We need to set it during staging

This sounds a upstream breaking change. Any clue why "bigQueryLoadingTemporaryDirectory" becomes needed in Beam 2.69.0 ?

checked that the check in Beam has always been there though: https://github.com/apache/beam/blob/af748d07a1a7ca244ff057554d68932c014d5eeb/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L3100

It only relates to staging, I think it is because of apache/beam#36209. (.get())
Value only available at runtime, but accessed from a non-runtime context

Abacn · 2025-10-20T14:57:38Z

Thanks @Amar3tto, we should fix #36209 to (customGcsTempLocation.isAccessible() && customGcsTempLocation.get() == null). This essentially breaks BigQueryIO for runtime ValueProvider

Abacn · 2025-10-20T17:27:08Z

a good news is Python xlang template tests, used to fail on RC validations, now working after apache/beam#36199

Amar3tto added 2 commits October 17, 2025 17:14

Update base requirements and pom

d62d248

Update python requirements

dbe2aee

pull-request-size bot added the size/XXL label Oct 17, 2025

Amar3tto added the ignore-for-release label Oct 17, 2025

Fix OutputReceiverForFinishBundle, DataStreamMongoDBToFirestoreTest

e2870b2

Amar3tto requested review from a team as code owners October 17, 2025 16:20

Amar3tto requested review from aasthabharill and manitgupta October 17, 2025 16:20

Fix assert

51657ce

Add bigQueryLoadingTemporaryDirectory

0459b07

This was referenced Oct 20, 2025

Fix Null Pointer Exception in BigQueryIO apache/beam#36209

Merged

Fix BigQueryIO File load validate runtime value provider apache/beam#36564

Merged

Amar3tto closed this Oct 21, 2025

Use RC3

85f6000

Amar3tto reopened this Oct 21, 2025

Amar3tto closed this Oct 21, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[DO NOT MERGE] Validate Beam 2.69.0rc1 #2892

[DO NOT MERGE] Validate Beam 2.69.0rc1 #2892

Uh oh!

Amar3tto commented Oct 17, 2025

Uh oh!

gemini-code-assist bot commented Oct 17, 2025

Uh oh!

codecov bot commented Oct 17, 2025 •

edited

Loading

Uh oh!

Amar3tto commented Oct 18, 2025

Uh oh!

Amar3tto commented Oct 20, 2025 •

edited

Loading

Uh oh!

Abacn commented Oct 20, 2025 •

edited

Loading

Uh oh!

Amar3tto commented Oct 20, 2025

Uh oh!

Abacn commented Oct 20, 2025

Uh oh!

Abacn commented Oct 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

[DO NOT MERGE] Validate Beam 2.69.0rc1 #2892

[DO NOT MERGE] Validate Beam 2.69.0rc1 #2892

Uh oh!

Conversation

Amar3tto commented Oct 17, 2025

Uh oh!

gemini-code-assist bot commented Oct 17, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

codecov bot commented Oct 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Amar3tto commented Oct 18, 2025

Uh oh!

Amar3tto commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Abacn commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Amar3tto commented Oct 20, 2025

Uh oh!

Abacn commented Oct 20, 2025

Uh oh!

Abacn commented Oct 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Oct 17, 2025 •

edited

Loading

Amar3tto commented Oct 20, 2025 •

edited

Loading

Abacn commented Oct 20, 2025 •

edited

Loading