Skip to content

Conversation

@liferoad
Copy link
Contributor

@liferoad liferoad commented Sep 11, 2025

Add support for BigQuery GEOGRAPHY type which works with Well-Known Text (WKT) format. The change includes:

  • Adding GEOGRAPHY to type mappings in bigquery_tools and bigquery_schema_tools
  • Implementing GeographyType logical type in schemas.py
  • Adding comprehensive tests for GEOGRAPHY type conversion and schema integration

Fixes #31945

Test with Direct Runner:

cd sdks/python && ./scripts/run_integration_test.sh --runner TestDirectRunner --test_opts "apache_beam/io/gcp/bigquery_geography_it_test.py::BigQueryGeographyIntegrationTests" 

or

cd sdks/python && python -m pytest apache_beam/io/gcp/bigquery_geography_it_test.py -v --test-pipeline-options="--runner=TestDirectRunner --project=apache-beam-testing" -s 

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

Add support for BigQuery GEOGRAPHY type which works with Well-Known Text (WKT) format. The change includes:
- Adding GEOGRAPHY to type mappings in bigquery_tools and bigquery_schema_tools
- Implementing GeographyType logical type in schemas.py
- Adding comprehensive tests for GEOGRAPHY type conversion and schema integration
- Add _get_project method to handle project billing in BigQuery source
- Update tests to explicitly specify project parameter
- Fix geography test data formats and simplify test cases
- Add temporary storage location for file load tests
@liferoad liferoad marked this pull request as ready for review September 13, 2025 22:58
@liferoad
Copy link
Contributor Author

/gemini review

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for the BigQuery GEOGRAPHY type, which is a great addition. The changes are comprehensive, including updates to type mappings, a new logical type for GEOGRAPHY, and extensive integration tests.

My review includes a couple of suggestions for improving code maintainability by reducing duplication and simplifying a test pipeline. Overall, the implementation is solid and the test coverage is excellent. Thank you for this contribution!

@github-actions
Copy link
Contributor

Assigning reviewers:

R: @shunping for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@github-actions
Copy link
Contributor

Reminder, please take a look at this pr: @shunping

@liferoad
Copy link
Contributor Author

R: @unography

@github-actions
Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

@kennknowles
Copy link
Member

Can you iterate and get the tests green?

@liferoad
Copy link
Contributor Author

Can you iterate and get the tests green?

Failed tests were not related to this PR. Just rebased it.

@kennknowles
Copy link
Member

Can you iterate and get the tests green?

Failed tests were not related to this PR. Just rebased it.

Thank you! I know that they are often not related. But we need to just get them green and feel the pain of the flakes, and to be entirely sure. TBH I would like to make it impossible to merge on red without extra hoops to jump through, as I have seen on other repos. That would align incentives well...

@liferoad liferoad merged commit 30fd958 into apache:master Oct 15, 2025
151 of 161 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: Add support for GEOGRAPHY data type in BigQuery Storge Write API in Python

3 participants