Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BQ Storage Write API wrapper for Python SDK #25521

Closed
wants to merge 10,000 commits into from

Conversation

ahmedabu98
Copy link
Contributor

Implementing a wrapper for Python SDK that uses Java's Storage API SchemaTransform to write to BigQuery

Fixes #21961

pabloem and others added 30 commits January 18, 2023 12:53
Bumps [torch](https://github.com/pytorch/pytorch) from 1.12.1 to 1.13.1.
- [Release notes](https://github.com/pytorch/pytorch/releases)
- [Changelog](https://github.com/pytorch/pytorch/blob/master/RELEASE.md)
- [Commits](pytorch/pytorch@v1.12.1...v1.13.1)

---
updated-dependencies:
- dependency-name: torch
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…#25072)

This was added as a temporary work-around when the XLang expansion service did not support returning additional dependencies. This removes that dependency prevent conflicts and allowing the user to control which Kafka dependency version they want to use.

For apache#25070
…ction. (apache#24871)

* timer fix rough draft (apache#24617)

* comments (apache#24617)

* comments (1) (apache#24617)

* cancel run on reset (apache#24617)

* regenerated mocks (apache#24617)

* runStopDate (apache#24617)

* OutputFilterTypeController (apache#24617)

* resetErrorMessageText (apache#24617)

Co-authored-by: alexeyinkin <leha@inkin.ru>

* resetErrorMessageText (apache#24617)

Co-authored-by: alexeyinkin <leha@inkin.ru>

* comments (apache#24617)

* missing license (apache#24617)

* a commit for re-running rat check on github

* deleted comment "a commit for re-running rat check on github"

* fixes after merge with master (apache#24617)

* merge fixes (apache#24617)

* regenerated yaml files (apache#24617)

* snippetEditingControllerGetter refinement (apache#24617)

Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com>
Co-authored-by: alexeyinkin <leha@inkin.ru>
…5075)

Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.107.0 to 0.108.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](googleapis/google-api-go-client@v0.107.0...v0.108.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix input type

* use sys for size
…che#25080)

* Add support for templating in Tour of Beam learning materials

* Process learning materials through the template engine

* refactoring ci/cd for templated learning materials

* adding template processing

* refactoring learning content

* fixing incorrect sample path

* minor formatting comments

* missed complexity added add code review comments

* fix possible  err overwrite

Co-authored-by: Timur Sultanov <timur.sultanov@akvelon.com>
* Add support for using builtins typing

* lint and formatting

* tests

* fix tests

* fix tests

* fix tests

* fix tests
…he#25090)

Bumps [cloud.google.com/go/spanner](https://github.com/googleapis/google-cloud-go) from 1.42.0 to 1.43.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](googleapis/google-cloud-go@spanner/v1.42.0...spanner/v1.43.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/spanner
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [torch](https://github.com/pytorch/pytorch) from 1.12.0 to 1.13.1.
- [Release notes](https://github.com/pytorch/pytorch/releases)
- [Changelog](https://github.com/pytorch/pytorch/blob/master/RELEASE.md)
- [Commits](pytorch/pytorch@v1.12.0...v1.13.1)

---
updated-dependencies:
- dependency-name: torch
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>
…he#25095)

Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.28.1 to 1.29.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](googleapis/google-cloud-go@storage/v1.28.1...spanner/v1.29.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@ahmedabu98
Copy link
Contributor Author

Run Python_Xlang_Gcp_Dataflow PostCommit

@ahmedabu98 ahmedabu98 force-pushed the storage_write_wrapper branch from 35fd734 to 95f8f87 Compare March 1, 2023 21:38
@ahmedabu98 ahmedabu98 changed the base branch from master to Issue#21430 March 1, 2023 21:54
@ahmedabu98 ahmedabu98 changed the base branch from Issue#21430 to master March 1, 2023 21:55
@ahmedabu98 ahmedabu98 changed the base branch from master to mypy March 1, 2023 21:56
@ahmedabu98 ahmedabu98 changed the base branch from mypy to master March 1, 2023 21:57
@ahmedabu98 ahmedabu98 changed the base branch from master to asf-site March 1, 2023 21:59
@ahmedabu98 ahmedabu98 closed this Mar 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment