BQ Storage Write API wrapper for Python SDK #25521

Bumps [torch](https://github.com/pytorch/pytorch) from 1.12.1 to 1.13.1. - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/master/RELEASE.md) - [Commits](pytorch/pytorch@v1.12.1...v1.13.1) --- updated-dependencies: - dependency-name: torch dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…er (apache#25062)

This was lost with apache#23877

…#25072) This was added as a temporary work-around when the XLang expansion service did not support returning additional dependencies. This removes that dependency prevent conflicts and allowing the user to control which Kafka dependency version they want to use. For apache#25070

…ction. (apache#24871) * timer fix rough draft (apache#24617) * comments (apache#24617) * comments (1) (apache#24617) * cancel run on reset (apache#24617) * regenerated mocks (apache#24617) * runStopDate (apache#24617) * OutputFilterTypeController (apache#24617) * resetErrorMessageText (apache#24617) Co-authored-by: alexeyinkin <leha@inkin.ru> * resetErrorMessageText (apache#24617) Co-authored-by: alexeyinkin <leha@inkin.ru> * comments (apache#24617) * missing license (apache#24617) * a commit for re-running rat check on github * deleted comment "a commit for re-running rat check on github" * fixes after merge with master (apache#24617) * merge fixes (apache#24617) * regenerated yaml files (apache#24617) * snippetEditingControllerGetter refinement (apache#24617) Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com> Co-authored-by: alexeyinkin <leha@inkin.ru>

…5075) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.107.0 to 0.108.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.107.0...v0.108.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix input type * use sys for size

…che#25080) * Add support for templating in Tour of Beam learning materials * Process learning materials through the template engine * refactoring ci/cd for templated learning materials * adding template processing * refactoring learning content * fixing incorrect sample path * minor formatting comments * missed complexity added add code review comments * fix possible err overwrite Co-authored-by: Timur Sultanov <timur.sultanov@akvelon.com>

* Add support for using builtins typing * lint and formatting * tests * fix tests * fix tests * fix tests * fix tests

…he#25090) Bumps [cloud.google.com/go/spanner](https://github.com/googleapis/google-cloud-go) from 1.42.0 to 1.43.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@spanner/v1.42.0...spanner/v1.43.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/spanner dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…5009)

Bumps [torch](https://github.com/pytorch/pytorch) from 1.12.0 to 1.13.1. - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/master/RELEASE.md) - [Commits](pytorch/pytorch@v1.12.0...v1.13.1) --- updated-dependencies: - dependency-name: torch dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…he#25092)

…rements (apache#25088)

Closes apache#19929

Co-authored-by: tvalentyn <tvalentyn@users.noreply.github.com>

…using Storage API writes.

…he#25095) Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.28.1 to 1.29.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@storage/v1.28.1...spanner/v1.29.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…runner (apache#25031)

…che#25101) * Fix truncate copyjob when WRITE_TRUNCATE in BigQuery batch load

* [apache#24515] Delete the JRH Move :runners:google-cloud-dataflow-java:worker:legacy-worker to :runners:google-cloud-dataflow-java:worker deleting the FnApi runner harness (aka JRH) and the code that was powering it. Towards apache#25103

* Add I/O Standards Page * Address feedback, fix trailing whitespace * Address feedback * Address feedback * Add SchemaIO and FieldAccessDescriptor links * swap screenshot for relational table, address feedback * fix whitespaces * Fix doc linking

* Add more info in documentation * Add documentation for downloading model Co-authored-by: Shubham Krishna <“shubham.krishna@ml6.eu”>

…nt setup (apache#25115) * Fix Python PreCommit dependency compatibility test not run or redundant setup * Move pyarrow and pandas matrix unit test to preCommitPyCoverage * Fix Dataframe compatibility test not run * Update readme * Also move dataframe comp tests to preCommitPyCoverage

* Move CombineValues override to non portable overrides * Fix extraneously added tab * Create combinevalues_it_test.py * License * Add backslashes * Format * Lint * Fix imports * Validates runner test * Spacing * Update sdks/python/apache_beam/transforms/combiners_test.py Co-authored-by: Lukasz Cwik <lcwik@google.com> Co-authored-by: Lukasz Cwik <lcwik@google.com>

Remove kileys from Java reviewers

…che#25121)

Co-authored-by: Vincent Devillers <>

* Bump timeout for default transient expansion service * Fix spotbug warning

) * Handle @RequiresStableInput in portable flink (apache#20812) * [runners-flink] Remove unnnecessary dependency on flink-annotations * Fix @RequiresStableInput for portable Flink (apache#20812) Fix FlinkRequiresStableInputTest flakiness (apache#21333) * Flink: Tests for stateful stable dofns (apache#20812) * Enable commit for kafka flink portable test * Apply suggestions from code review Co-authored-by: Lukasz Cwik <lcwik@google.com> * Add callback to BufferingDoFnRunner for flushing SDK harness results * revert changes in website Co-authored-by: Lukasz Cwik <lcwik@google.com>

…#25128) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.11.47 to 1.11.48. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](aws/aws-sdk-go-v2@feature/s3/manager/v1.11.47...feature/s3/manager/v1.11.48) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…pache#24667)

…e on the runner side (apache#25104) This allows us to cleanup the existing code base now that the Dataflow JRH is gone and should improve the stability and performance for Java based portable runners built on top of this code since we have had a lot more use of the SDK harness variants at scale.

…e#25107) * Adding autoservice annotations to all SchemaTransformProviders * spotless

* Add working example for Per Entity Training * Fix linting and formatting * Add documentation, fix docstrings, linting * add dynamic destination writing * Update saving models, documentation * remove extra lines and fix import order * Update documentation * Update documentation Co-authored-by: Shubham Krishna <“shubham.krishna@ml6.eu”>

… Javadoc

…r2 (apache#25144) This is a follow-up for a comment in apache#25104 Also remove InboundDataClient which was missed in apache#25104

* Reenable postgres integration test * Update SpannerChangeStreamPostgresIT.java Co-authored-by: Nancy Xu <nancyxu@google.com>

* update cloudpickle version * regenerate python container dependencies

* WIP: playground_components works in tour-of-beam only with configureondemand set to false (apache#24563) * added evaluationDependsOn playground_components (apache#24563) * comments (1) (apache#24563) * removed goToContextLine after merge (apache#24563) Co-authored-by: darkhan.nausharipov <darkhan.nausharipov@kzn.akvelon.com>

…Stream (apache#25108) (apache#25141)

apache#25147) * Update typehint code to allow primitive composite types in python 3.9+ * linting * address comments

…table

…imeout

* Increase transient expansion service timeout

…#25182) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.11.48 to 1.11.49. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](aws/aws-sdk-go-v2@feature/s3/manager/v1.11.48...feature/s3/manager/v1.11.49) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

This adds the protos for querying any data samples from the SDK. Co-authored-by: Sam Rohde <srohde@google.com>

Updates the requirements on [chromedriver-binary](https://github.com/danielkaiser/python-chromedriver-binary) to permit the latest version. - [Release notes](https://github.com/danielkaiser/python-chromedriver-binary/releases) - [Commits](danielkaiser/python-chromedriver-binary@v100.0.4896.20.0...v110.0.5481.30.0) --- updated-dependencies: - dependency-name: chromedriver-binary dependency-type: direct:development ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

) Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.52.0 to 1.52.3. - [Release notes](https://github.com/grpc/grpc-go/releases) - [Commits](grpc/grpc-go@v1.52.0...v1.52.3) --- updated-dependencies: - dependency-name: google.golang.org/grpc dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Initial commit of boilerplate of change stream pipeline for bigtable

…ds to make schemas a bit easier to read in Java

…to schema-convenience-functions

….5.0

Fixing a table existence verifier for BQ

This PR closes apache#24469 and addresses apache#20312 by implementing CsvIO.Write and supporting classes. Please see apache#24469 for more details.

…es page.

* Fix Jdbc Write after window assigned * Assign last seen window to the remaining batch in finishBundle * Change to use GroupIntoBatches in both case * Use GroupIntoBatches in streaming only

…#25170)

* Add timeout to some gcp IO unit tests * Reduce local file written for BigQueryWriteTest.testWriteDynamicDestinations * Fix a BigQuery bug revealed with improved test where writer recreated too early

These ususally finish in 5-7 minutes in the CI environment, but sometimes take longer (perhaps depending on load).

* Ignore flags for beam_sql magic * formatted the code. * added comments for the fix --------- Co-authored-by: xqhu <xqhu@google.com>

…eters for StorageWrites

* Update/add torch versions * Update tox.ini * Update tested doc string * Add torch versions to precommit * Update sdks/python/test-suites/tox/py38/build.gradle Co-authored-by: Danny McCormick <dannymccormick@google.com> --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>

Bumps [torch](https://github.com/pytorch/pytorch) from 1.12.1 to 1.13.1. - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/master/RELEASE.md) - [Commits](pytorch/pytorch@v1.12.1...v1.13.1) --- updated-dependencies: - dependency-name: torch dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump XVR_Direct timeout to 2h * Only run GoUsingJava test suite once

…#25239) * SpannerChangeStreamPostgresIT has been causing beam_PostCommit_Java_DataflowV1 timing out since it was enabled

…ava runner (apache#25230) * fixing dependency management for java * use correct kafka client * revert unneeded changes * minor formatting * code review comment * extract version to variable

…#25270) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.11.49 to 1.11.50. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md) - [Commits](aws/aws-sdk-go-v2@feature/s3/manager/v1.11.49...feature/s3/manager/v1.11.50) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…tput for SDFs (resolves apache#23852) (apache#24837) Co-authored-by: Moritz Mack <mmack@talend.com> Co-authored-by: Jan Lukavský <je.ik@seznam.cz>

* Add model pcoll param to the RunInference Ptransform fix up parentheses Add time.sleep Update PredictionResult Update Pcollection type hint Add model_path param in the ModelHandler * Add sklearn side input example refactor name * Add ModeMetadata and some refactoring Update example * refactor _convert_to_result and add it to the utils.py Update model path in pytorch_inference.py Add model path to sklearn model Add model id to sklearn inference PredictionResult clean up base.py Update utils update base.py Update base.py * Add tag to the RunInference DoFn * Add enable_side_input_loading flag * Add helper functions Add default value Add prefix/human readable id to ModelMetadata * Add doc string, refactor utils code * Fix pytorch inference tests * Fix up sklearn inference * Remove logging * Add thread Lock when there is an update to side input * Check if side input is EmptySideInput * Add unit test for side input loading * Remove examples * Add log when side input path is updated * Add test to Dataflow Fix snippets tests revert build settings Fix lint * Refactor side input loading code Add model_id to PredictionResult * Add documentation, changelog * Add Singleton view doc * Fix whitespace, tests * fix weird spacing * Remove beam website udpate * Revert "fix weird spacing" This reverts commit f9a61a1. * Add WatchFilePattern transform * undo changes to beam wesbite * Pass side inputs only in streaming mode * Revert "Add WatchFilePattern transform" This reverts commit ae91ffe. * Add lines to website page * Add test * Add unit test to catch --streaming flag and Singleton SideInput * Addressing PR comments * Add logic to detect windows on side inputs * Add more tests * Remove redundant code * Update test * Fix lint * Add postcommit markers. * Remove `and` * fixup lint * Modify message * Add check for default model * Update message * Add validates runner * Fix test * Add PipelineVisitor for RunInference during construction time * Address comments based on PR * Remove restriction on the side inputs * Remove/add tests * Modify logging * Add tests * Add 2.46.0 change log * fix typo --------- Co-authored-by: Anand Inguva <anandinguva@google.com> Co-authored-by: Danny McCormick <dannymccormick@google.com>

Co-authored-by: Veronica Wasson <VeronicaWasson@users.noreply.github.com>

After apache/calcite#1740 field names don't matter in the planner. We do care about them for outputs, so add them back to the set of things that matter.

* Implement windowed write using CdapIO * Fix spotbugs

…rrupted data due to naive usage of unsafe storage (fixes apache#25296). (apache#25297)

…ng overhead in case of large complex query plans (relates to apache#24710 and apache#23845) (apache#25187)

* Fix test_hourly_team_score_output_checksum_on_small_input flaky * Include python bigquery integration test created tables to clean up job

* Implement FileWriteSchemaTransformProvider * Final cleanup * Remove unused code * Refactor file structure; add more unit tests * XmlRowAdapter Tests * Remove unused class * Remove unused test code * Further cleanup * Convert PDone to PCollection<String> * WIP * Refactor tests * wip * Final cleanup * Revert settings.gradle.kts * Patch SchemaAwareJavaBeansTest * Remove 2nd call to readpipeline run * Set numshards=1 for xml content tests * Patch code comments * Update OWNERS * Comply with Python SDK external transforms * Refactor Python SDK compliant String check

* Embed ML video to docs * Fix link * Spacing

* Add banner highlighting beam ml * Add ml inference video to videos and podcasts * Attribute video

…n invocation apache#21250 (apache#25245) This shows up whenever transforms use output receivers. For example map/flatmap where the calls are expected to be really inexpensive so we don't want to take on the overhead of creating an object. We saw a small performance improvement overall but best overall was that we reduced the size of the stack by 1 in these scenarios. Before: ``` Benchmark Mode Cnt Score Error Units ProcessBundleBenchmark.testLargeBundle thrpt 15 3147.619 ± 130.414 ops/s ``` After: ``` Benchmark Mode Cnt Score Error Units ProcessBundleBenchmark.testLargeBundle thrpt 15 3251.226 ± 138.822 ops/s ```

… size change. apache#21250 (apache#25219) * Optimize PGBK table to only update cache when there is a large enough size change. apache#21250 This prevents an expensive scenario where a user is outputting lots of small values (e.g. ints) to be precombined and hence takes little to no space to store so updating the cache provides little value. Note the 5-10x change for all types except for unique keys. Some early profiles show that there is an issue with the G1 garbage collector when storing so many small values that the GC management overhead dominates 75% of the execution which requires further investigation. Before: ``` Benchmark (distribution) (globallyWindowed) Mode Cnt Score Error Units PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine uniform true thrpt 5 8.306 ± 1.255 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine uniform false thrpt 5 7.849 ± 0.476 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine normal true thrpt 5 10.575 ± 1.295 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine normal false thrpt 5 10.772 ± 0.141 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine hotKey true thrpt 5 9.131 ± 2.761 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine hotKey false thrpt 5 8.302 ± 1.078 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine uniqueKeys true thrpt 5 3.899 ± 1.737 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine uniqueKeys false thrpt 5 4.203 ± 2.170 ops/s ``` After: ``` Benchmark (distribution) (globallyWindowed) Mode Cnt Score Error Units PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine uniform true thrpt 5 88.740 ± 8.925 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine uniform false thrpt 5 76.005 ± 5.150 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine normal true thrpt 5 43.388 ± 1.966 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine normal false thrpt 5 37.804 ± 7.177 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine hotKey true thrpt 5 84.881 ± 5.040 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine hotKey false thrpt 5 74.183 ± 2.063 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine uniqueKeys true thrpt 5 5.567 ± 4.068 ops/s PrecombineGroupingTableBenchmark.sumIntegerBinaryCombine uniqueKeys false thrpt 5 6.957 ± 1.508 ops/s ```

…ts to only be done during bundle creation and teardown apache#21250 (apache#25291) * Swap setting a context from being on the hot path when we emit elements to only be done during bundle creation and teardown. This cuts our per element processing overhead significantly down. For example in the ProcessBundleBenchmark.testLargeBundle we saw a reduction of about half of the execution state management overhead for each element (10.6% to 5.6% CPU overhead per element). The benchmark reflects this with a nice 11.5% performance improvement. Before: ``` Result "org.apache.beam.fn.harness.jmh.ProcessBundleBenchmark.testLargeBundle": 3265.141 ±(99.9%) 238.744 ops/s [Average] (min, avg, max) = (2952.616, 3265.141, 3525.482), stdev = 223.321 CI (99.9%): [3026.398, 3503.885] (assumes normal distribution) ``` After: ``` Result "org.apache.beam.fn.harness.jmh.ProcessBundleBenchmark.testLargeBundle": 3642.512 ±(99.9%) 45.865 ops/s [Average] (min, avg, max) = (3582.394, 3642.512, 3713.820), stdev = 42.902 CI (99.9%): [3596.647, 3688.377] (assumes normal distribution) ```

@experimental

* Make the name of the desired pcollection name public so other people don't have to guess. * WIP SQLTransformSchemaTransformProvider. Should be able to provide access to most of the SQLTransform parameters, though probably not the UDF implementation or the error transform. That would require substantial refactoring/surgery I think. * WIP. Using enumerations to make the configuration schema more discoverable (particularly for tableproviders and query planners which could change over time) * Basic version of the SQLTransformSchemaTransformProvider. Has basically a complete configuration and I think a (hacky) approach to getting DLQ information out of the transform at least good enough to use for testing. There's no unit test coverage in this module so not clear the existing external transform is tested/working either. * Remove clutter * Add @experimental annotation and always emit an errors PCollection even if there are no calculations. * Fix up checkstyle violation

* issue24170 added google colab link * fixed github icon svg * issue24170 fixed code * issue24170 colab new line * issue24170 colors fixed * issue24170 github and colab buttons at embedded playground * issue24170 pr fix * issue23924 dataset button (#415) * issue23924 dataset button * issue23924 fix pr * issue23924 embedded shorten buttons * minor fixes * issue 23924 show buttons text arg fixed

* Clarify llm download/loading instructions * Fix instructions * Default model name * Install beam * Provide some extra info on LLM

) * Allow for setMaxRetryJobs in BigQueryIO to be configurable * Allow for setMaxRetryJobs in BigQueryIO to be configurable

Make Python's BatchElements more aggressive batching in some cases, capping at 10 second rather than 1 second batches by default and excluding fixed cost in this computation.

…che#25344 (apache#25349) * Fix output timestamp for multi output receiver in FnApiDoFnRunner apache#25344 Fixes apache#25344

…en (apache#25300) there is contention on the SynchronousQueue. ThreadPoolExecutor creates a new thread if offer(Runnable) returns false. With a SynchronousQueue under contention, this may return false even if there are already idle threads in the pool. To avoid creating a thread unnecessarily and growing the pool we modify offer to wait with a small timeout of 100ms. This may delay initial execution on the pool but that should be rare as it is encountered only once the high-watermark is exceeded. Also add a timeout to the non-core threads so that the pool reduces size. Both of these changes are motivated to reduce memory usage due to per-thread buffers in Beam and AvroCoder. Add a test contending adds on the pool which previously showed the pool growing beyond the necessary amount (ie created 104 threads when 100 is sufficient).

We add this as a new option rather than the new behavior as a new option because we want the default behavior to change, but more imporantly we want target_batch_duration_secs=None or target_batch_duration_secs=VERY_LARGE_VALUE to produce the same qualitative effect.

… SCHEDULED -> RUNNING -> FINISHED for the change stream metadata table. (apache#25311) * Made sure state transitions were correct when updating partition state * Metadata Table changes * Comitted * Fixed failing unittests * Spotless apply --------- Co-authored-by: Nancy Xu <nancyxu@google.com>

…Streams (apache#25246)

…d. (apache#25313)

apache#25263)

* Add TensorRT runinference example for Text Classification * Add docstrings, fix linting, formatting * Fix pylinting for docstrings * Add partial documentation * Edit docstrings, documentation, reading of input * Fix python formatting * Add benchmarking results * Add benchmarking, remove print statment * Edit documentation * Update website/www/site/layouts/partials/section-menu/en/documentation.html --------- Co-authored-by: Shubham Krishna <“shubham.krishna@ml6.eu”> Co-authored-by: Danny McCormick <dannymccormick@google.com>

apache#25377) Bumps [cryptography](https://github.com/pyca/cryptography) from 39.0.0 to 39.0.1. - [Release notes](https://github.com/pyca/cryptography/releases) - [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst) - [Commits](pyca/cryptography@39.0.0...39.0.1) --- updated-dependencies: - dependency-name: cryptography dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Playground Frontend Test workflow (apache#24728) * Trigger the workflow (apache#24728) * Test * Test * Test * Test * Test * Test * Test * Test * Test * Test * Test * Test * Test * Fix * Fix * Clean up (apache#24728) * Test * Add Chrome requirement in README (apache#24728) * Polish README (apache#24728) * Fix a typo (apache#24728)

* Replace strings.Title in specialize package * Replace usage in shimx

* Explicitly set mongo container version for testing * Remove direct:py37:mongodbioIT from Py37PostCommit it is already covered by beam_PostCommit_Python_MongoDBIO_IT

…che#25370) * Add support for batching kwargs * sklearn and use named params * Lint * Formatting * f-strings

apache#25268) * Adding support for @SchemaFieldDescription annotation that allows annotations on AutoSchema classes * Fixing type annotation for method * fixup - use equals to compare descriptions * fixup

…e#24999) * Adding SpannerIO.readChangeStreams support for SchemaTransform * Improve test and schema shape to include changestream information * add schema checks * fixup * remove unneeded file * reducing the number of tables that we cache

…5250) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.108.0 to 0.109.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.108.0...v0.109.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add HL7v2ReadParameter for passing metadata while reading all HL7v2 Message * Add HL7v2ReadResponse to retrun metadata in response when reading all HL7v2 messages * Add Coder for HL7v2ReadResponse * Add api to read all hl7v2 messages by passing metadata Function would receive a PCollection of HL7v2ReadParameter which contains id of messages and metadata and return PCollection of HL7v2Response which contains hl7v2 messages and metadata. * Add unit test and inregration test to test api for reading HL7v2Message with passing metadata * Fix spotless java check. * Update java doc comment in HL7v2IO. * Update sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2ReadParameter.java Co-authored-by: Ahmed Abualsaud <65791736+ahmedabu98@users.noreply.github.com> * Update sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2ReadResponse.java Co-authored-by: Ahmed Abualsaud <65791736+ahmedabu98@users.noreply.github.com> * Add equals method for HL7v2ReadParameter, fixed doc and naming. --------- Co-authored-by: Kamran Koupayi <kamrankoupayi@google.com> Co-authored-by: Ahmed Abualsaud <65791736+ahmedabu98@users.noreply.github.com>

apache#25320)

* fix doc on bq sxtorage * commentaddress

* Fix Test_getRunOrTestCmd on Go 1.20 * Use go_cmp to compare test result with expected value

* [prism] Initial commit for READMEs and go.mod * [prism] ws lint & license --------- Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com>

Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com>

* Added element counter and error counter for BQ write schema transform * Fixied styling issues with naming * Combined two trivial counter class for brevity. Used finishbundle annotation to reduce the number of calls to counter.inc() for better performance. * fix formatting --------- Co-authored-by: Nick Li <nickllx@google.com>

* [prism] Add internal/config package * [prism] internal/config address comments --------- Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com>

…ipelines

Co-authored-by: Shubham Krishna <“shubham.krishna@ml6.eu”>

…cessing change stream metadata database (apache#25193)

)

…reating the WorkerPool for Java pipelines

… extra packages

* Remove PBD id filter from SampleDataRequest * rename SampleData fields to mirror other instructions

* initial commit pre-test for onnx pytorch * passed all pytorch uts * add tf initial implementation * tf tests passing * sklearn impl * sklearn uts passing * refactor * address pr comments * handle option args * correctly handle empty optional arguments * adding tox initial env for uts * remove docker file for testing * fix yapf and lint (remaining lint problems are related to unimplemented gcp-dependent tests) * merge with upstream * enable remote model loading * add gcp-based uts * add integration test * test library version in tox and gradle * yapf pylint * typo * fix review comments * fix pydoc * yapf * fix mypy * isort * refactor * fix default * add readme and changes --------- Co-authored-by: Ziqi Ma <ziqima@Ziqis-MacBook-Pro.local> Co-authored-by: Ubuntu <ziqima@linux1beam.r4iegivvmdhuzdpg21ky5j42xb.xx.internal.cloudapp.net>

…atchLoad (apache#25410) * Handle dynamic table destination in UpdateSchemaDestination impl * Add SCHEMA_UPDATE job type for schema update load * Fix BigQuerySchemaUpdateOptionsIT to actually test temp table scenario * Rewrite BigQuerySchemaUpdateOptionsIT.runWriteTestTempTable to test dynamicDestination scenario

Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com>

…8.0.3 in /sdks/python (apache#25292) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Fixes apache#25345

* Add support for loading torchscript models * Add tests * Add example and benchmark * Add validate_constructor_args method * Addressing comments, fixing types * Fix/add tests * Add change log * revert changes * Add few more checks and refactor * Fixup lint * Revert "Fixup lint" This reverts commit b18e10f. * Add ignore for mypy * Make validate_constructor_args local to pytorch handler

* (apache#25316) Added naive first shot at enabling LZMA compression * (apache#25316) Added a draft line to CHANGES.md * (apache#25316) fix linter issues * (apache#25316) update tests (draft) * (apache#25316) import order in test file

* Add WatchFilePattern transform * Remove defaults and update instructions * Add batch args * Refactor example based on comments * Changes based on PR comments * Fix typo * Fix up lint * Fix doc precommit * Fix up pydocs * Fixup lint * Fix up test * Update docstring as per comments * Add unittest.main * Update sdks/python/apache_beam/examples/inference/pytorch_image_classification_with_side_inputs.py Co-authored-by: Danny McCormick <dannymccormick@google.com> --------- Co-authored-by: Danny McCormick <dannymccormick@google.com>

* Validate that GBK coders are always set correctly. * Ensure deterministic coders are used in the PortableRunner. This was exposed by tests whose GBK ouput key coder was the non-determinisitc form of the input key coder. For most coders, including the fallback FastPrimitivesCoder, the actual encoding is bitwise compatible, but we should ensure this is always the case for correctness. --------- Co-authored-by: Lukasz Cwik <lcwik@google.com>

…are misused

* Adding support for DLQ for ZetaSQL * fixed issue for not-all-fields are selected * fixup * fix spotless * fix test

* Fix pulling licenses * Update sdks/java/container/license_scripts/dep_urls_java.yaml Co-authored-by: Bruno Volpato <bvolpato@google.com> --------- Co-authored-by: Bruno Volpato <bvolpato@google.com>

…mark to PrecombineGroupingTableBenchmark (apache#25452) This also adds the distribution type as a tag as well.

* Add Playground embedding examples to README (apache#22180) * Improve README (apache#22180) * Integration test to load an example by URL (apache#24959) * Move and improve the go example (apache#24959) * Update the example (apache#24959) * Move the example (apache#24959) * Add a test (apache#24959) * Remove unused code, add comments (apache#24959) * Revert changes in extracted symbols (apache#24959) * Fix after review (apache#24959) * Fix after review (apache#24959) * Add ExampleDescriptor instances to replace example constants (apache#24959) * Remove unused code (apache#24959) * Fix testing HttpExampleLoader with visible sections (apache#24959) * Strip ExampleViewOptions when opening standalone playgounrd from embedded one (apache#24959) * Upgrade app_state and flutter_code_editor (apache#24959) * Test default examples on frontend (apache#25131) (#406) * Test default examples on frontend (apache#25131) * Add SCIO example descriptor (apache#25131) * Add missing files (apache#25131) * Add license comments (apache#25131) * Minor renaming (apache#25131) * Restore the real running for Java (apache#25131) * Display missing errors (apache#24959) * Upgrade flutter_code_editor (apache#24959) * Update backend URLs (apache#24959) * Add HTML embedding example in README, remove obsolete parts (apache#24959) * Trigger integration tests workflow (apache#24959) * Upgrade Flutter to 3.7.3 in playground_frontend_test workflow (apache#24959) * Less specific output expectations in a test, add comments (apache#24959) * Update README (apache#24959) * Upgrade Flutter in frontend Dockerfile (apache#24959) * Add headless testing to README (apache#24959) * Typo (apache#24959)

…used by lack of proper expansion (apache#25465)

* Stage CsvIO refactor * Refactor to common test beans * Add simple tests for displaydata * WIP: implement Csv * Complete CsvWrite tests * Refactor assertFolderContainsInAnyOrder * Clarify CSV header handling * Patch missing applied configuration * Add CSV time format test

* add retries to connect with expansion service * prevent override of res * change retries * remove unnecessary module * Revert "remove unnecessary module" This reverts commit 1508185. * remove unnecessary pkg

* SpannerIO: parameterizing partitionQuery timeout * SpannerIO: parameterizing partitionRead timeout * Removing default values of timeouts * formatting changes

* Handling pg spanner.commit_timestamp type * deleting extra file

* go lints * added tf model handler and tests * lint and formatting changes * correct lints * more lints and formats * auto formatted with yapf * rm spare lines * add readme file * test requirement file * add test to gradle * add test tasks for tf * unit test * lints * updated inferenceFn type * add tox info for py38 * pylint * lints * using tfhub * added tf model handler and tests * lint and formatting changes * correct lints * more lints and formats * auto formatted with yapf * rm spare lines * merge master * test requirement file * add test to gradle * add test tasks for tf * unit test * lints * updated inferenceFn type * add tox info for py38 * pylint * lints * using tfhub * tfhub example * update doc * sort imports * resolve pydoc,precommit * fix import * fix lint * address comments * fix optional inference args * change to ml bucket * fix doc

* [prism] coders + initial test file. * [prism] internal/coders.go * [prism] finish coder coverage * [prism] no-alloc doc * review commit --------- Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com>

* Fix beam.Row.__eq__ for rows with trailing columns * Correct the condition to make sure we still check the insertion order of the elements. --------- Co-authored-by: Valentyn Tymofieiev <valentyn@google.com>

…#25381) * TFMA notebook showing ExtractEvaluateAndWriteResult Tranfsorm * Apply suggestions from code review * update tfma notebook to include feedback and extra visualisations

This is causing https://ci-beam.apache.org/job/beam_PreCommit_Whitespace_Cron/ to persistently fail

* Add back links removed in apache#24744 * Undo accidental changes

… from opensource.org

) * [prism] internal/jobservices metrics + tests * typo --------- Co-authored-by: lostluck <13907733+lostluck@users.noreply.github.com>

…lue (apache#25437) * Allow context param and error return value in SDF validation * Use context param and error return value in SDF method invocation * Run go fmt * Clean up error messages from "fn reflect.methodValueCall" * Validate return value count in a more correct way

) * Implement the functionality to sample/encode/decode elements * Implement Python data sampling * Final cleanup for Python SDK data sampling implementation * lint * lint * add data sampler test * finish data sampler tests * Add comments, encode in the nested context, sample based on time instead * run tests * fix tests * run tests * rebase based on new proto names * trigger tests * linter * replace mockito clocks with simple FakeClock --------- Co-authored-by: Sam Rohde <srohde@google.com>

…mbedding on the control response. (apache#25463) * Add support for all Java based portable runners to consume elements embedding on the control response. Fixes apache#25462

… OpenJDK (apache#25402) As discussed on https://lists.apache.org/thread/h4m02o0br5g5t3o3m8r4lkl431vvzhsg Fixes apache#25371

…uery execution project (apache#25400) * Update google-bigquery.md Include details on how to define BigQueryIO query execution project in Java SDK.

…BigQueryIO (apache#24274) * initial impl * force unit tests * addressing comments * spotless and checkstyle checks fixes * spotless and checkstyle checks fixes * addressing more comments * merge from master, fix tests and validations * copying the changes in AvroUtils located in core module to the avro extensions module

…xamples links in READMEs and CdapIO website page fixed

…d direct runner jobs to test on jenkins

…d python collect marker

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BQ Storage Write API wrapper for Python SDK #25521

BQ Storage Write API wrapper for Python SDK #25521

Commits on Jan 18, 2023

Commits on Jan 19, 2023

Commits on Jan 20, 2023

Commits on Jan 21, 2023

Commits on Jan 23, 2023

Commits on Jan 24, 2023

Commits on Jan 25, 2023

Commits on Jan 26, 2023

Commits on Jan 27, 2023

Commits on Jan 28, 2023

Commits on Jan 30, 2023

Commits on Jan 31, 2023

Commits on Feb 1, 2023

Commits on Feb 2, 2023

Commits on Feb 3, 2023

Commits on Feb 4, 2023

Commits on Feb 5, 2023

Commits on Feb 6, 2023

Commits on Feb 7, 2023

Commits on Feb 8, 2023

Commits on Feb 9, 2023

Commits on Feb 10, 2023

Commits on Feb 11, 2023

Commits on Feb 13, 2023

Commits on Feb 14, 2023

Commits on Feb 15, 2023

Commits on Feb 16, 2023

Commits on Feb 17, 2023

Commits on Feb 23, 2023

Commits on Mar 1, 2023