Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/bigtable error handling #29885

Merged
merged 22 commits into from
Jan 11, 2024
Merged

Feature/bigtable error handling #29885

merged 22 commits into from
Jan 11, 2024
+491 −44

Commits on Dec 7, 2023

  1. first pass of wiring error handling into write files and adding tests

    @johnjcasey
    johnjcasey committed Dec 7, 2023
    Configuration menu
    Copy the full SHA
    c6ecbd4 View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2023

  1. fix error handling to solve constant filenaming policy returning a nu… 

    …ll destination
    @johnjcasey
    johnjcasey committed Dec 8, 2023
    Configuration menu
    Copy the full SHA
    ab68b1e View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2023

  1. fix tests, add a safety check to the error handler

    @johnjcasey
    johnjcasey committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    3d76ccd View commit details
    Browse the repository at this point in the history

  2. spotless

    @johnjcasey
    johnjcasey committed Dec 11, 2023
    Configuration menu
    Copy the full SHA
    abde5a3 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2023

  1. add documentation

    @johnjcasey
    johnjcasey committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    ce965d4 View commit details
    Browse the repository at this point in the history

  2. add textio error handler pass-through

    @johnjcasey
    johnjcasey committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    c61243d View commit details
    Browse the repository at this point in the history

Commits on Dec 18, 2023

  1. add avroio error handler pass-through

    @johnjcasey
    johnjcasey committed Dec 18, 2023
    Configuration menu
    Copy the full SHA
    de5bd4c View commit details
    Browse the repository at this point in the history

  2. add documentation to avroio

    @johnjcasey
    johnjcasey committed Dec 18, 2023
    Configuration menu
    Copy the full SHA
    92948aa View commit details
    Browse the repository at this point in the history

  3. wire error handling into bigtable io

    @johnjcasey
    johnjcasey committed Dec 18, 2023
    Configuration menu
    Copy the full SHA
    3e0cae5 View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2023

  1. catch batching error when errors are detected

    @johnjcasey
    johnjcasey committed Dec 19, 2023
    Configuration menu
    Copy the full SHA
    1f0b7f0 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2023

  1. continue work on bigtable error handling

    @johnjcasey
    johnjcasey committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    35c7124 View commit details
    Browse the repository at this point in the history

  2. break apart and retry batches when they fail in bigquery

    @johnjcasey
    johnjcasey committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    cba5bbd View commit details
    Browse the repository at this point in the history

Commits on Dec 28, 2023

  1. remove check for oversized mutation

    @johnjcasey
    johnjcasey committed Dec 28, 2023
    Configuration menu
    Copy the full SHA
    6d1cfeb View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/master' into feature/bigtable-… 

    …error-handling

# Conflicts:
#	sdks/java/core/src/main/java/org/apache/beam/sdk/io/FileIO.java
#	sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java
#	sdks/java/core/src/main/java/org/apache/beam/sdk/io/WriteFiles.java
#	sdks/java/core/src/test/java/org/apache/beam/sdk/io/WriteFilesTest.java
#	sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/io/AvroIO.java
    @johnjcasey
    johnjcasey committed Dec 28, 2023
    Configuration menu
    Copy the full SHA
    41508bb View commit details
    Browse the repository at this point in the history

Commits on Jan 2, 2024

  1. spotless

    @johnjcasey
    johnjcasey committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    d606f89 View commit details
    Browse the repository at this point in the history

  2. fix merge error

    @johnjcasey
    johnjcasey committed Jan 2, 2024
    Configuration menu
    Copy the full SHA
    ebc8577 View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2024

  1. address comments

    @johnjcasey
    johnjcasey committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    a420d52 View commit details
    Browse the repository at this point in the history

Commits on Jan 4, 2024

  1. add BigtableWriteIT test case. it unexpectedly passes

    @johnjcasey
    johnjcasey committed Jan 4, 2024
    Configuration menu
    Copy the full SHA
    c630bc1 View commit details
    Browse the repository at this point in the history

Commits on Jan 8, 2024

  1. support routing more malformed mutation types, clean ups based on PR … 

    …comments
    @johnjcasey
    johnjcasey committed Jan 8, 2024
    Configuration menu
    Copy the full SHA
    5a97efc View commit details
    Browse the repository at this point in the history

Commits on Jan 9, 2024

  1. make retrys non-async to remove race condition causing data loss

    @johnjcasey
    johnjcasey committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    8963686 View commit details
    Browse the repository at this point in the history

  2. spotless

    @johnjcasey
    johnjcasey committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    3086274 View commit details
    Browse the repository at this point in the history

  3. update comment on identifying data exception

    @johnjcasey
    johnjcasey committed Jan 9, 2024
    Configuration menu
    Copy the full SHA
    505319d View commit details
    Browse the repository at this point in the history