Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: implement MirrorinAsyncTable's scan() and scanAll() #131

Merged
merged 1 commit into from
Nov 26, 2021

Conversation

prawilny
Copy link
Collaborator

@prawilny prawilny commented Nov 2, 2021

IIRC we decided to be like Bigtable and accept only ScanResultConsumer (and not support AdvancedScanConsumer).
Some OOP-jitsu needs to be done so that we express this within Java type system (Bigtable just uses raw types rather than parametrized by consumer type 🤷 ).


This change is Reviewable

@prawilny prawilny changed the title feat: MirrorinAsyncTable: scan, scanAll feat: implement MirrorinAsyncTable#scan{,All} Nov 2, 2021
@prawilny prawilny changed the title feat: implement MirrorinAsyncTable#scan{,All} feat: implement MirrorinAsyncTable's scan() and scanAll() Nov 2, 2021
@prawilny prawilny linked an issue Nov 2, 2021 that may be closed by this pull request
Copy link
Collaborator

@mwalkiewicz mwalkiewicz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 2 of 3 files at r1, all commit messages.
Reviewable status: 2 of 3 files reviewed, 4 unresolved discussions (waiting on @mwalkiewicz and @prawilny)


bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 411 at r1 (raw file):

  @Override
  public void scan(Scan scan, C c) {

c -> consumer?


bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 418 at r1 (raw file):

          @Override
          public boolean onNext(Result result) {
            return ((ScanResultConsumer) c).onNext(result);

What is the user experience when Advanced consumer is passed? CastException? Can we do a check using Precondition (or whatever is this library called :/ ) that this specific class is passed and add a meaningful error that will explain why is Advanced not supported?

OTOH, can't we support both results consumers? if c isinstance Advanced { override and forward advanced } else if c ininstance NonAdvanced { override and forward non-advanced } else if c isinstance ScanResultConsumer { override and forward only the base interface } else throw . This way our code will be ready for bigtable implementing advanced consumer and the code won't be much more convoluted (if you move the logic to a separate method).


bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 439 at r1 (raw file):

  @Override
  public CompletableFuture<List<Result>> scanAll(Scan scan) {
    final CompletableFuture<List<Result>> future = new CompletableFuture<>();

Why can't we just call return primary.scanAll(scan)?


bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/test/java/com/google/cloud/bigtable/mirroring/hbase2_x/TestMirroringAsyncTable.java, line 896 at r1 (raw file):

nullable

do you need nullable here? I think that any() will be sufficient, because the method is not called anyways, so there won't be any null to match.

Copy link
Collaborator Author

@prawilny prawilny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 1 of 3 files reviewed, 4 unresolved discussions (waiting on @mwalkiewicz)


bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 411 at r1 (raw file):

Previously, mwalkiewicz (Mateusz Walkiewicz) wrote…

c -> consumer?

Done.


bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 418 at r1 (raw file):

Previously, mwalkiewicz (Mateusz Walkiewicz) wrote…

What is the user experience when Advanced consumer is passed? CastException? Can we do a check using Precondition (or whatever is this library called :/ ) that this specific class is passed and add a meaningful error that will explain why is Advanced not supported?

OTOH, can't we support both results consumers? if c isinstance Advanced { override and forward advanced } else if c ininstance NonAdvanced { override and forward non-advanced } else if c isinstance ScanResultConsumer { override and forward only the base interface } else throw . This way our code will be ready for bigtable implementing advanced consumer and the code won't be much more convoluted (if you move the logic to a separate method).

Changed to accepting both kinds of consumers (and thus the errors are going to be just as if the user used Bigtable directly).

The first version of code intercepted the consumers.
Then I remembered our f2f discussion that we can skip the reference counting here (as the user would crash the database if he tried to close it before calling ScanController#terminate() or receiving ScanConsumerBase#onComplete()) and so I removed the reference counting (which complicated the code).
But I pushed the intermediary code so you can hopefully check the difference using the previous version in reviewable.


bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/main/java/com/google/cloud/bigtable/mirroring/hbase2_x/MirroringAsyncTable.java, line 439 at r1 (raw file):

Previously, mwalkiewicz (Mateusz Walkiewicz) wrote…

Why can't we just call return primary.scanAll(scan)?

Changed it to as suggested.
Note that now scanAll() is never verified (in the previous version it used to be) using results from secondary database.


bigtable-hbase-mirroring-client-2.x-parent/bigtable-hbase-mirroring-client-2.x/src/test/java/com/google/cloud/bigtable/mirroring/hbase2_x/TestMirroringAsyncTable.java, line 896 at r1 (raw file):

Previously, mwalkiewicz (Mateusz Walkiewicz) wrote…
nullable

do you need nullable here? I think that any() will be sufficient, because the method is not called anyways, so there won't be any null to match.

It turns out I forgot to finish the test. Fixed it and nullable indeed wasn't needed.

@prawilny prawilny force-pushed the ac/integration-tests-2.x/1 branch 2 times, most recently from 8fb010a to 1dac7e4 Compare November 10, 2021 14:10
@mwalkiewicz mwalkiewicz changed the base branch from ac/integration-tests-2.x/1 to master November 26, 2021 14:13
Copy link
Collaborator Author

@prawilny prawilny left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 3 of 3 files at r2, all commit messages.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @mwalkiewicz)

@prawilny prawilny merged commit 09cea12 into master Nov 26, 2021
dopiera added a commit that referenced this pull request May 11, 2022
… review comments (googleapis#3347)

* chore: revert review comments

* feat: add MirroringOperationException exception markers (#125)

* feat: concurrent writes in MirroringBufferedMutator (#80)

* refactor: implement multiple argument operations on MirroringAsyncTable with specific operations rather than batch() (#75)

* feat: implement MirroringAsyncTable#getName() (#132)

* feat: use Logger rather than stdout in DefaultMismatchDetector (#128)

* feat: synchronous writes (#88)

* fix: implement heapSize method for RowCell (#111)

* feat: FlowController accounts for memory usage (#137)

* refactor: remove Configuration as a base of MirroringConfiguration (#127)

* feat: MirroringAsyncBufferedMutator (#81)

* refactor: rename WRITE_MISMATCH to SECONDARY_WRITE_ERROR (#138)

* fix: BufferedMutator close() waits for all secondary flushes to finish (#110)

* feat: 2.x reads sampling (#114)

* refactor: make MirroringResultScanner synchronize on itself rather than MirroringTable (#134)

* ConcurrentBufferedMutator integration tests (#135)

* feat: add synchronous MirroringConnection to 2.x (#109)

* fix: MirroringConnection in 2.x failed to compile (#139)

* fix: fix BufferedMutator ITs (#140)

* feat: run 1.x integration tests on MirroringConnection etc. from 2.x (#108)

* feat: 2.x - rewrite Increment and Append as Put in batch (#116)

* fix: fix build (#142)

* refactor: minor fixes after review (#117)

* feat: MirroringAsyncTable#getScanner() (#58)

* test: 2.x integration tests (#112)

* feat: implement MirroringAsyncBufferedMutatorBuilder (#144)

* feat: log rows and values in DefaultMismatchDetector (#129)

* fix: ITs - add expected parameter to MismatchDetectors (#153)

* fix: force Append and Increment to return results and discard that result before returning it to user (#136)

* fix: review fixes in utils

* fix: review fixes in BufferedMutator

* fix: review fixes in Faillog

* fix: fixed reference counting

* fix: review fixes in FlowController

* fix: review fixes in metrics

* fix: review fixes in verification

* fix: Review fixes in MirroringTable

* fix: review fixes in HBase 2.x client

* fix: fixes in ITs

* feat: MirrorinAsyncTable: scan(), scanAll() (#131)

* fix: review fixes in tests

* feat: MirroringConnection: timeout in close() and abort() (#133)

* feat: better mismatch detection of scan results (#130)

* feat: quickstart (#105)

* fix: 2.x scan ITs (#158)

* fix: DefaultMismatchDetector tests (#157)

* fix: ConcurrentBufferedMutator waits for both flushes to finish before closing (#161)

* fix: additional minor fixes after review (#163)

* fix: BufferedMutator review fixes (#164)

- Simplify #flush().
- Add javadocs.
- (sequential) Fix flush() exception handling.
- (sequential) Move error handling to a separate inner class.

* fix: PR fixes

* fix: report zeroed error metrics after successful operations

* fix: prepend MismatchDetectorCounter with Test to better reflect its purpose

* feat: Client-side timestamping (#165)

* fix: reduce timeout in TestBlocking to make the tests run faster

* fix: asyncClose -> closePrimaryAndScheduleSecondaryClose

* fix: remove unused Batcher#throwBatchDataExceptionIfPresent

* fix: remove unused Comparators#compareRows

* fix: extract failedReads from MatchingSuccessfulReadsResults to reduce confusion

* feat: remove unused MirroringTracer from FailedMutationLogger

* fix: MirroringAsyncBufferedMutator - test if failed mutation is passed to secondary write error consumer

* fix: TestMirroringAsyncTableInputModification typo fix

* fix: describe user flush() in Buffered Mutator in quickstart

* fix: MirroringBufferedMutator - move flush threshold from BufferedMutations to FlushSerializer

* refactor: MirroringBufferedMutator#close() - use AccumulatedExceptions insteand of List<Exception>

* BufferedMutator - add close timeout

* AsyncBufferedMutator - add close timeout

* fix: remove stale addSecondaryMutation comment

* fix: add a comment that addSecondaryMutation handles failed writes

* fix: unify implementations of flushBufferedMutatorBeforeClosing

* fix: BufferedMutator - throw exceptions on close

* fix: BufferedMutator - add comment explaining that chain of flush operations is created

* fix: BufferedMutator - clarify  comments

* fix: Concurrent BufferedMutator - fix throwFlushExceptionIfAvailable

* fix: explain why flush is called in Sequential BufferedMutator test

* fix: TestConcurrentMirroringBufferedMutator - make waiting for calls explicit

* refactor: BufferedMutator rename scheduleFlushAll() to scheduleFlush()

* refactor: make FlushSerializer non-static

* fix: BufferedMutator - use HierarchicalReferenceCounter

* feat: Add MirroringConnection constructor taking MirroringConfiguration

* refactor: move releaseReservations to finally

* fix: use less convoluted example in lastFlushFutures description

* fix: merge small Timeestamper files into a single file

* fix: add a comment explaining which exceptions are forwarded to the user and why in SequentialMirroringBufferedMutator

* fix: use UnsupportedOperationException instead of RuntimeException when forbidden mutation type is encountered

* fix: add comment explaining why batch is complicated

* fix: add a TODO to implement point writes without batch

Co-authored-by: Mateusz Walkiewicz <mwalkiewicz@unoperate.com>
Co-authored-by: Adam Czajkowski <prawilny@unoperate.com>
Co-authored-by: Kajetan Boroszko <kajetan@unoperate.com>
mwalkiewicz added a commit that referenced this pull request May 18, 2022
… review comments (googleapis#3347)

* chore: revert review comments

* feat: add MirroringOperationException exception markers (#125)

* feat: concurrent writes in MirroringBufferedMutator (#80)

* refactor: implement multiple argument operations on MirroringAsyncTable with specific operations rather than batch() (#75)

* feat: implement MirroringAsyncTable#getName() (#132)

* feat: use Logger rather than stdout in DefaultMismatchDetector (#128)

* feat: synchronous writes (#88)

* fix: implement heapSize method for RowCell (#111)

* feat: FlowController accounts for memory usage (#137)

* refactor: remove Configuration as a base of MirroringConfiguration (#127)

* feat: MirroringAsyncBufferedMutator (#81)

* refactor: rename WRITE_MISMATCH to SECONDARY_WRITE_ERROR (#138)

* fix: BufferedMutator close() waits for all secondary flushes to finish (#110)

* feat: 2.x reads sampling (#114)

* refactor: make MirroringResultScanner synchronize on itself rather than MirroringTable (#134)

* ConcurrentBufferedMutator integration tests (#135)

* feat: add synchronous MirroringConnection to 2.x (#109)

* fix: MirroringConnection in 2.x failed to compile (#139)

* fix: fix BufferedMutator ITs (#140)

* feat: run 1.x integration tests on MirroringConnection etc. from 2.x (#108)

* feat: 2.x - rewrite Increment and Append as Put in batch (#116)

* fix: fix build (#142)

* refactor: minor fixes after review (#117)

* feat: MirroringAsyncTable#getScanner() (#58)

* test: 2.x integration tests (#112)

* feat: implement MirroringAsyncBufferedMutatorBuilder (#144)

* feat: log rows and values in DefaultMismatchDetector (#129)

* fix: ITs - add expected parameter to MismatchDetectors (#153)

* fix: force Append and Increment to return results and discard that result before returning it to user (#136)

* fix: review fixes in utils

* fix: review fixes in BufferedMutator

* fix: review fixes in Faillog

* fix: fixed reference counting

* fix: review fixes in FlowController

* fix: review fixes in metrics

* fix: review fixes in verification

* fix: Review fixes in MirroringTable

* fix: review fixes in HBase 2.x client

* fix: fixes in ITs

* feat: MirrorinAsyncTable: scan(), scanAll() (#131)

* fix: review fixes in tests

* feat: MirroringConnection: timeout in close() and abort() (#133)

* feat: better mismatch detection of scan results (#130)

* feat: quickstart (#105)

* fix: 2.x scan ITs (#158)

* fix: DefaultMismatchDetector tests (#157)

* fix: ConcurrentBufferedMutator waits for both flushes to finish before closing (#161)

* fix: additional minor fixes after review (#163)

* fix: BufferedMutator review fixes (#164)

- Simplify #flush().
- Add javadocs.
- (sequential) Fix flush() exception handling.
- (sequential) Move error handling to a separate inner class.

* fix: PR fixes

* fix: report zeroed error metrics after successful operations

* fix: prepend MismatchDetectorCounter with Test to better reflect its purpose

* feat: Client-side timestamping (#165)

* fix: reduce timeout in TestBlocking to make the tests run faster

* fix: asyncClose -> closePrimaryAndScheduleSecondaryClose

* fix: remove unused Batcher#throwBatchDataExceptionIfPresent

* fix: remove unused Comparators#compareRows

* fix: extract failedReads from MatchingSuccessfulReadsResults to reduce confusion

* feat: remove unused MirroringTracer from FailedMutationLogger

* fix: MirroringAsyncBufferedMutator - test if failed mutation is passed to secondary write error consumer

* fix: TestMirroringAsyncTableInputModification typo fix

* fix: describe user flush() in Buffered Mutator in quickstart

* fix: MirroringBufferedMutator - move flush threshold from BufferedMutations to FlushSerializer

* refactor: MirroringBufferedMutator#close() - use AccumulatedExceptions insteand of List<Exception>

* BufferedMutator - add close timeout

* AsyncBufferedMutator - add close timeout

* fix: remove stale addSecondaryMutation comment

* fix: add a comment that addSecondaryMutation handles failed writes

* fix: unify implementations of flushBufferedMutatorBeforeClosing

* fix: BufferedMutator - throw exceptions on close

* fix: BufferedMutator - add comment explaining that chain of flush operations is created

* fix: BufferedMutator - clarify  comments

* fix: Concurrent BufferedMutator - fix throwFlushExceptionIfAvailable

* fix: explain why flush is called in Sequential BufferedMutator test

* fix: TestConcurrentMirroringBufferedMutator - make waiting for calls explicit

* refactor: BufferedMutator rename scheduleFlushAll() to scheduleFlush()

* refactor: make FlushSerializer non-static

* fix: BufferedMutator - use HierarchicalReferenceCounter

* feat: Add MirroringConnection constructor taking MirroringConfiguration

* refactor: move releaseReservations to finally

* fix: use less convoluted example in lastFlushFutures description

* fix: merge small Timeestamper files into a single file

* fix: add a comment explaining which exceptions are forwarded to the user and why in SequentialMirroringBufferedMutator

* fix: use UnsupportedOperationException instead of RuntimeException when forbidden mutation type is encountered

* fix: add comment explaining why batch is complicated

* fix: add a TODO to implement point writes without batch

Co-authored-by: Mateusz Walkiewicz <mwalkiewicz@unoperate.com>
Co-authored-by: Adam Czajkowski <prawilny@unoperate.com>
Co-authored-by: Kajetan Boroszko <kajetan@unoperate.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2.x: implement MirroringAsyncTable::scan
2 participants