Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cassandra Driver Config Loader for Cassandra Driver Config in GCS #2077

Merged
merged 1 commit into from
Dec 24, 2024
Merged

Cassandra Driver Config Loader for Cassandra Driver Config in GCS #2077

merged 1 commit into from
Dec 24, 2024

Conversation

VardhanThigle
Copy link
Contributor

@VardhanThigle VardhanThigle commented Dec 19, 2024 •
edited
Loading

Overview

  1. A common static utility to load Cassandra Driver config from GCS.
  2. The implementation uses the Cassandra Driver APIs to parse the file, so all the options that the driver supports (now and with upgrades) are supported for parsing out of the box.
  3. The implementation also provides utility to serialize and deserialize the Config to OptionMap

Testing

  1. All lines covered in UT.
  2. UT to parse the reference config as provided by the Cassandra community.

@VardhanThigle VardhanThigle added addition New feature or request ignore-for-release labels Dec 19, 2024
@codecov Codecov
Copy link

codecov bot commented Dec 19, 2024 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 54.31%. Comparing base (8e7ae4f) to head (56878f5).
Report is 2 commits behind head on main.

Additional details and impacted files 
@@             Coverage Diff              @@
##               main    #2077      +/-   ##
============================================
+ Coverage     46.35%   54.31%   +7.95%     
+ Complexity     3932     1545    -2387     
============================================
  Files           858      391     -467     
  Lines         51259    21402   -29857     
  Branches       5391     2145    -3246     
============================================
- Hits          23763    11625   -12138     
+ Misses        25783     9085   -16698     
+ Partials       1713      692    -1021
Components Coverage Δ
spanner-templates 69.57% <100.00%> (+1.39%) ⬆️
spanner-import-export ∅ <ø> (∅)
spanner-live-forward-migration 76.67% <100.00%> (+0.25%) ⬆️
spanner-live-reverse-replication 77.93% <100.00%> (+0.20%) ⬆️
spanner-bulk-migration 87.30% <100.00%> (+0.08%) ⬆️
Files with missing lines Coverage Δ
.../migrations/utils/CassandraDriverConfigLoader.java 100.00% <100.00%> (ø)

... and 485 files with indirect coverage changes

@VardhanThigle VardhanThigle force-pushed the cassandra-mapping branch 5 times, most recently from 87402e0 to 3275661 Compare December 23, 2024 12:06
@VardhanThigle VardhanThigle marked this pull request as ready for review December 23, 2024 12:32
@VardhanThigle VardhanThigle requested a review from a team as a code owner December 23, 2024 12:32
@VardhanThigle VardhanThigle changed the title [DRAFT-NOT-FOR-REVIEW] CassandraDriverConfigLoader from GCS CassandraDriverConfigLoader from GCS Dec 23, 2024
@VardhanThigle VardhanThigle changed the title CassandraDriverConfigLoader from GCS Cassandra Driver Config Loader for Cassandra Driver Config in GCS Dec 23, 2024
bharadwaj-aditya
bharadwaj-aditya reviewed Dec 24, 2024
View reviewed changes
Copy link
Contributor

@bharadwaj-aditya bharadwaj-aditya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few minor comments. Looks fine overall.

.../java/com/google/cloud/teleport/v2/spanner/migrations/utils/CassandraDriverConfigLoader.java Outdated Show resolved Hide resolved
.../java/com/google/cloud/teleport/v2/spanner/migrations/utils/CassandraDriverConfigLoader.java Outdated Show resolved Hide resolved
.../java/com/google/cloud/teleport/v2/spanner/migrations/utils/CassandraDriverConfigLoader.java Outdated Show resolved Hide resolved
.../java/com/google/cloud/teleport/v2/spanner/migrations/utils/CassandraDriverConfigLoader.java Outdated Show resolved Hide resolved
@bharadwaj-aditya bharadwaj-aditya mentioned this pull request Dec 24, 2024
Merged
bharadwaj-aditya
bharadwaj-aditya approved these changes Dec 24, 2024
View reviewed changes
Copy link
Contributor

@bharadwaj-aditya bharadwaj-aditya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@VardhanThigle VardhanThigle removed the addition New feature or request label Dec 24, 2024
@VardhanThigle VardhanThigle merged commit 177a218 into GoogleCloudPlatform:main Dec 24, 2024
15 checks passed
pawankashyapollion pushed a commit to ollionorg/DataflowTemplates-fork that referenced this pull request Dec 28, 2024
@VardhanThigle @pawankashyapollion
CassandraDriverConfigLoader from GCS (GoogleCloudPlatform#2077)
98036e3
akashthawaitcc pushed a commit to ollionorg/DataflowTemplates-fork that referenced this pull request Dec 28, 2024
@pawankashyapollion @liferoad
@labs-code-app @VardhanThigle
Cassandra metadata PR To accomodate Driver Loader Class (#33)
0e3cca3 
* Removed *

* Create README.md for UDF samples (GoogleCloudPlatform#2083)

This commit adds a README.md file to the  directory. The README file provides descriptions for each of the sample Javascript UDF files in the directory, including their purpose and usage examples.

Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com>

* CassandraDriverConfigLoader from GCS (GoogleCloudPlatform#2077)

* Added Config File Path

* Added Fix for Loading Driver Options

* Added Dependecy Fixes

* Fix UT

---------

Co-authored-by: liferoad <huxiangqian@gmail.com>
Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com>
Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>
pawankashyapollion added a commit to ollionorg/DataflowTemplates-fork that referenced this pull request Dec 30, 2024
@pawankashyapollion @liferoad
@labs-code-app @VardhanThigle
Cassandra metadata PR To accomodate Driver Loader Class (#33)
2235312 
* Removed *

* Create README.md for UDF samples (GoogleCloudPlatform#2083)

This commit adds a README.md file to the  directory. The README file provides descriptions for each of the sample Javascript UDF files in the directory, including their purpose and usage examples.

Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com>

* CassandraDriverConfigLoader from GCS (GoogleCloudPlatform#2077)

* Added Config File Path

* Added Fix for Loading Driver Options

* Added Dependecy Fixes

* Fix UT

---------

Co-authored-by: liferoad <huxiangqian@gmail.com>
Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com>
Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>
taherkl pushed a commit to ollionorg/DataflowTemplates-fork that referenced this pull request Jan 2, 2025
@pawankashyapollion @labs-code-app @VardhanThigle
Fixed UT for Metadata config (#30)
e659d0e 
Co-authored-by: Narendra Rajput <narendra.rajput@ollion.com>

Cassandra metadata PR To accomodate Driver Loader Class (#33)

* Removed *

* Create README.md for UDF samples (GoogleCloudPlatform#2083)

This commit adds a README.md file to the  directory. The README file provides descriptions for each of the sample Javascript UDF files in the directory, including their purpose and usage examples.

Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com>

* CassandraDriverConfigLoader from GCS (GoogleCloudPlatform#2077)

* Added Config File Path

* Added Fix for Loading Driver Options

* Added Dependecy Fixes

* Fix UT

---------

Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com>
Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>
taherkl pushed a commit to ollionorg/DataflowTemplates-fork that referenced this pull request Jan 2, 2025
@pawankashyapollion @labs-code-app @VardhanThigle
Fixed UT for Metadata config (#30)
b581577 
Co-authored-by: Narendra Rajput <narendra.rajput@ollion.com>

Cassandra metadata PR To accomodate Driver Loader Class (#33)

* Removed *

* Create README.md for UDF samples (GoogleCloudPlatform#2083)

This commit adds a README.md file to the  directory. The README file provides descriptions for each of the sample Javascript UDF files in the directory, including their purpose and usage examples.

Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com>

* CassandraDriverConfigLoader from GCS (GoogleCloudPlatform#2077)

* Added Config File Path

* Added Fix for Loading Driver Options

* Added Dependecy Fixes

* Fix UT

---------

Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com>
Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>
shreyakhajanchi pushed a commit that referenced this pull request Jan 7, 2025
@akashthawaitcc @taherkl
@pawankashyapollion @labs-code-app @VardhanThigle @darshan-sj
Metadata config and pipeline options (#2081)
23be7bc 
* Added Connection Helper

* Added Formatter

* Added File Reader

* Refectored

* Added Changes for Schema and Spanner Schema

* Added Schema Changes to read Spanner Table in Schema object

* Added Schema Changes

* Added Changes for Schma

* Added Source Writn Fn Changes

* Added Source Factory Changes

* Added Fixed for the Source factory and Casssandra Connection helper

* Added Cassandra Schema Reader

* Added Pipeline Process

* Removed Unwanted Validation

* Added Access validator

* removed unwanted Return

* Added Thread safe optimization in cassandra Connection helper

* Applied  spotless:apply

* spotless:apply

* Added Constructor for Test case

* Added DUMMY Generator For UT

* Fixed UT for Metadata config (#30)

Co-authored-by: Narendra Rajput <narendra.rajput@ollion.com>

Cassandra metadata PR To accomodate Driver Loader Class (#33)

* Removed *

* Create README.md for UDF samples (#2083)

This commit adds a README.md file to the  directory. The README file provides descriptions for each of the sample Javascript UDF files in the directory, including their purpose and usage examples.

Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com>

* CassandraDriverConfigLoader from GCS (#2077)

* Added Config File Path

* Added Fix for Loading Driver Options

* Added Dependecy Fixes

* Fix UT

---------

Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com>
Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>

* PR Review Comments (#35)

* Convert it to builder Pattern

* Convert Waring to Error

* remove the unwanted comments

* Removed Unwanted options

* Address the PR

* Address the PR Comments

* Added Missing Getter for Configuration

* Address to removed configuration changes

* removed unwanted getters

* Removal of getter from cassandra dao and test case fixes

* Handle Optionmap in cassandra DAO for serialization

* removed dependecy of getKeyOrder

* Removed And Update Exception

* Fix Checkstyle Voilation

* Missing UT Added (#37)

* Added Dummy Test case

* Removed * from import

* removed unwanted

* Added Fixes

* Added test case and fixes

* Added Some more PR comments

* Added Test case for Cassandra Reader

* Added New testcase

* Added Dependecny

* Update test case  and remove Dependency of Jupiter (#39)

* Update test case  and remove Dependecy of jupiter

* Added UT fixes

* Added Missing Getter for Configuration

* [Sourcedb-to-spanner] Bulk migration Mysql to spanner 1tb Load test (#2063)

* [Sourcedb-to-spanner] Bulk migration Mysql to spanner 1tb Load test

* Updating row counts, added static sql resources

* Renaming the test and addressing comments

* PipelineController Changes for Cassandra (#2086)

* Fix Spanner Load tests and add display test report (#2092)

* correcting lt failures

* Adding test report

* Meta code coverage (#41)

Meta code coverage

* Pr bug fixes (#42)

* Schema Reader Optimization (#43)

* Added Pr Fix related to Changes Year in Javadoc

* Address the revert of 2025 and removal of extra constructor

* Added Extra Testcase to reach target patch

---------

Co-authored-by: taherkl <taher.lakdawala@ollion.com>
Co-authored-by: Taher Lakdawala <78196491+taherkl@users.noreply.github.com>
Co-authored-by: pawankashyapollion <v-pawan.kumar@ollion.com>
Co-authored-by: labs-code-app[bot] <161369871+labs-code-app[bot]@users.noreply.github.com>
Co-authored-by: Vardhan Vinay Thigle <39047439+VardhanThigle@users.noreply.github.com>
Co-authored-by: darshan-sj <darshan-sj@users.noreply.github.com>
@taherkl taherkl mentioned this pull request Jan 7, 2025
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bharadwaj-aditya bharadwaj-aditya bharadwaj-aditya approved these changes

@darshan-sj darshan-sj Awaiting requested review from darshan-sj darshan-sj is a code owner automatically assigned from GoogleCloudPlatform/spanner-migrations-team

@Deep1998 Deep1998 Awaiting requested review from Deep1998 Deep1998 is a code owner automatically assigned from GoogleCloudPlatform/spanner-migrations-team

@shreyakhajanchi shreyakhajanchi Awaiting requested review from shreyakhajanchi

Assignees
No one assigned
Labels
ignore-for-release size/XXL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@VardhanThigle @bharadwaj-aditya