Add model for table column metadata for Global state #5136
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>
dinujoh
requested review from
sb2k16,
chenqi0805,
engechas,
graytaylor0,
kkondaka,
KarstenSchnitter,
dlvenable and
oeyh
as code owners
oeyh
reviewed
Oct 31, 2024
| private static final String DB_METADATA_KEY = "dbMetadata"; | ||
| private static final String TABLE_COLUMN_METADATA_KEY = "tableColumnDataTypeMap"; | ||
| private final DbMetadata dbMetadata; | ||
| private final Map<String, Map<String, String>> tableColumnDataTypeMap; |
There was a problem hiding this comment.
Can you clarify the key and value in this map? Maybe add a javadoc?
There was a problem hiding this comment.
It's Map of table name to table column data type map. e.g. { "table1" : { "column1" : "int", "column2" : "varchar" }, "table2" : { "column1" : "int" } }
Will add java doc
|
|
||
| import java.util.Map; | ||
|
|
||
| public class DbTableMetadata { |
There was a problem hiding this comment.
It's called DbTableMetadata, yet it doesn't contain metadata about the table, like table name and database (schema) name. Wondering how this model will be used.
There was a problem hiding this comment.
Ok. I got most of it from description. When do we need to pull DbMetadata from the coordination store?
There was a problem hiding this comment.
I will have following PRs. This will be stored in global state. It will be used in BinLogListener and ExportWorker.
chenqi0805
approved these changes
Oct 31, 2024
oeyh
approved these changes
Oct 31, 2024
san81
added a commit
to san81/data-prepper
that referenced
this pull request
Nov 2, 2024
# This is the 1st commit message: additional test coverage Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> # This is the commit message #2: cleaned up JiraOauthConfig file Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> # This is the commit message #3: addressing review comments and simplifying the exception handling Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> # This is the commit message #4: Add external origination time for events created from S3 Object (opensearch-project#5104) Signed-off-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> Co-authored-by: Krishna Kondaka <krishkdk@dev-dsk-krishkdk-2c-bd29c437.us-west-2.amazon.com> # This is the commit message #5: moved the wait block out of the catch block Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> # This is the commit message #6: Renewal logic adjusted Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> # This is the commit message #7: partial Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> fix merge issues Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> update Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> Add getColumnDataTypes method to SchemaManager to get datatype for table columns (opensearch-project#5135) Add getColumnDataTypes method to SchemaManager Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com> Add model for table column metadata for Global state (opensearch-project#5136) Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com> Rename the KDS source plugin name to "kinesis-data-streams" (opensearch-project#5138) Signed-off-by: Souvik Bose <souvbose@amazon.com> Co-authored-by: Souvik Bose <souvbose@amazon.com> Addressed review comments (opensearch-project#5108) Signed-off-by: Krishna Kondaka <krishkdk@amazon.com> fixes related to source config properties change Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> removed future handling for loop based operations Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> additional test cases Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> addressing review comments Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> Jira Service Test coverage Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> jirasourceconfigTest comments Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> introduced RestClient and moved rest template interactions to there. Similar chage on the test cases too Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> backingoff for any kind of exception. Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> restructured constants file Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com> JiraSourceTests Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> JiraItemInfo coverage Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> jira service branch coverage Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> branch coverage jira service Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> move add Items to queue logic into JiraItemInfo Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> fixing regex and adding date time formatter Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> Revert "Jira source" re add changes and fix issues Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> unneeded comment Signed-off-by: Maxwell Brown <mxwelwbr@amazon.com> using issue bean methods to simplify the logic Signed-off-by: Santhosh Gandhe <1909520+san81@users.noreply.github.com>
san81
pushed a commit
to san81/data-prepper
that referenced
this pull request
Nov 3, 2024
…ect#5136) Signed-off-by: Dinu John <86094133+dinujoh@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add model for table column metadata for Global state. The model will be stored in global state and will be available for export and bin log worker to fetch table column metadata such as data type.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.