Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix TINYINT(1) data type mapping in rds source #5214

Merged
merged 1 commit into from
Dec 6, 2024

Conversation

oeyh
Copy link
Collaborator

@oeyh oeyh commented Nov 25, 2024

Description

TINYINT(1) columns show up as BIT type when we query the column types, which results in error:

2024-11-25T14:06:41,668 [rds-source-binlog-processor-1] ERROR org.opensearch.dataprepper.plugins.source.rds.stream.BinlogEventListener - Failed to process change event of type EXT_WRITE_ROWS
java.lang.IllegalArgumentException: Unsupported value type. The value is of type: class java.lang.Integer
    at org.opensearch.dataprepper.plugins.source.rds.datatype.impl.NumericTypeHandler.handleBit(NumericTypeHandler.java:63) ~[rds-source-2.11.0-SNAPSHOT.jar:?]
    at org.opensearch.dataprepper.plugins.source.rds.datatype.impl.NumericTypeHandler.handleNumericType(NumericTypeHandler.java:38) ~[rds-source-2.11.0-SNAPSHOT.jar:?]
    at org.opensearch.dataprepper.plugins.source.rds.datatype.impl.NumericTypeHandler.handle(NumericTypeHandler.java:25) ~[rds-source-2.11.0-SNAPSHOT.jar:?]
    at org.opensearch.dataprepper.plugins.source.rds.datatype.impl.NumericTypeHandler.handle(NumericTypeHandler.java:12) ~[rds-source-2.11.0-SNAPSHOT.jar:?]
    at org.opensearch.dataprepper.plugins.source.rds.datatype.DataTypeHelper.getDataByColumnType(DataTypeHelper.java:29) ~[rds-source-2.11.0-SNAPSHOT.jar:?]
    at org.opensearch.dataprepper.plugins.source.rds.stream.BinlogEventListener.handleRowChangeEvent(BinlogEventListener.java:373) ~[rds-source-2.11.0-SNAPSHOT.jar:?]
    at org.opensearch.dataprepper.plugins.source.rds.stream.BinlogEventListener.handleInsertEvent(BinlogEventListener.java:269) ~[rds-source-2.11.0-SNAPSHOT.jar:?]
    at org.opensearch.dataprepper.plugins.source.rds.stream.BinlogEventListener.lambda$handleEventAndErrors$1(BinlogEventListener.java:448) ~[rds-source-2.11.0-SNAPSHOT.jar:?]
    at io.micrometer.core.instrument.composite.CompositeTimer.record(CompositeTimer.java:141) ~[micrometer-core-1.13.0.jar:1.13.0]
    at org.opensearch.dataprepper.plugins.source.rds.stream.BinlogEventListener.handleEventAndErrors(BinlogEventListener.java:448) ~[rds-source-2.11.0-SNAPSHOT.jar:?]
    at org.opensearch.dataprepper.plugins.source.rds.stream.BinlogEventListener.lambda$processEvent$0(BinlogEventListener.java:442) ~[rds-source-2.11.0-SNAPSHOT.jar:?]

This PR sets JDBC connector property tinyInt1isBit to false so that the column type returned for TINYINT(1) would be TINYINT instead of BIT.

Reference: https://dev.mysql.com/doc/connector-j/en/connector-j-connp-props-result-sets.html#cj-conn-prop_tinyInt1isBit

Issues Resolved

N/A

Check List

  • New functionality includes testing.
  • New functionality has a documentation issue. Please link to it in this PR.
    • New functionality has javadoc added
  • Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Hai Yan <oeyh@amazon.com>
@dlvenable
Copy link
Member

@oeyh , How does this change the plugin's implementation? What code path does it take when this is configured? What data type is the JDBC driver giving Data Prepper now?

@dinujoh
Copy link
Member

dinujoh commented Dec 2, 2024

Alternatively, we can also handle Number in handleBit method

https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/rds-source/src/main/java/org/opensearch/dataprepper/plugins/source/rds/datatype/impl/NumericTypeHandler.java#L62

if (value instanceof Number) {
    return (Number)value;
}

@oeyh
Copy link
Collaborator Author

oeyh commented Dec 3, 2024

@oeyh , How does this change the plugin's implementation? What code path does it take when this is configured? What data type is the JDBC driver giving Data Prepper now?

@dlvenable It shouldn't change the plugin's implementation. The configuration affects the column type name we get from this line when the MySQL type is TINYINT(1) specifically. The column type is first retrieved when rds source starts and later used in data type mapping (by these handlers).
For MySQL type TINYINT(1), JDBC driver gives Data Prepper now is BIT, after the config change here, it will give Data Prepper TINYINT.

@oeyh oeyh merged commit c47131d into opensearch-project:main Dec 6, 2024
45 of 47 checks passed
sb2k16 pushed a commit to sb2k16/data-prepper that referenced this pull request Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants