[HUDI-9691] Migrate MysqlDebeziumPayload to merge mode #13727
Conversation
nsivabalan
force-pushed
the
HUDI-8401-followups-myssql-debeziumpayload
branch
from
80fa4f1 to
3a7726d
Compare
github-actions
bot
added
size:M
| if (!params.contains(HoodieWriteConfig.PRECOMBINE_FIELD_NAME.key()) && tableConfig.getPreCombineFieldsStr.isPresent) { | ||
| missingWriteConfigs ++= Map(HoodieWriteConfig.PRECOMBINE_FIELD_NAME.key -> tableConfig.getPreCombineFieldsStr.orElse(null)) | ||
| } |
There was a problem hiding this comment.
There was one issue which I was facing. Basically when precombine config is not configured, we add it using table config.
Write config is therefore pointing to older table config but the table config is updated to newer config after upgrade. In order to avoid such a scenario removed addition of write config here.
There was a problem hiding this comment.
With the deprecation of write config in #13718, this should be further streamlined
...i-client-common/src/main/java/org/apache/hudi/table/upgrade/NineToEightDowngradeHandler.java
Outdated
Show resolved
Hide resolved
...tasource/hudi-spark/src/test/scala/org/apache/hudi/functional/RecordLevelIndexTestBase.scala
Outdated
Show resolved
Hide resolved
yihua
left a comment
yihua
left a comment
There was a problem hiding this comment.
LGTM overall, made a few minor revisions. Let's fix test failures.
| } | ||
|
|
||
| @Test | ||
| def testUpgradeDowngradeMySqlDebeziumPayload(): Unit = { |
There was a problem hiding this comment.
nit: we might want to decouple the upgrade and downgrade tests from RecordLevelIndexTestBase, which can be done separately. @lokeshj1703 could you file a JIRA to track this?
There was a problem hiding this comment.
Co-authored-by: Lokesh Jain <ljain@192.168.1.21> Co-authored-by: Lokesh Jain <ljain@192.168.0.234> Co-authored-by: Y Ethan Guo <ethan.guoyihua@gmail.com>
4 participants
Change Logs
Migrating MySqlDebeziumPayload to merge mode.
Pending work:
Adding tests to TestPayloadDeprecationFlow for MySqlDebeziumPayload
Impact
MySqlDebeziumPayload in V9 will be using merge modes w/ multiple ordering fields.
Risk level (write none, low medium or high below)
medium
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change. If not, put "none".
ticket number here and follow the instruction to make
changes to the website.
Contributor's checklist