Updated message table creates cdc fail #77
Comments
|
Fixed. Tested with example application.
|
|
Fixing approach: TABLE_MAP event reflects changes in schema: https://github.com/eventuate-foundation/eventuate-cdc/blob/wip-db-id-gen/eventuate-local-java-cdc-connector-mysql-binlog/src/main/java/io/eventuate/local/mysql/binlog/MySqlBinaryLogClient.java#L221 Ordering is requested from regular datasource: https://github.com/eventuate-foundation/eventuate-cdc/blob/wip-db-id-gen/eventuate-local-java-cdc-connector-mysql-binlog/src/main/java/io/eventuate/local/mysql/binlog/ColumnOrderExtractor.java#L25 But only if ordering is not known: https://github.com/eventuate-foundation/eventuate-cdc/blob/wip-db-id-gen/eventuate-local-java-cdc-connector-mysql-binlog/src/main/java/io/eventuate/local/mysql/binlog/AbstractMySqlBinlogExtractor.java#L23 Commit adds option to refresh ordering if necessary, TABLE_MAP event makes it necessary: 8976771 |
|
How often do TABLE_MAP events occur? |
|
@cer Chris, unfortunately I cannot answer that question right now. |
|
If TABLE_MAP occurs for every insert then refreshing the column order would be quite expensive. Also, it looks like the TABLE_MAP event contains the schema so it might not be necessary to query the database. |
|
@cer I will investigate it too |
|
@cer Chris, According the documentation: Under debug I did not find any information about column names. I was thinking about optimization, and I think that we can refresh column orders only if something in columnMetadata and columnTypes is changed. Please see these screenshots: https://gist.github.com/dartartem/c3d81f5de7252c583b7c77273422cfaf I did not find accurate information, but it seems columnMetadata is information about column attributes: PK, NOT NULL, etc. |
I think this would be a good approach - I don't think knowing the details of |
|
Will do, thank you |
|
During fixing I found another case: mysql extractor caches columns using schema and table as key. After migration, column types/metada were not considered as changed because to message table was assigned new id, since migration drops then recreates message table. In other words data not compared because tables are considered as different. To make everything work I added additional column refresh when new table id is passed via TABLE_MAP event. |
Can you link to the lines of code that implement this. |
|
during working on cdc schema migration tests I found a new failure. It seems if table altered (not recreated by drop/create), mysql connector does not see changes in schema. But columns order can be refreshed via datasource, so cdc expected columns != cdc received columns. And error is exactly same that I found during db id migration. Simple way to reproduce. We have test class that checks message handling: https://github.com/eventuate-foundation/eventuate-cdc/blob/master/eventuate-local-java-cdc-connector-test-util/src/main/java/io/eventuate/local/test/util/AbstractBinlogEntryReaderMessageTableTest.java
Some notes: cdc is starts/stops on each test run. It means that there is no cache in cdc mysql client. I am not sure what to do. I see at least 3 options:
|
|
Stack trace (similar to trace of initial fail): |
Steps 1, and 3 are |
Chris, it is an integration test. So yes, it starts/stops the cdc. |
|
Please replace With an more precise sequence of steps that actually describe what is happening at the database level and the starting and stopping of CDC. e.g. |
|
Actual steps:
execute ALTER TABLE eventuate.message drop destination;
|
|
I realized what is wrong. Integration test uses mocked offset store. My bad. But it shows another issue to think about:
So, before changing anything in schema needs to make sure that cdc finished processing. We have some status service, I will check. |
|
I checked the status service. It shows if processing is finished. It compares offset of last handled event with current binlog position. So, checking the status service should be sufficient. |
|
One more moment, currently cdc cannot reread already processed binlog. |
Won't the CDC reprocess TABLE_MAP events so will know what the schema is? |
Also, given that the CDC is restarted what is the actual problem? What is the sequence of events that it reads that causes it to fail. |
|
Summary: I used an integration test.
further steps happens inside the test:
next step executed via adminer: execute ALTER TABLE eventuate.message drop destination; the last steps executed inside the same test:
Example: initial column order:
message from step 2) is saved with that column order.
On step 3) that message is processed, but because mocked offset store is used, binlog is not saved and cdc reprocesses it after restart on step 6) How data event looks like: [some_id, some_destination, some_headers, some_payload, some_published_flag, some_creation_time] (For convenience lets assume that all indexes starting from 1) TABLE_MAP_EVENT does not contain column names (discussed here) So, cdc needs to check what the columns correspond to the data. It uses sql to query column order as recommended by author of mysql connector. cdc tries to process "published" column (step 10) ), in current schema it's order is 4 |
…d message table migratio test.
…d migration tests for mssql,postgres.
…ved separate test script for message table schema migration.
#77: Updated message table creates cdc fail.
Problem appeared when the eventuate-tram-examples-customers-orders project was tested with migration to database id support.
How was found (only mysql is currently tested):
Error in cdc logs:
Reason recreation of table message breaks expected column order here:
https://github.com/eventuate-foundation/eventuate-cdc/blob/master/eventuate-local-java-cdc-connector-mysql-binlog/src/main/java/io/eventuate/local/mysql/binlog/AbstractMySqlBinlogExtractor.java#L23
And in current case column was "payload" instead of "published".
The text was updated successfully, but these errors were encountered: