Breakdown the "dbms/src/Storages/Transaction" module

[JaySon-Huang]

Enhancement

The codes under "dbms/src/Storages/Transaction" are some codes we adapt TiFlash with TiDB. After years of development, it become kind of crowded and confusing.

Suggest breakdown "dbms/src/Storages/Transaction” into these modules:

• Some global context for TiFlash management
• Basic structure from TiDB (TableInfo/DatabaseInfo, etc)
• Syncing schema with TiDB
• Decoding data from row to column (Block)
• Raft layer
  • Region info
  • Handling admin command
  • Handling write command
  • Handling apply snapshot/ingest sst
  • Handling learner read

---

1. Suggest creating a dbms/src/TiDB as the module for basic components from TiDB. And
   • Rename TMTContext to TiFlashContext
   • Move TableInfo/DBInfo, etc from TiDB.h to dbms/src/TiDB
   • Move TiFlashContext/ManagedStorages to the dbms/src/TiDB module
   • Add a dbms/src/TiDB/Codec and move the logic of decoding data from row to column there
   • Add a dbms/src/TiDB/Schema and move schema syncing logic there
2. Rename dbms/src/Transaction to dbms/src/KVStore. And maybe we should further break down the codes for different Raft processes under this module.
   • Rename dbms/src/Transaction to dbms/src/KVStore Refactor kvstore #8073
   • Break down KVStore. Refactor kvstore #8073

[JaySon-Huang]

/cc @zanmato1984 @flowbehappy @solotzg @fuzhe1989

[hzh0425]

Hi, can you assign this issue to me, I want to try it @JaySon-Huang

[JaySon-Huang]

/assign @hzh0425

Yes. And suggest beginning with a PR that does a part of the changes, cause it will change lots of files.

[hzh0425]

/assign @hzh0425

Yes. And suggest beginning with a PR that does a part of the changes, cause it will change lots of files.

Ok, thanks, I will try it

[flowbehappy]

@hzh0425
Good to catch a volunteer!
And instead of submitting a PR, I suggest you can start by making an RFC that includes the basic design and the main parts of the code you would like to modify. We will be very happy to discuss with you and then come up with a great plan.

[detongz]

Hi, I think I can help with this refactor.

I am a user of tidb, sync data to another database with ticdc is slow which costs 1s at least, so I came up with an idea to change tiflash source code and impliment our own data storage.

Will this refactor support 5.4.0? We are using version 5.4.0 in production.

[JaySon-Huang]

Hi @mischaZhang, it would be great if you could help with this refactor!

Will this refactor support 5.4.0? We are using version 5.4.0 in production.

Sorry that we only pick bug fixes to the later patch version of 5.4.x, so this refactor won't be backported to release-5.4 on our repo.

[detongz]

Hi @mischaZhang, it would be great if you could help with this refactor!

Will this refactor support 5.4.0? We are using version 5.4.0 in production.

Sorry that we only pick bug fixes to the later patch version of 5.4.x, so this refactor won't be backported to release-5.4 on our repo.

I may be help with this refactor on September, please assign me some not so complicated tasks to begin with.

And I really would like to hear your thoughts on my "use tiflash as data sync" idea, thank you for replying :D

[JaySon-Huang]

I may be help with this refactor on September, please assign me some not so complicated tasks to begin with.

Maybe you can begin with the task "Move TableInfo/DBInfo, etc from TiDB.h to dbms/src/TiDB". There are some closed PRs that you can take a look at: #4976, #4915.

[JaySon-Huang]

And I really would like to hear your thoughts on my "use tiflash as data sync" idea

Maybe you can describe more about the design of how do you reuse the TiFlash code for syncing data to another database?

Based on my knowledge, handling the table schema change in TiFlash is not as easy as using TiCDC

[detongz]

And I really would like to hear your thoughts on my "use tiflash as data sync" idea

Maybe you can describe more about the design of how do you reuse the TiFlash code for syncing data to another database?

Based on my knowledge, handling the table schema change in TiFlash is not as easy as using TiCDC

I watched the DDL introducion live, good news is that we don't really change schemas in months.

I want to capture row change events, these data are captured in doLearnerRead. To make data syncronized faster than TiCDC, maybe sending some my-system-related rpc calls before Tiflash flush data into delta store to sync my own data.

[JaySon-Huang]

/assign @mischaZhang

[CalvinNeo]

It is not fully finished