PingCAP Special Week 2019 Q4: Dumpling / Mydumper replacement for DM integration and Lightning performance

[kennytm]

Dumpling / Mydumper replacement for DM integration and Lightning performance

Full RFC at #123.

Abstract

We propose introduce a library to replace Mydumper, code named Dumpling, optimized for TiDB Lightning and to be usable as a library/plugin inside DM and TiDB, as well as be an independent program.

Problem statement

Mydumper is a third-party tool to dump MySQL databases into local filesystem as SQL dump. TiDB Lightning relies on output of Mydumper for importing into TiDB, and DM embeds Mydumper to quickly extract data from upstream.

Using Mydumper in the TiDB ecosystem has the following problems:

• as a third-party tool, it does not match our development pace
• Mydumper is licensed in GPLv3, which is not compatible with TiDB (Apache 2.0)

Therefore, we would like to replace Mydumper with our own tool, and develop new features on top of it, like

• create a custom output format to reduce parsing effort and speed up Lightning
• support dumping directly to cloud storage

Success criteria

1. Replacement. Created a Go module with a CLI front-end which supports the Mydumper features required for DM and Lightning.

   • Resulting data are sorted by primary key
   • SQL files are split into size close to the given configuration
   • Single tables can be dumped in parallel, if a primary key or unique btree key exists
   • Consistency: dumping a snapshot instead of live data (either acquire a read lock or ignore new updates)
   • E2E test succeeds
   • Performance matching Mydumper

2. Extension. Implements features which further helps the ecosystem

   • Support an easy-to-decode output format
   • Support directly writing to cloud storage

TODO list

Phase 1: Essentials — 8000 points total

• Create the https://github.com/pingcap/dumpling repository
• Support dumping as SQL files to a local filesystem (6200 points / hard) Support dumping SQL files to a local filesystem  dumpling#1
  • Ensure the program has sufficient logging and metrics
  • Unit test coverage ≥ 75%
• Implement black-white-list filtering (450 points / easy) Implement black-white-list filtering dumpling#2
• E2E test (MySQL/TiDB → Dumpling → Loader → MySQL/TiDB) (900 points / medium) E2E test dumpling#3
• Benchmark against Mydumper (450 points / easy) Benchmark against mydumper dumpling#4
• Documentation

Phase 2: Features — 2000 points total

• Decide and implement the custom binary output format (600 points / medium) Support a custom binary output format dumpling#5
• Support dumping as CSV (300 points / easy) Support dumping CSV dumpling#6
• Support output compression (50 points / easy) Support output compression dumpling#7
• Support writing to external storage (S3, GCS, …), reusing library from BR (450 points / medium) Support writing to cloud storage (S3, GCS, ...) dumpling#8
• Support --where clause (600 points / medium) Support --where clause dumpling#9

Difficulty

• (Mixed)

Score

• 10000

Mentor(s)

• @kennytm

Recommended skills

• Go language
• Software architecture (structuring and API design)
• Task scheduling strategy for parallel programs

[tisonkun]

closed as passed.