lightning: support import compressed csv/sql files to save cost

[lichunzhu]

Feature Request

Is your feature request related to a problem? Please describe:

Dumpling has already supported export compressed csv files in pingcap/dumpling#7.
We want lightning to support this format's compressed files.

Describe the feature you'd like:

make lightning support import compressed csv/sql files to save cost

Task

• Compress Reader supports reading lzo/snappy/gzip type compressed files
• Compress Writer supports writing lzo/snappy/gzip type compressed files
• Dumpling compress adds lzo/snappy
• Lightning file router default regular support for file name parsing compression
• Lightning reads files with compression reader
• Lightning disables splitting of large files in compressed CSV files
• Lightning support breakpoint read for compressed files
• Additional changes to the Lightning file router documentation

[sleepymole]

Is this change compatible with checkpoint?

[lichunzhu]

Is this change compatible with checkpoint?

No. The new change can't use SetPos. This function is used in SplitLargeFile and checkpoint. We can ban SplitLargeFile for compressed files. For checkpoint I'm thinking of skip the leading bytes until reader's pos exceeds chunk.Offset. WDYT?

[sleepymole]

For checkpoint I'm thinking of skip the leading bytes until reader's pos exceeds chunk.Offset.

It does work. However, can we look into whether we can seek directly on the compressed file? One idea is to save the state of compress reader to checkpoint, so we continue reading from last pos after lightning restarts.

[lichunzhu]

For checkpoint I'm thinking of skip the leading bytes until reader's pos exceeds chunk.Offset.

It does work. However, can we look into whether we can seek directly on the compressed file? One idea is to save the state of compress reader to checkpoint, so we continue reading from last pos after lightning restarts.

I'm afraid that not every compress reader supports reading start from an offset. I will do a survey to check whether it's okay.