Add TransCompression Feature

[asfimport]

When ZSTD becomes more popular, there is a need to translate existing data to ZSTD compressed which can achieve a higher compression ratio. It would be useful if we can have a tool to convert a Parquet file directly by just decompressing/compressing each page without decoding/encoding or assembling the record because it is much faster. The initial result shows it is ~5 times faster.

Reporter: Xinli Shang / @shangxinli
Assignee: Xinli Shang / @shangxinli

Subtasks:

• Add to Parquet-tools
• Add to parquet-cli
• Add bloom filter support

Related issues:

• Mark Parquet-1872 with not support bloom filter yet (Blocked)

PRs and other links:

• PR-796

_{Note: This issue was originally created as PARQUET-1872. Please see the migration documentation for further details.}

[asfimport]

Gabor Szadovszky / @gszadovszky:
[~shangx@uber.com], I don't know why the PR was not linked here automatically. Please, add it manually to have the reference.
I don't get the sub-tasks. In the PR #796 header you reference this jira while you already resolve the parquet-tools related sub-task in it. I think, adding this functionality to parquet-cli shouldn't be a big deal to separate to another task. (I would suggest implementing the functionality at one place and invoke it from parquet-tools and parquet-cli.
What is the bloom filter support is about? I am not sure about the bloom filters but offset indexes surely have to be updated as the page offsets will change. Without it the feature is incorrect so I would not merge a PR to master without implementing it.

[asfimport]

Xinli Shang / @shangxinli:
@gszadovszky Thanks for the reply! I just manually linked the PR.

For the subtask, I was thinking to have a review & changes first with parquet-tools then I can add it to parquet-cli instead of changing both at the same time. But that is also fine for me to have the two places changes at the same PR. I just add to parquet-cli in the newest PR.

For Column and OffsetIndex, they are taken care of in my PR. I also added tests for both ColumnIndex and OffsetIndex validation.

For bloom filter, I will work on the subtask when this PR is done. That would require to copy over the existing bloom filters to the new files.

Xinli

[asfimport]

Gabor Szadovszky / @gszadovszky:
[~shangx@uber.com], if I understand correctly this PR does not handle bloom filters meaning it won't create any bloom filter in the new file. If that's true, I'm fine with implementing the bloom filter support separately because it means that this PR does not hurt correctness.

[asfimport]

Xinli Shang / @shangxinli:
That is correct understanding @gszadovszky.

[asfimport]

Gabor Szadovszky / @gszadovszky:
[~shangx@uber.com], based on the git logs this feature is more or less complete. Could you please resolve the jiras accordingly?
Also, bloom filter support is missing from the feature. Do you have bandwidth to implement it for 1.12.0? If not do you think it would be a good idea to show in the command that bloom filters are not supported (yet)?

[asfimport]

Xinli Shang / @shangxinli:
Thanks @gszadovszky for working on this! I just created the PR to add comments in the command. Once you review it and we merge, I will resolve this Jira.