[Parquet] Provide only encrypted column stats in plaintext footer

[rok]

As per spec:

In the plaintext footer mode, the optional ColumnMetaData meta_data is set in the ColumnChunk structure for all columns, but is stripped of the statistics for the sensitive (encrypted) columns. These statistics are available for new readers with the column key - they decrypt the encrypted_column_metadata field, described in the section 5.3, and parse it to get statistics and all other column metadata values. The legacy readers are not aware of the encrypted metadata field; they parse the regular (plaintext) field as usual. While they can’t read the data of encrypted columns, they read their metadata to extract the offset and size of encrypted column data, required for column chunk vectorization.

Current writer writes stats into plaintext part of footers too. We would want to write stats into encrypted part of footer only as per the spec.

[adamreeve]

I've looked a bit into this and just want to document my findings.

This behaviour is discussed in the format docs in section 5.3.

• encrypted_column_metadata in the ColumnChunk is the encrypted version of meta_data and should be set when a plaintext footer is used and the column is encrypted, or when a column is encrypted with a different key to the footer.
• When reading with decryption properties provided, the encrypted_column_metadata should be decrypted and used to replace the meta_data. It looks like this should already happen here, although I'm not 100% sure we hit this code path with a plaintext footer.
  • If reading and we have a decryptor but don't have access to a specific column, it looks like we'll be unable to read the file at all as this will raise an error. This might be something to look into separately. Maybe the column metadata should only be decrypted on-demand, or maybe we need to handle when we can't get a key for a particular column. It looks like this is decrypted on-demand in C++ Parquet but I haven't tested this.
• Currently, when a ColumnChunk is created for writing the encrypted metadata is always None, so we need to fix both the case of a plaintext footer and when a column is encrypted with a different key to the footer
• It looks like here is where the logic for writing encrypted column metadata is handled in C++ Parquet
  • When the footer is encrypted and a column is encrypted with another key, the whole meta_data field is removed. I guess this is because readers that support encryption are expected to handle when this is missing if they can't decrypt the column.
  • When a plaintext footer is used, the unencrypted meta_data field is kept, but its statistics and encoding_stats are removed.