GH-31869: [Python][Parquet] First pass at implementing external key material features in python

[pfparsons]

Rationale for this change

Enables external key material and rotation for individual parquet files in PyArrow. This change does not address any parquet dataset encryption functionality. (though I would be happy to keep going with any work required for other applications of external store including the dataset API once I get some feedback on this PR)

What changes are included in this PR?

This PR enables external key material for parquet encryption from PyArrow:
Optional parquet_file_path and FileSystem paramters to CryptoFactory - mirroring the interface for CryptoFactory in C++

1. Exposes the rotate_master_keys method of CryptoFactory

2. Adds Cython classes for FileKeyMaterialStore, FileSystemKeyMaterialStore, and KeyMaterial - but does not expose these from PyArrow encryption. I included these changes only so that a unit test may verify an external store without leaking the implementation details for the store into the test.

Are these changes tested?

Yes - I've modified an existing test (previously marked pytest.xfail) to do a basic read write test and verify creation of the external key material store.

I would like to get some feedback as to whether this change is welcome and headed in the right direction before going any further.

Are there any user-facing changes?

This change enables users to optionally supply a parquet file path and FileSystem to CryptoFactory methods that provide en/decryption_properties.

• GitHub Issue: [Python][Parquet] Enable usage of external key material and rotation for encryption keys in PyArrow #31869

[github-actions]

Thanks for opening a pull request!

If this is not a minor PR. Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose

Opening GitHub issues ahead of time contributes to the Openness of the Apache Arrow project.

Then could you also rename the pull request title in the following format?

GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}

or

MINOR: [${COMPONENT}] ${SUMMARY}

See also:

• Other pull requests
• Contribution Guidelines - Contributing Overview

[github-actions]

⚠️ GitHub issue #31869 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #31869 has been automatically assigned in GitHub to PR creator.

[github-actions]

⚠️ GitHub issue #31869 has been automatically assigned in GitHub to PR creator.

[pfparsons]

Hi @raulcd would you mind taking a look at this?