Manifest list encryption #7770

ggershinsky · 2023-06-05T06:09:39Z

No description provided.

ggershinsky · 2024-03-26T05:38:55Z

api/src/main/java/org/apache/iceberg/Snapshot.java

@@ -162,6 +162,15 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO io) {
   */
  String manifestListLocation();

+  /**
+   * Return the size of this snapshot's manifest list. For encrypted tables, a verified plaintext


fix comment

core/src/main/java/org/apache/iceberg/BaseSnapshot.java

RussellSpitzer · 2024-03-28T16:03:42Z

api/src/main/java/org/apache/iceberg/Snapshot.java

@@ -162,6 +162,25 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO io) {
   */
  String manifestListLocation();

+  /**
+   * Return the size of this snapshot's manifest list file. Must be a verified value, taken from a


Confused here, we have a default of -1 as well set in base Snapshot which seemed to also be allowed as an "unset". Should we mention that here or is it always required?

We can define this field to be required only for encrypted tables. It will be not set in the snapshot file for unencrypted tables - where this method can return 0 (or -1, I'll make it consistent across all implementation classes).

RussellSpitzer · 2024-03-28T16:08:06Z

core/src/main/java/org/apache/iceberg/BaseSnapshot.java

+      if (manifestListKeyMetadata != null) { // encrypted manifest list file
+        Preconditions.checkArgument(
+            fileIO instanceof EncryptingFileIO,
+            "No encryption in FileIO class " + fileIO.getClass());


Cannot read manifest list (%s) because it is encrypted but the configured FileIO (%s) does not implement EncryptingFileIO)

RussellSpitzer · 2024-03-28T16:09:21Z

core/src/main/java/org/apache/iceberg/BaseSnapshot.java

+        EncryptingFileIO encryptingFileIO = (EncryptingFileIO) fileIO;
+        Preconditions.checkArgument(
+            encryptingFileIO.encryptionManager() instanceof StandardEncryptionManager,
+            "Encryption manager for encrypted manifest list files can currently only be an instance of "


Cannot decrypt manifest list (%s) because the encryption manager (%s) does not implement StandardEncryptionManager

RussellSpitzer · 2024-03-28T16:23:26Z

core/src/main/java/org/apache/iceberg/SnapshotParser.java

+      generator.writeStringField(MANIFEST_LIST_KEY_METADATA, snapshot.manifestListKeyMetadata());
+    }
+
+    // TODO discuss: do we need to sign the size value? Or sign the whole snapshot?


How would this attack work? Wouldn't the user also need the key to encrypt the replacement files? I thought we were storing the metadata.json key in the catalog so an attacker could replace everything but still not be able to trick a client using the catalog.

This is the question. Some thoughts on the scenarios and protection options:

currently, we don't have a metadata.json key. We have only a key for snapshot's manifest list file. Besides using it for encrypting the manifest list file, we can also use this key for signing snapshot's sensitive parts like the manifest list size field. Or for signing the whole metadata.json file (should be possible with some effort) - then we also protect the integrity of e.g. the table properties (like the table key id).

snapshot (metadata.json file) doesn't keep secret values, so encrypting it might not be required. The signatures, mentioned above, would be kept in added snapshot fields - sufficient for detecting the file modification attacks.

these protection techniques are not required with the REST catalog - because we trust the catalog service (we don't trust the storage service). Since the whole snapshot is stored in the REST catalog, we don't need to sign anything.

the manifest list key is not stored in the catalog. Instead, it is wrapped in a KMS with the table master key, and stored in the snapshot MANIFEST_LIST_KEY_METADATA field. Only the KMS-authorized (for the table key) users/processes will be able to get the manifest list key.

In catalogs other than the REST, the signatures provide a partial protection - because the metadata.json is kept in the untrusted storage. With the signatures, it can't be modified. But the whole folder can be replaced (e.g. a replay attack - where all table files are removed, and replaced with files of an older version of the table). To prevent this attack in non-REST catalogs, we will have to update the catalog per each table snapshot (setting eg the latest table version/sequence number, or a random AAD prefix)

RussellSpitzer · 2024-03-28T16:24:44Z

core/src/main/java/org/apache/iceberg/SnapshotProducer.java

+        && encryptedManifestList.keyMetadata().buffer() != null) {
+      Preconditions.checkArgument(
+          encryptionManager instanceof StandardEncryptionManager,
+          "Encryption manager for encrypted manifest list files can currently only be an instance of "


similar comment as above. "Cannot X because Y"

core/src/test/java/org/apache/iceberg/TestManifestEncryption.java

RussellSpitzer · 2024-03-28T16:30:08Z

core/src/test/java/org/apache/iceberg/TestManifestListEncryption.java

+  private static final long EXISTING_ROWS = 857273L;
+  private static final int DELETED_FILES = 1;
+  private static final long DELETED_ROWS = 22910L;
+  private static final List<ManifestFile.PartitionFieldSummary> PARTITION_SUMMARIES =


We probably need a test example which has a non-empty list of partition field summaries

RussellSpitzer · 2024-03-28T16:31:23Z

core/src/test/java/org/apache/iceberg/TestManifestListEncryption.java

+
+  @Test
+  public void testV2Write() throws IOException {
+    ManifestFile manifest = writeAndReadManifestList();


nit: writeAndReadEncryptedManifestList

RussellSpitzer · 2024-03-28T16:33:32Z

core/src/test/java/org/apache/iceberg/TestManifestListEncryption.java

+  public void testV2Write() throws IOException {
+    ManifestFile manifest = writeAndReadManifestList();
+
+    // all v2 fields should be read correctly


Assert J has some helper for this, Not sure if it is correct

assertThat(actual).usingRecursiveComparison().isEqualTo(expected);

This unitest/code is mostly copy/paste from https://github.com/apache/iceberg/blob/main/core/src/test/java/org/apache/iceberg/TestManifestListVersions.java#L137

RussellSpitzer · 2024-03-28T21:34:00Z

core/src/main/java/org/apache/iceberg/SnapshotParser.java

+
+    // TODO discuss: do we need to sign the size value? Or sign the whole snapshot?
+    // Or rely on REST catalog? - the only option that prevents "full folder replacement" attack.
+    if (snapshot.manifestListSize() >= 0) {


another small question here, we essentially are doing a transform here

manifestlists sizes < 0 become 0.

Also nit: we are also ignoring 0's that get passed through although we will read this as 0 if it is missing.

Just wondering what the intent here is. I think it may be better to have a defined missing value? Not sure

Yep, part of this thread #7770 (comment)

RussellSpitzer

Is it important that we store the manifest list size? Won't the encryption be enough to prove the file is the right one?

ggershinsky · 2024-04-01T08:59:02Z

Yep, this is due to https://github.com/apache/iceberg/blob/main/format/gcm-stream-spec.md#file-length . There are options for table modification attacks if this field is not (safely) stored.

RussellSpitzer · 2024-04-08T17:16:47Z

core/src/main/java/org/apache/iceberg/BaseSnapshot.java

    this.v1ManifestLocations = v1ManifestLocations;
+    this.manifestListKeyMetadata = null;


Same as comment above (group manifest vars)

rdblue · 2024-04-11T15:51:22Z

api/src/main/java/org/apache/iceberg/Snapshot.java

+   * In encrypted tables, return the size of this snapshot's manifest list file. Must be a verified
+   * value, taken from a trusted source. In unencrypted tables, can return 0.
+   */
+  default long manifestListSize() {


Rather than adding new methods for each piece of new information we want to pass, what about adding a ManifestList object that contains the location, size, and key metadata?

anuragmantri · 2024-06-11T11:41:22Z

api/src/main/java/org/apache/iceberg/Snapshot.java

@@ -162,6 +162,16 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO io) {
   */
  String manifestListLocation();


Is String manifestListLocation() same as what is stored in ManifestListFile? If so, do we still need it as a separate field?

@rdblue @RussellSpitzer what do you think?

There are dozens of calls to this method today (tests etc), so we'll likely need to keep it for a while.

Yeah, I saw that. We can consolidate in a separate PR.

Yeah, I agree. We need to keep this because it is part of the public API and used in many cases.

anuragmantri · 2024-06-11T11:49:31Z

core/src/main/java/org/apache/iceberg/SnapshotParser.java

+          ByteBuffer.wrap(
+              Base64.getDecoder().decode(snapshot.manifestListFile().wrappedKeyMetadata()));
+
+      NativeEncryptionKeyMetadata wrappedKeyMetadata =


Is this unused?

Good catch. I'll remove these lines.

anuragmantri · 2024-06-11T11:53:03Z

core/src/main/java/org/apache/iceberg/SnapshotParser.java

+      ByteBuffer manifestListKeyMetadata = null;
+      ByteBuffer wrappedManifestListKeyMetadata = null;
+      String wrappedKeyEncryptionKey = null;


Nit: Declare these inside if (node.has(MANIFEST_LIST_KEY_METADATA)) ?

They are used later, in the BaseSnapshot constructor. We can use a different constructor (if no manifest list encryption), but the code won't be more compact.

ggershinsky · 2024-06-11T07:14:43Z

api/src/main/java/org/apache/iceberg/ManifestListFile.java

+    return size;
+  }
+
+  public String wrappedKeyEncryptionKey() {


Add javadoc: "Manifest list keys are encrypted with a table "key encryption key". This function returns a KMS wrap of the key encryption key.

rdblue · 2024-09-16T18:16:18Z

@ggershinsky, I'm not too concerned with the size of the cache. I'm okay with 1 day, but that seems like a long time to have unencrypted key material in memory. I'll defer to your judgement here.

ggershinsky · 2024-09-17T06:59:37Z

Ok. We don't have clear guidelines on key caching in memory (key copies are spread all over the process memory - cache, plug-in KMS client code, an HTTP library in the KMS client code; the Java GC - so there are no guarantees for when an uncached key is deleted from memory, if ever, before the process stops). But I agree a day could be too long. I'll change it to 1 hour - might be a reasonable trade-off between performance requirements (KMS call overhead), and safety requirements (there is a chance a key will be deleted from the memory within a business day).

ggershinsky · 2024-09-27T13:33:18Z

Hi @rdblue , I've built the integration code with the latest version of this patch, works ok. Can we merge this PR?

RussellSpitzer · 2024-10-31T22:01:33Z

@rdblue Did you have any more comments on this one? I can do another pass as well but I'd like to finish this up as well soon

ggershinsky · 2024-11-07T13:41:02Z

@rdblue @RussellSpitzer All current comments should have been addressed in this thread and in the last commit. An additional review round is always welcome, I too would like to complete this feature.

smaheshwar-pltr · 2024-11-15T17:26:57Z

core/src/main/java/org/apache/iceberg/encryption/EncryptionUtil.java

+   * @param key unwrapped snapshot key bytes
+   * @param snapshotId ID of the table snapshot
+   * @param keyMetadata unencrypted EncryptionKeyMetadata
+   * @return a Pair of the key ID used to encrypt and the encrypted key metadata


Suggested change

* @return a Pair of the key ID used to encrypt and the encrypted key metadata

* @return the encrypted key metadata

(nit)

caushie-akamai · 2024-12-15T20:51:55Z

Hi! Are there any plans to have this feature merged in 1.8.0? This would be extremely helpful

rshkv · 2024-12-17T16:58:49Z

+1, we have some compliance requirements that prevent us from adopting Iceberg without client-side encryption.

ggershinsky · 2024-12-18T11:52:28Z

^^ cc @rdblue @RussellSpitzer

amogh-jahagirdar

I feel like this is pretty close, had some comments on deprecation comments that needed to be updated. It seems like the main point that may be lingering is around the decryptKeyMetadata(EncryptionManager) interface but I don't think there's a good alternative to that cc @rdblue

amogh-jahagirdar · 2025-01-17T15:37:29Z

core/src/main/java/org/apache/iceberg/encryption/StandardEncryptionManager.java

+   * @deprecated will be removed in 1.8.0; use {@link #unwrapKey(String)}} instead.
+   */


Note, if this PR doesn't have any more comments and we think we can get it in for 1.8 we should update this to be "will be removed in 1.9.0"

Per the slack discussion, the target release is not yet finalized.

amogh-jahagirdar · 2025-01-17T15:37:35Z

core/src/main/java/org/apache/iceberg/encryption/StandardEncryptionManager.java

@@ -81,22 +136,75 @@ private SecureRandom workerRNG() {
    return lazyRNG;
  }

+  /**
+   * @deprecated will be removed in 1.8.0; use {@link #currentSnapshotKeyId()} instead.


same as below

core/src/test/java/org/apache/iceberg/TestManifestEncryption.java

amogh-jahagirdar · 2025-01-17T15:51:40Z

core/src/main/java/org/apache/iceberg/SnapshotProducer.java

+      if (writer != null) {
+        try {
+          writer.close(); // must close before getting file length
+        } catch (IOException e) {
+          throw new RuntimeIOException(e, "Failed to close manifest list file writer");


Did this part need to change?

Oh I see we need to explicitly flush so that by the time "toManifestListFile" is called we are guaranteed to have a length to work with

amogh-jahagirdar · 2025-01-17T16:20:21Z

core/src/main/java/org/apache/iceberg/BaseManifestListFile.java

+  }
+
+  @Override
+  public ByteBuffer decryptKeyMetadata(EncryptionManager em) {


Yeah I'm not sure if there's an alternative to passing the EncryptionManager at least with the current design of EncryptingFileIO

amogh-jahagirdar · 2025-01-17T16:24:13Z

I'd also suggest to make sure we rebase because it's been a while since this was open, and we should see CI pass with the rebased changes before merging.

ggershinsky · 2025-01-22T07:48:55Z

I'd also suggest to make sure we rebase because it's been a while since this was open, and we should see CI pass with the rebased changes before merging.

Sure, I'll rebase this after Iceberg 1.8 is out (per the slack discussion)

gumartinm · 2025-02-13T23:51:54Z

If you need help with this pull request @ggershinsky, perhaps I could help you.

update snapshot producer update the patch clean up fix for previous patch address review comments move key wrapping to metadata encryption encrypt manifest list key metadata new aad util null key needs no encryption comment; clearer method/var names use key encryption key for manifest list keys add encryption util changes update EncryptionTestHelpers handle api change remove unused lines revert revapi.yml KEK cache unitest update rename var address review comments fix timeout default change writer kek timeout default Updates from review. cache unwrapped keys

Co-authored-by: Ryan Blue <rdblue@gmail.com>

ggershinsky · 2025-02-25T09:08:56Z

This PR is rebased and synced with the spec patch.

github-actions bot added API core labels Jun 5, 2023

ggershinsky marked this pull request as draft June 5, 2023 06:10

ggershinsky force-pushed the manifest-list-encryption branch from aba7650 to 3162f9e Compare March 26, 2024 05:37

ggershinsky commented Mar 26, 2024

View reviewed changes

ggershinsky force-pushed the manifest-list-encryption branch from 3162f9e to a2d7b10 Compare March 27, 2024 06:38

ggershinsky marked this pull request as ready for review March 27, 2024 07:07

RussellSpitzer reviewed Mar 28, 2024

View reviewed changes

core/src/main/java/org/apache/iceberg/BaseSnapshot.java Outdated Show resolved Hide resolved

RussellSpitzer reviewed Mar 28, 2024

View reviewed changes

core/src/test/java/org/apache/iceberg/TestManifestEncryption.java Show resolved Hide resolved

RussellSpitzer reviewed Mar 28, 2024

View reviewed changes

RussellSpitzer reviewed Mar 29, 2024

View reviewed changes

RussellSpitzer reviewed Apr 8, 2024

View reviewed changes

rdblue reviewed Apr 11, 2024

View reviewed changes

ggershinsky force-pushed the manifest-list-encryption branch from 6c7f5a1 to 5486f6f Compare May 26, 2024 06:41

ggershinsky closed this May 28, 2024

ggershinsky reopened this May 28, 2024

anuragmantri reviewed Jun 11, 2024

View reviewed changes

ggershinsky commented Jun 11, 2024

View reviewed changes

smaheshwar-pltr reviewed Nov 15, 2024

View reviewed changes

RussellSpitzer added this to the Iceberg 1.8.0 milestone Jan 15, 2025

amogh-jahagirdar reviewed Jan 17, 2025

View reviewed changes

rdblue removed this from the Iceberg 1.8.0 milestone Jan 22, 2025

ggershinsky and others added 8 commits February 24, 2025 08:12

compile and ci fixes

c94d58a

spotless apply

51e8ccd

re-apply unwrapped cache patch

3ff8f83

Co-authored-by: Ryan Blue <rdblue@gmail.com>

caching limit

a30744a

address review comments

517378c

spotless apply

96feb5c

sync with spec changes

39363fa

ggershinsky force-pushed the manifest-list-encryption branch from dcb5e4d to 39363fa Compare February 24, 2025 06:16

ggershinsky added 5 commits February 24, 2025 08:23

add missing method

a0de0ee

rm empty line

55ea9bb

pass encrypted output

139ca9e

a better unitest

79c2e80

rm type, rename class

661ee2b

		this.v1ManifestLocations = v1ManifestLocations;
		this.manifestListKeyMetadata = null;

		@@ -162,6 +162,16 @@ default Iterable<DeleteFile> removedDeleteFiles(FileIO io) {
		*/
		String manifestListLocation();

	* @return a Pair of the key ID used to encrypt and the encrypted key metadata
	* @return the encrypted key metadata

		* @deprecated will be removed in 1.8.0; use {@link #unwrapKey(String)}} instead.
		*/

Manifest list encryption #7770

Are you sure you want to change the base?

Manifest list encryption #7770

Conversation

ggershinsky commented Jun 5, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer Mar 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussellSpitzer left a comment

Choose a reason for hiding this comment

ggershinsky commented Apr 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rdblue commented Sep 16, 2024

ggershinsky commented Sep 17, 2024

ggershinsky commented Sep 27, 2024

RussellSpitzer commented Oct 31, 2024

ggershinsky commented Nov 7, 2024

smaheshwar-pltr Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

caushie-akamai commented Dec 15, 2024

rshkv commented Dec 17, 2024

ggershinsky commented Dec 18, 2024

amogh-jahagirdar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amogh-jahagirdar commented Jan 17, 2025

ggershinsky commented Jan 22, 2025

gumartinm commented Feb 13, 2025

ggershinsky commented Feb 25, 2025

RussellSpitzer Mar 28, 2024 •

edited

Loading

smaheshwar-pltr Nov 15, 2024 •

edited

Loading