Skip to content

Core: Add support for encryption.kms-type with aws/azure/gcp #15272

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
kevinjqliu merged 2 commits into from
Feb 15, 2026
+44 −5
Merged

Core: Add support for encryption.kms-type with aws/azure/gcp #15272

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f82ff68
Core: Add support for encryption.kms-type with aws/azure/gcp
ebyhr Feb 9, 2026
415603a
fixup! Core: Add support for encryption.kms-type with aws/azure/gcp
ebyhr Feb 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions core/src/main/java/org/apache/iceberg/CatalogProperties.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,5 +162,15 @@ private CatalogProperties() {}
public static final long AUTH_SESSION_TIMEOUT_MS_DEFAULT = TimeUnit.HOURS.toMillis(1);

public static final String ENCRYPTION_KMS_TYPE = "encryption.kms-type";
public static final String ENCRYPTION_KMS_TYPE_AWS = "aws";
public static final String ENCRYPTION_KMS_TYPE_AZURE = "azure";
public static final String ENCRYPTION_KMS_TYPE_GCP = "gcp";

public static final String ENCRYPTION_KMS_IMPL = "encryption.kms-impl";
public static final String ENCRYPTION_KMS_IMPL_AWS =
"org.apache.iceberg.aws.AwsKeyManagementClient";
public static final String ENCRYPTION_KMS_IMPL_AZURE =
"org.apache.iceberg.azure.keymanagement.AzureKeyManagementClient";
public static final String ENCRYPTION_KMS_IMPL_GCP =
"org.apache.iceberg.gcp.GcpKeyManagementClient";
}
15 changes: 13 additions & 2 deletions core/src/main/java/org/apache/iceberg/encryption/EncryptionUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
import java.nio.ByteBuffer;
import java.nio.charset.StandardCharsets;
import java.util.List;
import java.util.Locale;
import java.util.Map;
import java.util.Set;
import org.apache.iceberg.CatalogProperties;
Expand Down Expand Up @@ -53,8 +54,18 @@ public static KeyManagementClient createKmsClient(Map<String, String> catalogPro
kmsType,
kmsImpl);

// TODO: Add KMS implementations
Preconditions.checkArgument(kmsType == null, "Unsupported KMS type: %s", kmsType);
if (kmsType != null) {
kmsImpl =
switch (kmsType.toLowerCase(Locale.ROOT)) {
case CatalogProperties.ENCRYPTION_KMS_TYPE_AWS ->
CatalogProperties.ENCRYPTION_KMS_IMPL_AWS;
case CatalogProperties.ENCRYPTION_KMS_TYPE_AZURE ->
CatalogProperties.ENCRYPTION_KMS_IMPL_AZURE;
case CatalogProperties.ENCRYPTION_KMS_TYPE_GCP ->
CatalogProperties.ENCRYPTION_KMS_IMPL_GCP;
default -> throw new IllegalStateException("Unsupported KMS type: " + kmsType);
};
Comment on lines +58 to +67
Copy link
Contributor

@singhpk234 singhpk234 Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have we give a thought about tables which span across multiple cloud providers ? do we need something like ResolvingFileIO eq ?

Copy link
Contributor

@ggershinsky ggershinsky Feb 9, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, we have three single-cloud implementations, built-in the Iceberg code. When a patch for a new KMS client (for a widely used platform) is sent and merged in Iceberg, a new type would be added here.
A multi-cloud KMS client sounds better fitted for the custom encryption.kms-impl parameter; but if there is a popular usecase, then no reason not to have it as a new type.

Copy link
Contributor

@ggershinsky ggershinsky Feb 9, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hopefully the github glitch is resolved; sorry if this appears twice:
An updated comment wrt the ResolvingFileIO. Looks like it infers the type from the file location prefix (like "s3" and others in the SCHEME_TO_FILE_IO map). This is less applicable to encryption keys.

Copy link
Contributor

@nandorKollar nandorKollar Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it make sense to make this pluggable? I've seen requests related to for example Hashicorp Vault (#14451), a pluggable key manager could be beneficial to integrate any 3rd party key management solution into Iceberg, as long as there's a working client implementation for it.

Copy link
Contributor

@ggershinsky ggershinsky Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other (encryption.kms-impl) parameter is designed for a pluggable support of the 3rd party KMS systems.
We need it since there is a lot of KMS options out there, basically an undefined number.
The encryption.kms-type parameter is only for a few highly popular KMS platforms that Iceberg plans to actively support, inc security leak fixes, version updates, bug fixes etc.

Copy link
Contributor

@nandorKollar nandorKollar Feb 10, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Gidon for the clarification!

}

KeyManagementClient kmsClient;
DynConstructors.Ctor<KeyManagementClient> ctor;
Expand Down
17 changes: 17 additions & 0 deletions core/src/test/java/org/apache/iceberg/encryption/TestEncryptionUtil.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
package org.apache.iceberg.encryption;

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;

import java.io.ByteArrayOutputStream;
import java.io.File;
Expand All @@ -27,6 +28,7 @@
import java.lang.reflect.InvocationTargetException;
import java.util.Map;
import org.apache.iceberg.CatalogProperties;
import org.apache.iceberg.relocated.com.google.common.collect.ImmutableMap;
import org.junit.jupiter.api.Test;

public class TestEncryptionUtil {
Expand All @@ -52,6 +54,21 @@ public void testClassLoader()
assertThat(kmsClientObj.getClass().getClassLoader()).isSameAs(customClassLoader);
}

@Test
public void testInvalidTypeAndImpl() {
assertThatThrownBy(
() ->
EncryptionUtil.createKmsClient(
ImmutableMap.of(
CatalogProperties.ENCRYPTION_KMS_TYPE,
CatalogProperties.ENCRYPTION_KMS_TYPE_AWS,
CatalogProperties.ENCRYPTION_KMS_IMPL,
CatalogProperties.ENCRYPTION_KMS_IMPL_AWS)))
.isInstanceOf(IllegalArgumentException.class)
.hasMessage(
"Cannot set both KMS type (aws) and KMS impl (org.apache.iceberg.aws.AwsKeyManagementClient)");
}

static class UnitTestCustomClassLoader extends ClassLoader {

@Override
Expand Down
4 changes: 2 additions & 2 deletions docs/docs/encryption.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ Currently, encryption is supported in the Hive and REST catalogs for tables with

Two parameters are required to activate encryption of a table:

1. Catalog property `encryption.kms-impl`, that specifies the class path for a client of a KMS ("key management service").
1. Catalog property that specifies the KMS ("key management service"). It can be either `encryption.kms-type` for pre-defined KMS clients (`aws`, `azure` or `gcp`) or `encryption.kms-impl` with the client class path for custom KMS clients.
2. Table property `encryption.key-id`, that specifies the ID of a master key used to encrypt and decrypt the table. Master keys are stored and managed in the KMS.

For more details on table encryption, see the "Appendix: Internals Overview" [subsection](#appendix-internals-overview).
Expand All @@ -42,7 +42,7 @@ spark-sql --packages org.apache.iceberg:iceberg-spark-runtime-{{ sparkVersionMaj
--conf spark.sql.catalog.spark_catalog.type=hive \
--conf spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog \
--conf spark.sql.catalog.local.type=hive \
--conf spark.sql.catalog.local.encryption.kms-impl=org.apache.iceberg.aws.AwsKeyManagementClient
--conf spark.sql.catalog.local.encryption.kms-type=aws
```

```sql
Expand Down
3 changes: 2 additions & 1 deletion hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,7 +126,8 @@ public void initialize(String inputName, Map<String, String> properties) {
? new HadoopFileIO(conf)
: CatalogUtil.loadFileIO(fileIOImpl, properties, conf);

if (catalogProperties.containsKey(CatalogProperties.ENCRYPTION_KMS_IMPL)) {
if (catalogProperties.containsKey(CatalogProperties.ENCRYPTION_KMS_TYPE)
|| catalogProperties.containsKey(CatalogProperties.ENCRYPTION_KMS_IMPL)) {
this.keyManagementClient = EncryptionUtil.createKmsClient(properties);
}

Expand Down