Bucket replication with KMS is broken in 7.0.0 #2390
Comments
|
Just wanted to confirm that after downgrading all operators and tenants to version 6.0.4, and having made no other changes, replication is working again, and |
|
@dhess This behavior has been introduced in However, a side effect of this bug fix is that now all sites would be required to have access to all cryptographic keys used on any site. We understand that this is not always desirable. We can provide an option to disable the replication of SSE-KMS key IDs. This means, MinIO still replicates an SSE-KMS header indicating to the target site that the object should be encrypted using SSE-KMS but without specifying the KMS key ID. Then the site will apply either the site-level default KMS key or a key specified in a bucket config, if any. |
|
@aead Thanks for the reply. We do indeed have site-level default KMS enabled for all buckets at both Site A and Site B. Each site uses a different default key, and neither site has access to the other's key. We would like to keep it that way. Let me make sure I understand MinIO's behavior prior to minio/minio#20542, and the proposed fix's behavior:
|
Only if you have a site-level or bucket-level encryption config on site B. For example, a bucket encryption config on all buckets on site B. Before minio/minio#20542 , site A would replicate the object without requesting encryption on the target. Hence, the target site was free to choose how to store the object. If the target site stored objects encrypted then it also stored replicated objects encrypted.
The behavior (in your case) will be the same as before. However, there is a difference. From now on, site A will tell site B that it should store the object encrypted using SSE-KMS. Site B can choose the KMS key but it has to store the object encrypted. |
|
@aead Thanks. That will work for us, thank you! |
|
@aead Should we add this changed behaviour to the release notes? |
Hi!
We believe that version 7.0.0 of the operator has broken bucket replication, specifically something related to
kesand KMS.Our setup:
Site A
cluster-aruns the MinIO operator and 2 tenants,lake-0andnvme-0.Site C
cluster-cruns the MinIO operator and 1 tenant,lake-1.All 3 tenants are encrypted using
kes, and share a common Vault service for KMS.Site A tenants
lake-0andnvme-0each have multiple buckets that are configured to replicate (via bucket replication) to Site C tenantlake-1. This has been working very well for about 8 months.On Jan 30, we upgraded the MinIO operators and tenants in both
cluster-aandcluster-cfrom version 6.0.4 to to version 7.0.0, via the operator and tenant Helm charts.Since then, we noticed that new objects in replicated buckets in Site A tenants
lake-0andnvme-0were no longer replicating to Site C tenantlake-1. All of the objects in question are marked asFAILED, and attempts to retry them viamc ls,mc stat, etc., do not resolve the issue, nor does restarting the tenants and/or operators, as we've sometimes needed to do in the past to restart the replication process.After some investigation and spelunking through logs, it appears that shortly after we upgraded the operators and tenants to version 7.0.0,
lake-1'skesstarted requesting non-existent keys from the Vault KMS. These bogus requests appear to be caused by replication attempts from thelake-0andnvme-0tenants.Here's an example of a bad request from the logs of
lake-1'skes, sampled from shortly after the upgrade:{ "time": "2025-01-30T23:43:03.415228411Z", "request": { "ip": "10.70.1.176", "path": "/v1/key/generate/nvme-0-key", "identity": "624268fb0afa9a64b0ae8047c54e37024d5808aa3d7110bcca9ccf8c997401cf" }, "response": { "code": 404, "time": 1975831000 } }Here's an example of what Vault sees, taken from the logs of the shared Vault KMS and sampled around the same time as the previous request. (This may not be the precise response to that request, as there are a lot of them, but they all have the same path.)
{ "auth": { "accessor": "hmac-sha256:e0d4beb86b3546cd6d0b9615160631d83864cdfebaca238444b1543087849455", "client_token": "hmac-sha256:4b348f3f093987f1be08ebc0c963e5cdf271426cc5f9f109e59dc1c7e8ee7a07", "display_name": "cluster-c-minio-lake-1-kes-sa", "entity_id": "9bfde683-4e64-0faa-1ad8-2b1aaa105020", "metadata": { "role": "minio-lake-1-kes", "service_account_name": "kes-sa", "service_account_namespace": "minio-lake-1", "service_account_secret_name": "", "service_account_uid": "f365f367-a347-440a-9508-b51406b58e62" }, "policies": [ "default", "minio-lake-1-kes" ], "policy_results": { "allowed": true, "granting_policies": [ { "type": "" }, { "name": "minio-lake-1-kes", "namespace_id": "root", "type": "acl" } ] }, "token_policies": [ "default", "minio-lake-1-kes" ], "token_issue_time": "2025-01-26T18:23:40Z", "token_ttl": 86400, "token_type": "service" }, "request": { "client_id": "9bfde683-4e64-0faa-1ad8-2b1aaa105020", "client_token": "hmac-sha256:b579ca85e3df6c5c406587569210cecd46a959690afbb701595844aad2f5d290", "client_token_accessor": "hmac-sha256:e0d4beb86b3546cd6d0b9615160631d83864cdfebaca238444b1543087849455", "id": "2a873ef9-7f49-9f07-3ea3-a655f1a3b75f", "mount_class": "secret", "mount_point": "minio-lake-1/", "mount_running_version": "v0.20.0+builtin", "mount_type": "kv", "namespace": { "id": "root" }, "operation": "read", "path": "minio-lake-1/data/minio-lake-1/nvme-0-key", "remote_address": "10.70.1.135", "remote_port": 60642 }, "time": "2025-01-30T23:43:03.288757958Z", "type": "request" }Indeed, no such key
minio-lake-1/data/minio-lake-1/nvme-0-keyexists in the Vault KMS's key-value store; onlyminio-lake-1/data/minio-lake-1/lake-1-keyexists there.Here's an example of how this problem manifests in
mc admin trace lake-1:The examples shown above are for
nvme-0bucket replication, but we see similar errors in the logs for thelake-0tenant, the key difference being that, for this tenant,lake-1'skesis requesting/v1/key/generate/lake-0-keyrather than/v1/key/generate/nvme-0-key.According to our logs, prior to the upgrade to 7.0.0,
lake-1'skesprevious only requested the KMS path/v1/key/generate/lake-1-key, and never/v1/key/generate/lake-0-keynor/v1/key/generate/nvme-0-key.Expected Behavior
Bucket replication between 2 tenants, each using KMS, should work.
Current Behavior
kesin the replication destination tenant appears to look up the KMS key for the source bucket, which it does not know or have access to, instead of its own key.Steps to Reproduce (for bugs)
keslogs and see thatkesis requesting an invalid key from the KMS.Context
Our bucket replication is no longer replicating new objects.
Regression
Yes, as of 7.0.0.
Your Environment
minio-operator): 7.0.0uname -a): Talos v1.9.2.The text was updated successfully, but these errors were encountered: