Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Option to enable disable ILM expiry for site replication #4698

Merged
merged 12 commits into from
Nov 22, 2023
Merged

Option to enable disable ILM expiry for site replication #4698

merged 12 commits into from
Nov 22, 2023

Conversation

shtripat
Copy link
Contributor

@shtripat shtripat commented Sep 28, 2023 •
edited
Loading

Community Contribution License

All community contributions in this pull request are licensed to the project maintainers
under the terms of the [Apache 2 license] (https://www.apache.org/licenses/LICENSE-2.0).
By creating this pull request I represent that I have the right to license the
contributions to the project maintainers under the Apache 2 license.

Description

Added options replicate-ilm-expiry and disable-ilm-expiry-replication for mc admin replicate add and mc admin replicate update commands respectively.

Motivation and Context

How to test this PR?

Follow below detailed steps for verifying the different scenarios

Step-1: Create three MinIO sites sat m1, m2 and m3

Step-2: Setup site replication between the first two sites now
mc admin replicate add m1 m2 --replicate-ilm-expiry

Step-3: Create a bucket for first site (test-bucket)
mc mb m1/test-bucket

Step-4: Setup warm tier to m3 for site m1

$ ./mc mb m3/test-bucket
$ ./mc ilm tier add minio m1 WARM-TIER --endpoint http://localhost:9020 --access-key minioadmin --secret-key minioadmin --bucket test-bucket

Step-5: Check details of the replication

$ mc admin replicate info m1
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | true                
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | true     

$ mc admin replicate info m2 
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | true                
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | true

Step-6: Add ILM rules to the first site's bucket test-bucket

$ ./mc ilm add m1/test-bucket --transition-days 0 --transition-tier WARM-TIER --transition-days 0 --noncurrent-expire-days 2 --expire-days 3 --prefix "shu" --tags "tag1=val1&tag2=val2&tag3=val3"

Step-7: Check ILM rules list for sites

$ ./mc ilm rule list m1/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                                                         │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬──────────────┬───────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO TIER │ TIER      │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼──────────────┼───────────┤
│ cl69k9db4dtnd3f3odm0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            0 │ WARM-TIER │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴──────────────┴───────────┘
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69k9db4dtnd3f3odm0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              3 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69k9db4dtnd3f3odm0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule list m2/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69k9db4dtnd3f3odm0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              3 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69k9db4dtnd3f3odm0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

Step-8: Check replication status details as below

$ ./mc admin replicate status m1
Bucket replication status:
●  1/1 Buckets in sync

Policy replication status:
●  5/5 Policies in sync

User replication status:
No Users present

Group replication status:
No Groups present

ILM Expiry Rules replication status:
●  1/1 ILM Expiry Rules in sync

Object replication status:
Replication status since 7 minutes 
Summary:
Replicated:    0 objects (0 B)
Queued:        ● 0 objects, (0 B) (avg: 0 objects, 0 B; max: 0 objects, 0 B)
Received:      0 objects (0 B)

$ ./mc admin replicate status m2
Bucket replication status:
●  1/1 Buckets in sync

Policy replication status:
●  5/5 Policies in sync

User replication status:
No Users present

Group replication status:
No Groups present

ILM Expiry Rules replication status:
●  1/1 ILM Expiry Rules in sync

Object replication status:
Replication status since 8 minutes 
Summary:
Replicated:    0 objects (0 B)
Queued:        ● 0 objects, (0 B) (avg: 0 objects, 0 B; max: 0 objects, 0 B)
Received:      0 objects (0 B)

$ ./mc admin replicate status m1 --ilm-expiry-rules
ILM Expiry Rules replication status:
●  1/1 ILM Expiry Rules in sync

$ ./mc admin replicate status m1 --ilm-expiry-rule cksbdd5b4dtt2fpuur80 
●  ILM Expiry Rule replication summary for: cksbdd5b4dtt2fpuur80

ILMExpiryRule   | M1              | M2             
ILM Expiry Rule | ✔               | ✔

Step-9: Check the replication of removal of expiry rules across sites

$ ./mc ilm rule remove --id cl1l3n5b4dti3t4apagg m1/test-bucket 
Rule ID `cl1l3n5b4dti3t4apagg` from target m1/test-bucket removed.

$./mc ilm rule list m1/test-bucket
mc: <ERROR> Unable to ls lifecycle configuration: lifecycle configuration not set.

$./mc ilm rule list m2/test-bucket 
mc: <ERROR> Unable to get lifecycle. The lifecycle configuration does not exist.

The replication of removal of expiry rule would take few seconds to reflect in site2 as it happens as part of a sync process
Repeat this process keeping second site up and down both. The replication of removed expiry rules should happen as part of sync when second site comes up.

Step-10: Check removal of all ILM rules from first site and all expiry rules should get replicated (removed) from second site as well

$ ./mc ilm rule list m1/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                                                         │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬──────────────┬───────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO TIER │ TIER      │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼──────────────┼───────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            0 │ WARM-TIER │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴──────────────┴───────────┘
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              3 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule list m2/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              3 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule remove --all --force m1/test-bucket 
Rules for `m1/test-bucket` removed.

$ ./mc ilm rule list m1/test-bucket 
mc: <ERROR> Unable to ls lifecycle configuration: lifecycle configuration not set.

$ ./mc ilm rule list m2/test-bucket 
mc: <ERROR> Unable to get lifecycle. The lifecycle configuration does not exist.

again this replication of expiry rules removal would take few secs for site2 (as runs as part of sync)

Step-11: Test editing of expiry rules details and their replication

$ ./mc ilm rule list m1/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                                                         │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬──────────────┬───────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO TIER │ TIER      │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼──────────────┼───────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            0 │ WARM-TIER │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴──────────────┴───────────┘
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              3 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule list m2/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              3 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘


$ ./mc ilm edit --id "cl69m4lb4dtn1aio3rp0" --expire-days "10" m1/test-bucket
Lifecycle configuration rule with ID `cl69m4lb4dtn1aio3rp0` modified  to m1/test-bucket.

$ ./mc ilm rule list m1/test-bucket
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                                                         │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬──────────────┬───────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO TIER │ TIER      │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼──────────────┼───────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            0 │ WARM-TIER │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴──────────────┴───────────┘
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │             10 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule list m2/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │             10 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

Step-12: Check disabling and re-enabling of replication of ilm expiry across site replicated sites

$ ./mc admin replicate info m1
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | true                
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | true         

$ ./mc admin replicate info m2
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | true                
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | true                


$ ./mc admin replicate update m1 --disable-ilm-expiry-replication
Cluster replication configuration updated successfully with:
- replicate-ilm-expiry: false

$ ./mc admin replicate info m1
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | false               
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | false                

$ ./mc admin replicate info m2
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | false               
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | false                

$ ./mc admin replicate update m1 --enable-ilm-expiry-replication 
Cluster replication configuration updated successfully with:
- replicate-ilm-expiry: true

$ ./mc admin replicate info m1
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | true                
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | true                

$ ./mc admin replicate info m2
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
9d3ba3d6-e1b0-4fe7-bd15-0119ead88cae | m1              | http://localhost:9000                          |      | N/A        | true                
56712687-d45e-4fbf-bc98-25e8d5b84c9b | m2              | http://localhost:9010                          |      | N/A        | true

Step-13: Verify healing of enabled/disabled ILM expiry to sites which were offline while changes done
Bring down one of the remote sites and update the ILM expiry flag for the up site

$ ./mc admin replicate update m1 --disable-ilm-expiry-replication
Cluster replication configuration updated successfully with:
- replicate-ilm-expiry: false
- partially failed for few remote sites as they could be down/unreachable at the moment
$ ./mc admin replicate info m1 
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
8ec1747f-a8c7-4061-8fa1-e3bb23982690 | m1              | http://localhost:9000                          |      | N/A        | false               
a6b7e015-ff9b-49db-ae0c-958d2d68de5a | m2              | http://localhost:9010                          |      | N/A        | false                

$ ./mc admin replicate info m2
mc: <ERROR> Unable to get cluster replication information. Get "http://localhost:9010/minio/admin/v3/site-replication/info?api-version=1": dial tcp [::1]:9010: connect: connection refused.

Bring back the remote site and wait for few secs and then check the replication info for the site

$ ./mc admin replicate info m2
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync | Bandwidth  | ILM Expiry Replication
                                     |                 |                                                |      | Per Bucket |                     
8ec1747f-a8c7-4061-8fa1-e3bb23982690 | m1              | http://localhost:9000                          |      | N/A        | false               
a6b7e015-ff9b-49db-ae0c-958d2d68de5a | m2              | http://localhost:9010                          |      | N/A        | false

Update same rule on both site separately with different values. The values should not be healed even after few mins (as it needs ILM expiry to be enabled for all sites for heal to happen)

$ ./mc ilm edit --id "cl69m4lb4dtn1aio3rp0" --expire-days "333" m1/test-bucket 
Lifecycle configuration rule with ID `cl69m4lb4dtn1aio3rp0` modified  to m1/test-bucket.
$ ./mc ilm edit --id "cl69m4lb4dtn1aio3rp0" --expire-days "444" m2/test-bucket
Lifecycle configuration rule with ID `cl69m4lb4dtn1aio3rp0` modified  to m2/test-bucket.

$ ./mc ilm rule list m1/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                                                         │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬──────────────┬───────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO TIER │ TIER      │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼──────────────┼───────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            0 │ WARM-TIER │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴──────────────┴───────────┘
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            333 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule list m2/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            444 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

Re-enable ILM expiry for the site we disabled earlier and wait for few secs. Rules should be healed and latest one should win to update to all sites

$ ./mc admin replicate update m1 --enable-ilm-expiry-replicatione-ilm-expiry 
Cluster replication configuration updated successfully with:
- replicate-ilm-expiry: true

$ ./mc ilm rule list m1/test-bucket
┌────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Transition for latest version (Transition)                                                         │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬──────────────┬───────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO TIER │ TIER      │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼──────────────┼───────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            0 │ WARM-TIER │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴──────────────┴───────────┘
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            444 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

$ ./mc ilm rule list m2/test-bucket 
┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for latest version (Expiration)                                                                     │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬─────────────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ EXPIRE DELETEMARKER │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼─────────────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │            444 │ false               │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴─────────────────────┘
┌──────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Expiration for older versions (NoncurrentVersionExpiration)                                              │
├──────────────────────┬─────────┬────────┬───────────────────────────────┬────────────────┬───────────────┤
│ ID                   │ STATUS  │ PREFIX │ TAGS                          │ DAYS TO EXPIRE │ KEEP VERSIONS │
├──────────────────────┼─────────┼────────┼───────────────────────────────┼────────────────┼───────────────┤
│ cl69m4lb4dtn1aio3rp0 │ Enabled │ shu    │ tag1=val1&tag2=val2&tag3=val3 │              2 │             0 │
└──────────────────────┴─────────┴────────┴───────────────────────────────┴────────────────┴───────────────┘

Needs corresponding changes from minio, console, minio-go and madmin-go

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Optimization (provides speedup with no functional changes)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • Fixes a regression (If yes, please add commit-id or PR # here)
  • Unit tests added/updated
  • Internal documentation updated
  • Create a documentation update request here

@shtripat shtripat force-pushed the support-replicate-ilm-expiry branch 3 times, most recently from 36d2ba1 to 0df318e Compare September 29, 2023 12:47
@shtripat shtripat changed the title Added flag replicate-ilm-expiry to mc admin replication add Option to enable diable ILM expiry for site replication Sep 29, 2023
@shtripat shtripat force-pushed the support-replicate-ilm-expiry branch 2 times, most recently from a36701f to 95b5435 Compare October 9, 2023 06:04
@shtripat shtripat changed the title Option to enable diable ILM expiry for site replication Option to enable disable ILM expiry for site replication Oct 9, 2023
@shtripat shtripat marked this pull request as ready for review October 9, 2023 11:11
@shtripat shtripat requested a review from poornas October 9, 2023 11:12
@shtripat shtripat force-pushed the support-replicate-ilm-expiry branch 2 times, most recently from cdeb151 to 5ee893f Compare October 12, 2023 10:05
@poornas
Copy link
Contributor

poornas commented Oct 16, 2023

you should provide the option to enable ilm expiry replication later as well?

@shtripat
Copy link
Contributor Author

you should provide the option to enable ilm expiry replication later as well?

Ah ok, makes sense. Will add option for the same.

@shtripat shtripat force-pushed the support-replicate-ilm-expiry branch 2 times, most recently from 8c92d6f to be3553c Compare October 17, 2023 12:58
@shtripat
Copy link
Contributor Author

you should provide the option to enable ilm expiry replication later as well?

Ah ok, makes sense. Will add option for the same.

Done @poornas

@poornas
Copy link
Contributor

poornas commented Oct 19, 2023

validations are broken - on update of enable|disable of the new flag, it expects endpoint also to be passed

mc admin replicate info should display the ilm-expiry replication setting
mc admin replicate status needs to display mismatches in ilm expiry repl. as well

@shtripat
Copy link
Contributor Author

shtripat commented Oct 19, 2023 •
edited
Loading

validations are broken - on update of enable|disable of the new flag, it expects endpoint also to be passed

mc admin replicate info should display the ilm-expiry replication setting mc admin replicate status needs to display mismatches in ilm expiry repl. as well

I am able to enable|disable ILM expiry replication below

$ ./mc admin replicate update m1 --deployment-id f212f972-7dc0-4f8e-89ed-f58bbf8a6f32 --enable-ilm-expiry-replication
Cluster replication configuration updated successfully with:
- replicate-ilm-expiry: true

$ ./mc admin replicate update m1 --deployment-id f212f972-7dc0-4f8e-89ed-f58bbf8a6f32 --enable-ilm-expiry-replicationcate-ilm-expiry 
Some sites could not be configured for replication.
ILM expiry already set to true

$ ./mc admin replicate update m1 --deployment-id f212f972-7dc0-4f8e-89ed-f58bbf8a6f32 --disable-ilm-expiry-replicatione-ilm-expiry 
Cluster replication configuration updated successfully with:
- replicate-ilm-expiry: false

$ ./mc admin replicate update m1 --deployment-id f212f972-7dc0-4f8e-89ed-f58bbf8a6f32 --disable-ilm-expiry-replicationate-ilm-expiry 
Some sites could not be configured for replication.
ILM expiry already set to false

It does not mandate to pass --endpoint argument. Am I missing something?

Regarding mc admin replicate info to show ilm expiry settings, I will update the PR. --json output already shows this.
And mc admin replicate status to show mismatches, I will check this and update accordingly.

@shtripat shtripat force-pushed the support-replicate-ilm-expiry branch from be3553c to 62aa11c Compare October 20, 2023 07:25
@shtripat
Copy link
Contributor Author

Updated and now ILM expiry replication details are shown as below

$ ./mc admin replicate info m1                                                                ST 1   support-replicate-ilm-expiry 
SiteReplication enabled for:

Deployment ID                        | Site Name       | Endpoint                                       | Sync
7141750e-6f48-4960-8669-200dc9619552 | m1              | http://localhost:9000                          |     
0ecce7e4-ea6a-41be-a066-b873605c68f8 | m2              | http://localhost:9010                          |     

ILM Expiry Replication: true

and added option --ilm-expiry-rules amd --ilm-expiry-rule <name> to mc admin replicate status command as below

$ ./mc admin replicate status m1                                                              ST 1   support-replicate-ilm-expiry 
Bucket replication status:
●  1/1 Buckets in sync

Policy replication status:
●  5/5 Policies in sync

User replication status:
No Users present

Group replication status:
No Groups present

ILM Expiry Rules replication status:
●  1/1 ILM Expiry Rules in sync

Object replication status:
Replication status since 1 minute 
Summary:
Replicated:    0 objects (0 B)
Queued:        ● 0 objects, (0 B) (avg: 0 objects, 0 B; max: 0 objects, 0 B)
Received:      0 objects (0 B)

$ ./mc admin replicate status m1 --ilm-expiry-rules                                           ST 1   support-replicate-ilm-expiry 
ILM Expiry Rules replication status:
●  1/1 ILM Expiry Rules in sync

$ ./mc admin replicate status m1 --ilm-expiry-rule ckok9v5b4dtgofkbi6tg                       ST 1   support-replicate-ilm-expiry 
●  ILM Expiry Rule replication summary for: ckok9v5b4dtgofkbi6tg

ILMExpiryRule   | M1              | M2             
ILM Expiry Rule | ✔               | ✔

@shtripat shtripat force-pushed the support-replicate-ilm-expiry branch from 62aa11c to 3bdc1d6 Compare October 23, 2023 11:13
@shtripat shtripat mentioned this pull request Oct 23, 2023
Merged
8 tasks
@shtripat shtripat force-pushed the support-replicate-ilm-expiry branch 7 times, most recently from 94a8e9d to bc4efea Compare October 30, 2023 11:39
@shtripat shtripat force-pushed the support-replicate-ilm-expiry branch from bc4efea to 0348585 Compare November 2, 2023 10:29
@shtripat shtripat force-pushed the support-replicate-ilm-expiry branch from 0348585 to eafbf36 Compare November 7, 2023 12:59
@shtripat shtripat requested a review from poornas November 7, 2023 13:46
poornas
poornas requested changes Nov 8, 2023
View reviewed changes
cmd/ilm/options.go Outdated Show resolved Hide resolved
@shtripat shtripat force-pushed the support-replicate-ilm-expiry branch from eb128bf to 6c94adc Compare November 9, 2023 08:33
@shtripat shtripat requested a review from poornas November 9, 2023 08:34
poornas
poornas approved these changes Nov 10, 2023
View reviewed changes
Copy link
Contributor

@poornas poornas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@poornas poornas requested a review from vadmeste November 21, 2023 17:37
harshavardhana
harshavardhana requested changes Nov 22, 2023
View reviewed changes
go.mod Outdated Show resolved Hide resolved
@shtripat
Option to enable diable ILM expiry for site replication
6f017f6 
Added options `replicate-ilm-expiry` and `disable-ilm-expiry-replication`
for `mc admin replicate add` and `mc admin replicate update` commands
respectively.

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat
Added option to enable back ILM expiry for replication
f89a922 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat
Show ILM expiry replication flag in mc admin replicate info
14c6c13 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat
ILM expiry replication details for mc admin replicatte status
93bf324 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat
Support replication of deleted ILM expiry rules
2387387 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat
Updated to latest madmin-go changes from show-ilm-expiry-details
5d7d4a0 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat
Use updated minio-go
980df46 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat
Report ILM Expiry Replication flag sitewise
a90b29d 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat
Apply ilm expiry replication changes to all sites
d766137 
Don't pass `--deployment-id` while using `--disable-ilm-expiry-replication`
or `--enable-ilm-expiry-replication` with `mc admin replicate update`

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat
Correct suppressed header
10b2b90 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat
Dont send expiryRuleRemoved from client
1cb960e 
It would be calculated at server side

Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat
Use latest madmin-go v3.0.32
f550126 
Signed-off-by: Shubhendu Ram Tripathi <shubhendu@minio.io>
@shtripat shtripat force-pushed the support-replicate-ilm-expiry branch from f4921bd to f550126 Compare November 22, 2023 04:25
@harshavardhana harshavardhana merged commit 56d84e6 into minio:master Nov 22, 2023
5 checks passed
@djwfyi djwfyi mentioned this pull request Dec 4, 2023
Closed
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@poornas poornas poornas approved these changes

@harshavardhana harshavardhana harshavardhana approved these changes

@vadmeste vadmeste vadmeste approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@shtripat @poornas @vadmeste @harshavardhana