Skip to content

Batch replication demo

Jump to bottom
Allan Roger Reid edited this page May 2, 2024 · 1 revision

Setup aliases

mc alias set acme-source https://127.0.0.1:9000 minioadmin minioadmin
mc alias set acme-target https://127.0.0.1:9001 minioadmin minioadmin

Setup users and policies

cat << EOF > sourcepolicy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "admin:SetBucketTarget",
                "admin:GetBucketTarget"
            ],
            "Effect": "Allow",
            "Sid": "EnableRemoteBucketConfiguration"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold",
                "s3:PutReplicationConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationRuleConfiguration"
        }
    ]
}
EOF

cat << EOF > targetpolicy.json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:GetBucketLocation",
                "s3:GetBucketVersioning",
                "s3:GetBucketObjectLockConfiguration",
                "s3:GetEncryptionConfiguration"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicationOnBucket"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetReplicationConfiguration",
                "s3:ReplicateTags",
                "s3:AbortMultipartUpload",
                "s3:GetObject",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionTagging",
                "s3:PutObject",
                "s3:PutObjectRetention",
                "s3:PutBucketObjectLockConfiguration",
                "s3:PutObjectLegalHold",
                "s3:DeleteObject",
                "s3:ReplicateObject",
                "s3:ReplicateDelete"
            ],
            "Resource": [
                "arn:aws:s3:::*"
            ],
            "Sid": "EnableReplicatingDataIntoBucket"
        }
    ]
}
EOF

Create users with policies

mc admin user add acme-source sourceuser source123

mc admin policy detach acme-source sourcepolicy --user sourceuser
mc admin policy rm acme-source sourcepolicy
mc admin policy create acme-source sourcepolicy sourcepolicy.json
mc admin policy info acme-source sourcepolicy
mc admin policy attach acme-source sourcepolicy --user sourceuser
mc admin policy entities acme-source --user sourceuser

mc admin user add acme-target targetuser target123

mc admin policy detach acme-target targetpolicy --user targetuser
mc admin policy rm acme-target targetpolicy
mc admin policy create acme-target targetpolicy targetpolicy.json
mc admin policy info acme-target targetpolicy
mc admin policy attach acme-target targetpolicy --user targetuser
mc admin policy entities acme-target --user targetuser

Create test buckets

mc rm --force --recursive --versions acme-source/target
mc rb --force acme-source/target
mc mb acme-source/target

mc rm --force --recursive --versions acme-target/target
mc rb --force acme-target/target
mc mb acme-target/target

Add test file to source

mc cp --recursive /etc/hosts acme-source/target

Check for any differences between source and target

mc diff acme-source/target acme-target/target

< https://127.0.0.1:9000/target/hosts
< https://127.0.0.1:9000/target/hosts.equiv

Generate batch replicate yaml. See acme-target-replicate.yaml below.

Also note that since mc is run with respect to acme-source, the source endpoint and credentials must be omitted

mc batch generate acme-source replicate

Edit then run batch replicate

mc batch start acme-source acme-target-replicate.yaml

Successfully started 'replicate' job `Ls8cM5JgPCaQ3CcbW59wHk:-1` on '2024-05-02 20:07:43.51907 +0000 UTC'

Admin on batch replicate

mc batch list acme-source
mc admin info acme-target
mc batch status acme-source 'Ls8cM5JgPCaQ3CcbW59wHk'
mc batch describe acme-source 'Ls8cM5JgPCaQ3CcbW59wHk'

Check for any differences between source and target

mc diff acme-source/target acme-target/target

cat << EOF > acme-target-replicate.yaml
replicate:
  apiVersion: v1
  # source of the objects to be replicated
  source:
    type: minio # valid values are "s3" or "minio"
    bucket: target
    prefix: "" # 'PREFIX' is optional

  # target where the objects must be replicated
  target:
    type: minio # valid values are "s3" or "minio"
    bucket: target
    prefix: ""
    # If your source is the 'local' alias specified to 'mc batch start', then the 'endpoint' and 'credentials' fields are optional and can be omitted

    # Either the 'source' or 'remote' *must* be the "local" deployment
    endpoint: "https://127.0.0.1:9001" 
    # path: "on|off|auto" # "on" enables path-style bucket lookup. "off" enables virtual host (DNS)-style bucket lookup. Defaults to "auto"
    credentials:
      accessKey: targetuser
      secretKey: target123
  
  flags:
    notify:
      endpoint: "https://webhook.site/8776533a-b8d7-4d61-b11e-1bbd8b617dd2" # notification endpoint to receive job status events

    retry:
      attempts: 3 # number of retries for the job before giving up
      delay: "500ms" # least amount of delay between each retry
EOF
Clone this wiki locally