Backbeat is an engine with a messaging system at its heart. It's part of Zenko, Scality’s Open Source Multi-Cloud Data Controller. Learn more about Zenko at Zenko.io
Backbeat is optimized for queuing metadata updates and dispatching work to long-running tasks in the background. The core engine can be extended for many use cases, which are called extensions, as listed below.
This feature replicates objects from one S3 bucket to another S3 bucket in a different geographical region. The extension uses the local Metadata journal as the source of truth and replicates object updates in a FIFO order.
- Backbeat core design
- CRR to AWS S3 workflow
- Object Lifecycle management
- Metrics
- CRR Pause and Resume
- Healthcheck
- Site Level CRR
- Transient Source
- Out of Band updates from RING
This guide assumes the following:
- Using MacOS
brew
is installed (get it here)node
is installed (version 6.9.5)yarn
is installed (version 3.10.10)aws
is installed (version 1.11.1)
mkdir ~/kafka && \
cd ~/kafka && \
curl https://archive.apache.org/dist/kafka/0.11.0.0/kafka_2.11-0.11.0.0.tgz | \
tar xvz && \
sed 's/zookeeper.connect=.*/zookeeper.connect=localhost:2181\/backbeat/' \
kafka_2.11-0.11.0.0/config/server.properties > \
kafka_2.11-0.11.0.0/config/server.properties.backbeat
Start the zookeeper server:
zookeeper-server-start kafka_2.11-0.11.0.0/config/zookeeper.properties
In a different Shell start the kafka server:
kafka-server-start kafka_2.11-0.11.0.0/config/server.properties.backbeat
The "Failed Entry Processor" (started by Queue Populator) stores the "failed replication entries" to the Redis sorted set.
zkCli -server localhost:2181/backbeat
create /queue-populator
kafka-topics --create \
--zookeeper localhost:2181/backbeat \
--replication-factor 1 \
--partitions 1 \
--topic backbeat-replication
kafka-topics --create \
--zookeeper localhost:2181/backbeat \
--replication-factor 1 \
--partitions 1 \
--topic backbeat-replication-status
kafka-topics --create \
--zookeeper localhost:2181/backbeat \
--replication-factor 1 \
--partitions 1 \
--topic backbeat-replication-failed
git clone https://github.com/scality/Vault ~/replication/vault && \
cd ~/replication/vault && \
yarn i
Run vault with "in-memory backend"
chmod 400 ./tests/utils/keyfile
VAULT_DB_BACKEND="MEMORY" yarn start
⚠️ with "in-memory backend", all data is lost after you stop the process.
or with a "mongodb backend"
chmod 400 ./tests/utils/keyfile
VAULT_DB_BACKEND="MONGODB" yarn start
mongodb can be installed follwing these steps
git clone https://github.com/scality/cloudserver ~/replication/cloudserver && \
cd ~/replication/cloudserver && \
yarn i
Generate AWS access/secret key with full IAM and S3 access.
Add your keys aws configure --profile aws-account
Create AWS destination versioning-enabled bucket.
aws s3api create-bucket --bucket <DESTINATION_BUCKET_NAME> --profile aws-account
aws s3api put-bucket-versioning \
--bucket <DESTINATION_BUCKET_NAME> \
--versioning-configuration Status=Enabled \
--profile aws-account
Replace existing ./locationConfig.json
with:
{
"us-east-1": {
"details": {
"supportsVersioning": true
},
"isTransient": false,
"legacyAwsBehavior": false,
"objectId": "0b1d9226-a694-11eb-bc21-baec55d199cd",
"type": "file"
},
"aws-location": {
"type": "aws_s3",
"legacyAwsBehavior": true,
"details": {
"awsEndpoint": "s3.amazonaws.com",
"bucketName": "<DESTINATION_BUCKET_NAME>",
"bucketMatch": false,
"credentialsProfile": "aws-account",
"serverSideEncryption": true
}
}
}
Update ./config.json
with
"replicationEndpoints": [{ "site": "aws-location", "type": "aws_s3" }],
In ./config.json
, make sure recordLog.enabled
is set to true
"recordLog": {
"enabled": true,
...
}
Run Cloudserver
S3DATA=multiple S3METADATA=mongodb REMOTE_MANAGEMENT_DISABLE=true \
S3VAULT=scality yarn start
Create a "Zenko" account and generate keys
bin/vaultclient create-account --name bart --email dev@backbeat --port 8600
bin/vaultclient generate-account-access-key --name bart --port 8600
aws configure --profile bart
git clone https://github.com/scality/backbeat ~/replication/backbeat && \
cd ~/replication/backbeat && \
yarn i
Update conf/authdata.json
with bart informations and keys.
{
"accounts": [{
"name": "bart",
"arn": "aws:iam::331457510670:/bart",
"canonicalID": "2083781e15384e30f48c651a948ec2dc1e1801c4af24c2750a166823e28ca570",
"displayName": "bart",
"keys": {
"access": "20TNCD06HOCSLQSABFZP",
"secret": "1P43SL0ekJjXnQvliV0KgMibZ=N2lKZO4dpnWzbF"
}
}
]
}
Update conf/config.json
section extensions.replication.source.auth
"auth": {
"type": "account",
"account": "bart",
"vault": {
"host": "127.0.0.1",
"port": 8500,
"adminPort": 8600
}
}
Make sure conf/config.json
section
extensions.replication.destination.bootstrapList
includes:
{ "site": "aws-location", "type": "aws_s3" }
S3_REPLICATION_METRICS_PROBE=true REMOTE_MANAGEMENT_DISABLE=true \
yarn run queue_populator
S3_REPLICATION_METRICS_PROBE=true REMOTE_MANAGEMENT_DISABLE=true \
yarn run queue_processor aws-location
S3_REPLICATION_METRICS_PROBE=true REMOTE_MANAGEMENT_DISABLE=true \
yarn run replication_status_processor
Create a source bucket with versioning enabled:
aws s3api create-bucket \
--bucket sourcebucket \
--endpoint-url http://127.0.0.1:8000 \
--profile bart
aws s3api put-bucket-versioning \
--bucket sourcebucket \
--versioning-configuration Status=Enabled \
--endpoint-url=http://127.0.0.1:8000 \
--profile bart
Create replication.json
{
"Role": "arn:aws:iam::root:role/s3-replication-role",
"Rules": [
{
"Status": "Enabled",
"Prefix": "",
"Destination": {
"Bucket": "arn:aws:s3:::sourcebucket",
"StorageClass": "aws-location"
}
}
]
}
aws s3api put-bucket-replication \
--bucket sourcebucket \
--replication-configuration file://replication.json \
--endpoint-url=http://127.0.0.1:8000 \
--profile bart
aws s3api put-object \
--key key0 \
--body file \
--bucket sourcebucket \
--endpoint-url=http://127.0.0.1:8000 \
--profile bart
aws s3api head-object \
--key bucketsource/key0 \
--bucket <DESTINATION_BUCKET_NAME> \
--profile aws-account
In our $HOME
directory, we now have the following directories:
$HOME
├── kafka
│ └── kafka_2.11-0.11.0.0
├── replication
├── backbeat
├── cloudserver
└── vault