Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add compose file for multi-cluster setup #4032

Merged
merged 1 commit into from
Mar 4, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ We also provide several other compose files for different features/modes:
* docker-compose-mysql.yml uses MySQL as persistence storage
* docker-compose-postgres.yml uses PosstgreSQL as persistence storage
* docker-compose-prometheus.yaml runs with Prometheus
* docker-compose-multiclusters.yaml runs with 2 cadence clusters

Also feel free to make your own to combine the above features.

Expand Down
28 changes: 20 additions & 8 deletions docker/config_template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ services:
metrics:
statsd:
hostPort: {{ .Env.STATSD_ENDPOINT }}
prefix: "cadence-frontend"
prefix: {{ default .Env.STATSD_FRONTEND_PREFIX "cadence-frontend" }}
{{- else if .Env.PROMETHEUS_ENDPOINT }}
metrics:
prometheus:
Expand All @@ -138,7 +138,7 @@ services:
metrics:
statsd:
hostPort: {{ .Env.STATSD_ENDPOINT }}
prefix: "cadence-matching"
prefix: {{ default .Env.STATSD_MATCHING_PREFIX "cadence-matching" }}
{{- else if .Env.PROMETHEUS_ENDPOINT }}
metrics:
prometheus:
Expand All @@ -159,7 +159,7 @@ services:
metrics:
statsd:
hostPort: {{ .Env.STATSD_ENDPOINT }}
prefix: "cadence-history"
prefix: {{ default .Env.STATSD_HISTORY_PREFIX "cadence-history" }}
{{- else if .Env.PROMETHEUS_ENDPOINT }}
metrics:
prometheus:
Expand All @@ -180,7 +180,7 @@ services:
metrics:
statsd:
hostPort: {{ .Env.STATSD_ENDPOINT }}
prefix: "cadence-worker"
prefix: {{ default .Env.STATSD_WORKER_PREFIX "cadence-worker" }}
{{- else if .Env.PROMETHEUS_ENDPOINT }}
metrics:
prometheus:
Expand All @@ -196,14 +196,26 @@ services:
clusterMetadata:
enableGlobalDomain: {{ default .Env.ENABLE_GLOBAL_DOMAIN "false" }}
failoverVersionIncrement: 10
masterClusterName: "master"
currentClusterName: "master"
masterClusterName: "primary"
{{- if .Env.IS_NOT_PRIMARY }}
currentClusterName: "secondary"
{{- else }}
currentClusterName: "primary"
{{- end }}
clusterInformation:
master:
primary:
enabled: true
initialFailoverVersion: 0
rpcName: "cadence-frontend"
rpcAddress: {{ default .Env.BIND_ON_IP "127.0.0.1" }}:7933
rpcAddress: {{ default .Env.PRIMARY_SEEDS "cadence" }}:7933
{{- if .Env.ENABLE_GLOBAL_DOMAIN }}
secondary:
enabled: true
initialFailoverVersion: 2
rpcName: "cadence-frontend"
rpcAddress: {{ default .Env.SECONDARY_SEEDS "cadence-secondary" }}:7933
{{- end }}


dcRedirectionPolicy:
policy: {{ default .Env.DC_REDIRECT_POLICY "selected-apis-forwarding" }}
Expand Down
72 changes: 72 additions & 0 deletions docker/docker-compose-multiclusters.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
version: '3'
services:
cassandra:
image: cassandra:3.11
ports:
- "9042:9042"
statsd:
image: graphiteapp/graphite-statsd
ports:
- "8080:80"
- "2003:2003"
- "8125:8125"
- "8126:8126"
cadence:
image: ubercadence/server:master-auto-setup
ports:
- "7933:7933"
- "7934:7934"
- "7935:7935"
- "7939:7939"
environment:
- "CASSANDRA_SEEDS=cassandra"
- "STATSD_ENDPOINT=statsd:8125"
- "DYNAMIC_CONFIG_FILE_PATH=config/dynamicconfig/development.yaml"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The replication has to be config from the yaml file. Take a look at the development_active.yaml and development_standby.yaml

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it uses config_template.yaml to generate config file. This environment variable is set to get dynamic config.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see. This is a dynamic config yaml. Then it looks good to me.

- "ENABLE_GLOBAL_DOMAIN=true"
- "KEYSPACE=cadence_primary"
- "VISIBILITY_KEYSPACE=cadence_visibility_primary"
- "STATSD_FRONTEND_PREFIX=cadence-frontend-primary"
- "STATSD_MATCHING_PREFIX=cadence-matching-primary"
- "STATSD_HISTORY_PREFIX=cadence-history-primary"
- "STATSD_WORKER_PREFIX=cadence-worker-primary"
depends_on:
- cassandra
- statsd
cadence-secondary:
image: ubercadence/server:master-auto-setup
ports:
- "7943:7933"
- "7944:7934"
- "7945:7935"
- "7949:7939"
Comment on lines +38 to +41
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Can we make the port number consistent with those in development_standby.yaml? Makes it a bit easier when switching back and forth between different setup.

  bootstrapHosts: [ "127.0.0.1:8933", "127.0.0.1:8934", "127.0.0.1:8935", "127.0.0.1:8940" ]

environment:
- "CASSANDRA_SEEDS=cassandra"
- "STATSD_ENDPOINT=statsd:8125"
- "DYNAMIC_CONFIG_FILE_PATH=config/dynamicconfig/development.yaml"
- "IS_NOT_PRIMARY=true"
- "ENABLE_GLOBAL_DOMAIN=true"
- "KEYSPACE=cadence_secondary"
- "VISIBILITY_KEYSPACE=cadence_visibility_secondary"
- "STATSD_FRONTEND_PREFIX=cadence-frontend-secondary"
- "STATSD_MATCHING_PREFIX=cadence-matching-secondary"
- "STATSD_HISTORY_PREFIX=cadence-history-secondary"
- "STATSD_WORKER_PREFIX=cadence-worker-secondary"
depends_on:
- cassandra
- statsd
cadence-web:
image: ubercadence/web:latest
environment:
- "CADENCE_TCHANNEL_PEERS=cadence:7933"
ports:
- "8088:8088"
depends_on:
- cadence
cadence-web-secondary:
image: ubercadence/web:latest
environment:
- "CADENCE_TCHANNEL_PEERS=cadence-secondary:7933"
ports:
- "8089:8088"
depends_on:
- cadence-secondary