Replicate instances over multiple hosts

[dimitar-ivanov-93]

Hi guys, I am currently trying to set up the Mimir instances from the demo that replicate data between each other. But instead of setting the containers on one host, I am trying to run them from different EC2 instances for higher availability in case of single instance downtime. The current configuration is as follows:

The docker-compose.yml part from the first EC2 instance:

  mimir-1:
    image: grafana/mimir:latest
    command: ["-config.file=/etc/mimir.yml"]
    hostname: mimir-1
    volumes:
      - ./config/mimir.yml:/etc/mimir.yml
      - ./config/alertmanager-fallback-config.yml:/etc/alertmanager-fallback-config.yml
      - mimir-1-data:/data

The mimir.yml config from the first machine:

target: all,alertmanager,overrides-exporter

# Configure Mimir to use Minio as object storage backend.
common:
  storage:
    backend: s3
    s3:
      endpoint: --
      region: --
      access_key_id: --
      secret_access_key: --
      insecure: --
      bucket_name: --

# Blocks storage requires a prefix when using a common object storage bucket.
blocks_storage:
  storage_prefix: blocks
  tsdb:
    dir: /data/ingester

# Use memberlist, a gossip-based protocol, to enable the 3 Mimir replicas to communicate
memberlist:
  join_members: [ IP_OF_FIRST_HOST, IP_OF_SECOND_HOST ]

ruler:
  rule_path: /data/ruler
  alertmanager_url: http://127.0.0.1:8080/alertmanager
  ring:
    # Quickly detect unhealthy rulers to speed up the tutorial.
    heartbeat_period: 2s
    heartbeat_timeout: 10s

alertmanager:
  data_dir: /data/alertmanager
  fallback_config_file: /etc/alertmanager-fallback-config.yml
  external_url: http://localhost:9009/alertmanager

server:
  log_level: warn

Respectively, the compose part from the second EC2 instance:

  mimir-2:
  image: grafana/mimir:latest
  command: ["-config.file=/etc/mimir.yml"]
  hostname: mimir-2
  volumes:
    - ./config/mimir.yml:/etc/mimir.yml
    - ./config/alertmanager-fallback-config.yml:/etc/alertmanager-fallback-config.yml
    - mimir-2-data:/data

The mimir.yml on the second EC2 instance is identical to the first.

As expected, the joined instances were 0 due to the port not being exposed, which led to the following warning:

containers-mimir-2-1     | level=warn ts=2022-11-02T11:19:42.216948225Z caller=memberlist_client.go:563 msg="memberlist fast-join finished" joined_nodes=0 elapsed_time=20.001673025s
containers-mimir-2-1     | level=warn ts=2022-11-02T11:20:02.217984797Z caller=memberlist_client.go:598 msg="joining memberlist cluster: failed to reach any nodes" retries=0 err="2 errors occurred:\n\t* Failed to join IP_ADDRESS:7946: dial tcp IP_ADDRESS:7946: i/o timeout\n\t* Failed to join IP_ADDRESS:7946: dial tcp IP_ADDRESS:7946: i/o timeout\n\n"

After that I have linked port 7946 from the container to the same port on the host in hope that it will be able to reach eachother, but I had no luck.
Added block to both compose files:

    ports:
      - 8080:8080
      - 7946:7946

When making a request to the hosts' port 7946, I get the following error from mimir:

containers-mimir-2-1     | level=error ts=2022-11-02T11:53:49.502969966Z caller=tcp_transport.go:318 component="memberlist TCPTransport" msg="unknown message type" msgType=G remote=IP_ADDRESS:59631

I have looked into Mirroring requests to a second Mimir cluster using Envoy Proxy, but I would very much like to avoid that and keep the high-availability setup with replication of instances, just on separate hosts.

Any suggestions are welcome!

[dimitarvdimitrov]

Hey, the port 7946 is not meant to serve HTTP, but the memberlist gossip protocol. It's meant for communication between the 3 Mimir nodes. I believe this is the reason for the last error in your post.

I'm trying to understand the problem you are facing - is ingestion not working or can you not query data from the cluster?

Regarding the HA setup - running 3 nodes with replication factor of 3 gives you some resilience to failures. I assume the 3 instances run in the same availability zone. If you want to be resilient to AZ failure then you can spin up a second cluster in a different AZ. It's also an option to deploy the 3 instances of that cluster to a different AZ each, but that will have increased network costs compared to running two separate clusters (that's network costs only, excluding compute costs).

As for mirroring traffic to two clusters: for writes the easiest way is to configure the clients (e.g. Prometheus or Grafana Agent) to remote-write to both clusters. Then you need not worry about spinning up a proxy. For querying I am aware of promxy which can query multiple prometheus-compatible servers and stitch together the results, but I have not tried it. There may be other alternatives for querying.

[dimitar-ivanov-93]

Thank you for the information.

As of now, I am not as worried about AZ failure as much as single EC2 failure. That is why I don't neceserily want to run the 3 Mimir instances on a single EC2. The thing I am struggling with is enabling the replication between the 3 Mimir instances, positioned inside docker containers, each on a different EC2 instance within the same AZ.

"It's also an option to deploy the 3 instances of that cluster to a different AZ each, but that will have increased network costs compared to running two separate clusters (that's network costs only, excluding compute costs)."

If I were to go with this option, how would I make the 3 instances replicate one another using the memberlist gossip protocol? So far, I have only managed to make that work by having the 3 instances containerized on the same EC2.

As for mirroring the traffic by remote writing to both clusters, that is a really good idea and I will implement that, thank you. Would you in this case use different S3 buckets for each cluster, or make them write to the same bucket and avoid duplication somehow?

[dimitarvdimitrov]

If I were to go with this option, how would I make the 3 instances replicate one another using the memberlist gossip protocol? So far, I have only managed to make that work by having the 3 instances containerized on the same EC2.

Mimir does this by default :) it replicas data 3 ways. So in your case when you have only 3 nodes, it replicates it completely across the 3. So if the remote-write clients are successfully sending data (i.e. no error logs and you see the data when querying), you can assume it's replicated.

Note that with replication factor of 3 Mimir can only survive a single node failure because it always searches for a majority quorum. You can read about replication and failure modes in this article (it talks about the ingester, which is included in the monolithic nodes that you are running with target: all).

Would you in this case use different S3 buckets for each cluster, or make them write to the same bucket and avoid duplication somehow?

You should use separate buckets. If you use the same bucket you will at least run into occasional inconsistencies when querying the data. In our experience storage is roughly 10% of the cost of running Mimir

[dimitar-ivanov-93]

Thank you so much for the article about zone-aware replication. That should take care of the rest of my problems. Also, I will follow your suggestion regarding the different buckets. Please keep being as helpful as you are!