Has anyone tried other memcached alternatives for chunks cache?

[jakubgs]

I have a deployment of Cortex which uses a hetzner host for chunks cache Memcached because large memory hosts are cheap compared to our cloud provider where the Cortex cluster lives. The issue with this is that latency and bandwidth issues make it less than optimal.

What I've been thinking is maybe I could deploy one of these which have a Memcached interface but store data on disk:

• https://github.com/twitter/fatcache
• https://github.com/twitter/pelikan
• https://github.com/LMDB/memcachedb/

Because the main reason for having this cache as far as I can tell is avoiding making too many requests to the block storage, which usually would result in getting throttled.

The documentation states that the only chunks cache supported is memcached:

  chunks_cache:
    # Backend for chunks cache, if not empty. Supported values: memcached.
    # CLI flag: -blocks-storage.bucket-store.chunks-cache.backend
    [backend: <string> | default = ""]

So my idea was to us a host in the same DC as the cluster, but use storage instead of RAM for caching chunks. Good idea? Bad?

Honestly, I wish Cortex would just support filesystem option for bucket_store.chunks_cache.backend.

[yeya24]

Hey @jakubgs, memcached has a mode called extstore which uses disk to cache items.

Grafana Loki team has a blog post about it https://grafana.com/blog/2023/08/23/how-we-scaled-grafana-cloud-logs-memcached-cluster-to-50tb-and-improved-reliability/.

[jakubgs]

I wasn't aware of the extstore extension for memcached. It needs to be built with it, but looks promising.

Thanks for this.