Distributed Meilisearch

[gmourier]

Hello everyone 👋

We've been asked to implement a way to distribute Meilisearch out of the box for a while now.

We had to delay all related questions to finish a major and stable Meilisearch release, akka v1.0; now that we've done it, I'd like to reboot that subject with the interested parties here to understand your needs better and determine where we should go and what's feasible regarding some prior need we may identify.

To start this topic, I would like to ask you a list of questions:

• Do you need high availability for read ops?
• Do you need high availability for write ops?
• What do you feel about eventual consistency? Is it critical that the whole cluster returns the same document representation no matter the node being queried?
• Is there a need for geo-replication? If so, please share more context!
• I know some talented teams here have been experimenting with things on their own; if you are in the same case, please share the work-around you have implemented and list the difficulties you have encountered; it could be very valuable for everyone here.

If you want to discuss it less formally, you can come to our Discord forum dedicated to this topic and have a chat with us!

[liesislukas]

Overview of setup:

We use DynamodB+Meilisearch combo. DynamoDB has insane scalability, stability, stable speed and actually even increasing speed when scales up, which is not the case for any database which has any kind of data aggregation, but DynamoDB has close to none searching capabilities, while it has no data aggregation. So we use DynamoDB as main data database but we also want to provide great search ux. Meilisearch has amazing capabilities to search something in data set, so we put searchable stuff to meilisearch and use it to provide best possible search capabilities. Once data is searched, any action on top of the data is done with DynamoDB. After data is modified, it's modified in DynamoDB and then synced to Meilisearch so search results would reflect the changes.

1. Do you need high availability for read ops?

yes, especially multi-region.

2. Do you need high availability for write ops?

reads more important, writing can happen from single place.

3. What do you feel about eventual consistency? Is it critical that the whole cluster returns the same document representation no matter the node being queried?

we use meilisearch in combo with DynamoDB, so meilisearch's index is always "eventually consistent". Indexing also takes time even when data sent to meilisearch, so eventually consistency is fine. It's search engine not main database, at least in my view. Caching layer. So it helps to make search patterns but once data is found and we need to do with it something, those queries go to main database, which is dynamodb in my case. In other cases it may be other like sql or mongo.

4. Is there a need for geo-replication? If so, please share more context!

Yes, that's all the point. I run product which use base in Asia, EU and USA. I want to have all edges covered with low-latency search queries. Backend is already hosted in multiple regions. Main database - dynamodb is running at global table setup, so it syncs data between those regions to have local connection.

Though important point is i need it running in my local network. So self-hosted.

5. I know some talented teams here have been experimenting with things on their own; if you are in the same case, please share the work-around you have implemented and list the difficulties you have encountered; it could be very valuable for everyone here.

I have draft of micro service which takes in node's credentials - host, api key and this puts node to the pool. Then service takes in data modification requests, like index create/remove, document set/remove. Those actions are in queue and service applies those actions to nodes. It also keeps all the history of actions so if you would add new node, it would apply all actions on that node too and after some time it would get into sync too.

Node of meilisearch is hosted in local network close to API's backend and backend is making direct queries to meilisearch to get results.

This service is not yet running in production env though but it's pretty small & simple. Just keeps track of actions applied on meilisearch nodes & applies those actions on any new node to keep all nodes up-to-date.

[irevoire]

Super interesting; thanks a lot for your feedback!

If I try to sum up the essential points for us:

• You want meilisearch to index your data as fast as possible
• Your data can be completely unsynchronized between multiple instances. i.e. some documents are searchable on one instance while they're not yet indexed on another one

---

How would you like the errors to be handled?
If one instance is able to index a batch of documents but another can't for some reason (could be an internal error, no space left on the device, or anything).
Would you prefer to continue as-is with missing documents, or should the instance try to download an updated snapshot/dump to come back to where the others are? (or any other solution)

[liesislukas]

in case of error because host is not capable enough - no disk, no ram, cpu 100% etc. node is failed and should not be used. It's up to devops/infrastructure setup to monitor your machines and keep them in healthy state.

i had best experience in any multi-region multi-master setup with DynamoDB global tables. Check how it works. DynamoDB is eventually consistent too when running in multi-region. It also has multi-master, you can do all queries, including modifying data, on any edge and have all of it in eventually consistent. Sync happens within 1s mostly, which is crazy good. It comes down a lot to infrastructure, what network is connecting the nodes but if you could do something like Dynamo, where you just enable global table & select new regions - that's perfect.

While Dynamo also manages all infrastructure, they can automate a lot. With self-hosted meilisearch i would imagine best UX could be this:

1. you run your 1st node and it would host mini dashboard.
2. you run 2nd node and it also host it's own mini dashboard.

these 2 nodes has no clue about each other at this point but you can open each dashboard and see some basic monitoring info and also multi-nodes setup option.

3. you create link between those nodes by inviting/accepting each other & probably exchanging credentials of some kind to keep the nodes in sync. You could add several nodes as needed. Every node has latest data snapshot, if any node goes down, other nodes know about that from heartbeat requests.

In meilisearch SDK you should be able to add several hosts, primary and backup.

https://docs.meilisearch.com/reference/api/overview.html#authorization

From primary hosts it could just round-robin take next for api calls. If primary not responding, it would take any best-responding backup host, meanwhile continue heartbeat to check if primary is up already.

Classical cluster of nodes where 1 is selected "main" and then others just trying to keep in sync with "main". sql does that, aws has it also as managed service called AWS RDS Aurora. What i really dislike though is when database solutions try to make multi-master setup & allows to write to any node but fails to deliver that. E.g. in SQL in theory you can do that, yet in practise i had way too many issues related to this. Then i started to appreciate DynamoDB even more while it just works but again, they have AWS infrastructure on their hands.

Node's dashboard should show data replication lag metric. How long it takes to become consistent. So it would allow to debug things & optimise infrastructure for self-hosted. Also it would be opportunity for your managed hosting solution to show off.

From business side & not self-hosted but rather managed instances - you would be able to setup AWS Private Link connection. Check mongodb atlas, they have done this for users who don't want to manage infrastructure & hosting, but rather buy mongodb hosted solution yet have it in local network inside aws. I think you could do the same. It cost extra though, mongo requires higher end plan and also private link is not free on aws side (iirc ±$20/month + depends on traffic), so the database will cost you probably at least ±$70-$100/month/node which is quite expensive for smaller scale projects but you need to make $ too, so providing such robust solution would attract companies with deeper pockets but that's side note :))

please check https://aws.amazon.com/blogs/apn/connecting-applications-securely-to-a-mongodb-atlas-data-plane-with-aws-privatelink/

[gmourier]

Hey everyone!

Here is some update on that; We are currently experimenting with advancements on that subject, and we may be able to share a prototype soon.

For now, here is what we are headed at:

• It's a model with a leader and followers
  • The leader receives the write ops (e.g Add documents, settings update)
  • The followers answer search requests
• The write consistency level (ALL, QUORUM, ONE, TWO) is configurable via the CLI
• If a follower crashes, it can come back to the cluster and get updated
• A new follower can join the cluster and get updated
• If the leader crashes, there is no re-election, the cluster no longer works, but the followers can still answer search requests. We are still thinking about what we could do about this.
• There is no node auto-discovery yet

cc @irevoire

[sandstrom]

Answers to your questions above:

• Do you need high availability for read ops?
  • Yes
• Do you need high availability for write ops?
  • Short answer: probably not. Long answer: Depends a bit on how quickly a new leader is elected, but generally a search cluster is a slave to a primary database (Postgresl or similiar) and there is an async queue pushing updates from SQL to search cluster. In such a scenario, when a leader fails, as long as new one can be elected within a minute or two, it's fine. Search results will just lag behind for a while.
• What do you feel about eventual consistency? Is it critical that the whole cluster returns the same document representation no matter the node being queried?
  • Eventual consistency is totally fine! It's not critical that the whole cluster agrees on a document representation.
• Is there a need for geo-replication? If so, please share more context!
  • Can be useful! Something like an affinity column/property per document, with an array of affinity tags. Then each cluster member is given one or multiple tags, and it will subscribe to updates only for those tags. That way, the tags could be e.g. ['eu', 'asia'] or ['us', 'ca', 'mx'], but could also be non-geographic affinity.

Other thoughts

• You've probably thought about it, but even if you aren't targeting advanced cluster functionality now, consider using Raft as your algorithm. A good foundation to build upon for the future.
  • If you have this, it's fairly easy to just elect a new master if one fails (eligibility for being a master is usually a config value though, but you'll usually want to have 2-3 possible master nodes, and then via raft you'll elect a new one if a majority of the cluster cannot reach the current master).
• Node auto-discovery is often easy to handle via e.g. Consul or etcd. Doesn't have to be baked into Meilisearch.

[akager123]

Hey. We have a search use case for a large scale enterprise SAAS application. We currently use typesense since meili search does not have any self deployable HA configuration, but would consider switching to meili search since typesense has issues on k8s. Curious if there is any timeline on this as there have been no updates in 8 months.

Do you need high availability for read ops?

• Yes

Do you need high availability for write ops?

• Less critical but a path to eventually having this would be ideal

What do you feel about eventual consistency? Is it critical that the whole cluster returns the same document representation no matter the node being queried?

• Eventual consistency is fine for our use case

Is there a need for geo-replication? If so, please share more context!

• Geo replicated read replicas would be great, though not needed for us short term. Higher write lag for these would be acceptable

[pequalsnp]

• Do you need high availability for read ops?
  Yes. This is the primary scaling challenge we have, it is not that we can't fit the index in a single instance, it's that we need to manually keep shards in sync.

• Do you need high availability for write ops?
  We can queue writes if the cluster is temporarily unavailable. What would be incredibly useful is a way for index updates to fanout to the cluster without having to handle this manually.

• What do you feel about eventual consistency? Is it critical that the whole cluster returns the same document representation no matter the node being queried?
  For most if not all of my use cases, I can handle eventual consistency.

• Is there a need for geo-replication? If so, please share more context!
  It is probably easier to leave this to the implementer. For example, I would use Kafka or something of that nature to broadcast writes to all of my DCs and just index in both places.

• I know some talented teams here have been experimenting with things on their own; if you are in the same case, please share the work-around you have implemented and list the difficulties you have encountered; it could be very valuable for everyone here.
  My challenge is I want to be able to flex capacity because we have a very bursty, but predictable, traffic pattern. So let's say, for example, I need 10 shards all the time, but I need 100 shards for 2 hours. My plan is to snapshot the index periodically and include the offset of the last processed event in the index itself. That way, a new shard can load the index snapshot, consume all of the events they have missed since the offset, and then start accepting requests. Each shard consumes all events and applies it to their own copy of the index.

[adrian-lux]

Do you need high availability for read ops?
Yes, absolutely necessary. The search feature represents the main value of our product so without search we don't have a usable product.

Do you need high availability for write ops?
Not really, writes only need to be done every couple of months when we update our master data. That time we will run a batch job/queue setup that can finish over night and failed jobs will run again until they succeed. So, user actions don't translate to write ops only maintenance tasks carried out by our system admins.

What do you feel about eventual consistency? Is it critical that the whole cluster returns the same document representation no matter the node being queried?
Eventual consistency is a must. However, we don't need consistency in real time. A couple of minutes lag between nodes would be tolerable because we only update our indexes very infrequently and over night as the bulk import of new data takes a long time anyways.

Is there a need for geo-replication?
No, we only target users in Japan so no geo-replication necessary.

[pawarren]

Do you need high availability for read ops?

Yes.

Do you need high availability for write ops?

No.

What do you feel about eventual consistency? Is it critical that the whole cluster returns the same document representation no matter the node being queried?

Yes.

Is there a need for geo-replication? If so, please share more context!

My primary interest is having users around the world experience very fast search. Currently users outside of NA get a slow search experience.

[valenterry]

Do you need high availability for read ops?

Yes.

Do you need high availability for write ops?

No.

What do you feel about eventual consistency? Is it critical that the whole cluster returns the same document representation no matter the node being queried?

No. Eventual consistency is fine, it should recover at some point but temporary different results are no problem.

Is there a need for geo-replication? If so, please share more context!

I personally think if there is support for HA for read ops then geo-replication should be included in that. There will be different instances and where exactly they run does not matter (unless I misunderstand what geo-replication means in this context).