Idea: use a search engine to power search

[vignesh-sankaran]

Just putting an idea out there, maybe we could look into using something like Lucene or ElasticSearch (powered by Lucene too if I'm not mistaken) sometime down the road. I interned in the Search Quality team at Seek Limited a couple of years back, so can ask some developers about some of the intricacies of setting this up if this is something of interest :).

[kureuil]

An https://www.algolia.com/ employee once said that we were eligible for their free plan.
It could be simpler to setup in the near-term, though it's probably "safer" to be in control of all of the stack (also would avoid private crates.io instances depending on an external service).

[vignesh-sankaran]

Yep, I feel we should be running it locally, because if we change the way we do search to an online service, it could be a problem for a private crates.io instance running behind a corporate firewall. That and we can't develop + test crates.io offline :).

So I've come back with a bag of alternative Rust-based search engines too we could look at, depending on what features we need and how complete they are. The list I have currently is the following:

• tantivy
• noise
• perlin
• kite
• rusticsearch

If this is something of interest to implement, I can start researching how to set this up. If I'm going to work on this, it could be a while before I get to this.

[carols10cents]

I'm interested in knowing exactly how a different search will improve on results, and what they make possible that isn't possible with postgres, before adding another dependency. Postgres has quite powerful search capabilities, but we're definitely not using them all in the best ways we could be. I'd be worried we'd switch to something else that we also wouldn't spend time tuning and be no better off except for an extra dependency.

This may involve researching the different systems, and it may involve actually using them and comparing results to what we have today.

[vignesh-sankaran]

This is something that needs to be researched, and I'm currently unable to do that :(

[Deedasmi]

Just tagging #370 on here as an alternative and #159 as a motivation.

[erlend-sh]

I'd love to chat more about this if there's still interest. We made a live demo for Crates powered by MeiliSearch:

https://crates.meilisearch.com/

As a first trial project, you could start by using MeiliSearch just for its client-side, search-as-you-type functionality. You could self-host MeiliSearch or we'd provide you with managed API hosting for free.

[TheGP]

The https://lib.rs/ even shows proper results for "dotenv" query, I'm switching to it, thanks @erlend-sh!

So will there be any movements regarding improving the search? It should be a priority as finding the right tools for building a project is important. You don't have resources or are you working on some less important tasks?

[Turbo87]

It should be a priority as finding the right tools for building a project is important

what good is better search if the API is offline because of scaling issue? we've mostly been focused on dealing with growth issues and unfortunately we're not quite done with that yet. I agree that a good search is important, but there are a lot of other things that are also important.

So will there be any movements regarding improving the search?

do you want to help build this? crates.io is a free service and the code is open-source, so if you see a way to improve the service you are welcome to implement it.

You don't have resources or are you working on some less important tasks?

we have very few resources and, as mentioned above, yes, they are currently allocated differently