[Search] Investigate Full-Text Search

[loic-sharma]

Today, the database-backed search service cannot handle nuget.org's scale. Investigate:

• Over-all issue: Support to Full-Text Search dotnet/efcore#1590
• Full-text search for SQLite: SQLite full-text search support dotnet/efcore#4823
• External full-text search (like Azure Search): Support for external full-text search engines dotnet/efcore#11489

[WernerMairl]

Can we spec out the requirements for search as first ?
(assuming that you can provide detail about scaling problems on nuget.org)

the search service doc is really short:

The search API allows a client to query for a page of packages matching a specified search query. The interpretation of the search query (e.g. the tokenization of the search terms) is determined by the server implementation but the general expectation is that the search query is used for matching package IDs, titles, descriptions, and tags. Other package metadata fields may also be considered.

generalized usecase:
Nuget search usually is implemented as a singe textbox/searchfield.
typically the user inserts his search as a single string in the following pattern:
„xx[ yy][ zz]

how the search query looks like ?

• fields where the search should be applied
• operators (startsWith, contains, endwith)
• case sensitivity
• AND or OR between xx,yy,zz
• there is a minimum lengt for xx ?

[loic-sharma]

• Fields where the search should be applied - Package id, description, and tags seem reasonable
• operators (startsWith, contains, endwith) - Ideally, we should support as much of lucene's query syntax as possible.
• there is a minimum lengt for xx ? - The current search service implementation has no minimum or maximum. This gives the best user experience as search results are available as soon as possible. This isn't a hard requirement though, and I'm fine with imposing limits for technical reasons.