rustdoc: compute maximum Levenshtein distance based on the query

[notriddle]

Preview: https://notriddle.com/notriddle-rustdoc-demos/search-lev-distance/std/index.html?search=regex

The heuristic is pretty close to the name resolver, maxLevDistance = Math.ceil(queryLen / 3).

Fixes #103357
Fixes #82131

[rustbot]

r? @jsha

(rustbot has picked a reviewer for you, use r? to override)

[rustbot]

Some changes occurred in HTML/CSS/JS.

cc @GuillaumeGomez, @Folyd, @jsha

[GuillaumeGomez]

Thanks, looks like a nice improvement! Considering it's quite a big change for users, I'd prefer to have more people from rustdoc to take a look first.

cc @rust-lang/rustdoc

[jsha]

Looking at just the results, not the code: this is a big improvement in the results for short and no-results queries. Thanks!

There's still a problem where single-character searches return a lot of ::0 enum fields, which I thought would happen to be fixed by this. But that's not a blocker by any means; presumably we just need to find the fix elsewhere.

[notriddle]

That’s because Math.ceil(1/3)=1, and the edit distance between 0 and X is 1. The problem with tweaking the heuristic to make the number come out 0 is that the results aren’t really better: I would like X to match XY, but that one also has an edit distance of 1.

I’d almost rather just exclude tuple indices from the search index. Is anyone really going to type Some::0 into the search engine?

[Manishearth]

Yeah we should exclude them

[GuillaumeGomez]

That’s because Math.ceil(1/3)=1, and the edit distance between 0 and X is 1. The problem with tweaking the heuristic to make the number come out 0 is that the results aren’t really better: I would like X to match XY, but that one also has an edit distance of 1.

Maybe we should use floor instead? Also, it should still have xy if you search for x since xy starts with x.

I’d almost rather just exclude tuple indices from the search index. Is anyone really going to type Some::0 into the search engine?

+1

[jsha]

Oh, I thought it was something specific about the 0; but it's just the combination of:

• maxLevenshtein for query of length 1 is 1, plus
• 0 sorts ahead of everything else, plus
• we cut off results past 100

So, yes, I agree we should just not index these; but also we should use floor. Otherwise, once all the 0s are gone, any other single-letter idents will match any single-letter queries.

[GuillaumeGomez]

Seems like we all agree on removing the tuple elements from the index. I'll send a PR tomorrow.

[notriddle]

Also, it should still have xy if you search for x since xy starts with x.

The problem with searching with maxLevDistance = 0 is that substring matches are not necessary treated as lev = 0. They just act as a modifier for levenshtein distance.

rust/src/librustdoc/html/static/js/search.js

Lines 1328 to 1349 in cc8155f

	if (searchWord.indexOf(elem.pathLast) > -1 ||
	row.normalizedName.indexOf(elem.pathLast) > -1
	) {
	index = row.normalizedName.indexOf(elem.pathLast);
	}
	lev = levenshtein(searchWord, elem.pathLast);
	if (lev > 0 && elem.pathLast.length > 2 && searchWord.indexOf(elem.pathLast) > -1) {
	if (elem.pathLast.length < 6) {
	lev = 1;
	} else {
	lev = 0;
	}
	}
	lev += lev_add;
	if (lev > maxLevDistance) {
	return;
	} else if (index !== -1 && elem.fullPath.length < 2) {
	lev -= 1;
	}
	if (lev < 0) {
	lev = 0;
	}

[GuillaumeGomez]

Maybe we should change this behaviour? Something like:

• If the item name starts with X, then give it a lev distance of 2.
• If the item name contains X, then give it a lev distance of 3.

I'm curious of what the results would look like.

[jsha]

I think it's a mistake to use levenshtein distance also the ranking metric and apply boosts and demerits to it. The existing code already does this to a large extent and it makes things very confusing and hard to maintain. It would be better to more explicitly separate the matching and ranking functionality.

[GuillaumeGomez]

That would make things simpler. I'm also curious to see what the results will look like.

[bors]

☔ The latest upstream changes (presumably #103829) made this pull request unmergeable. Please resolve the merge conflicts.