Skip to content

Commit c56f632

Browse files
committed
rustdoc-search: use set ops for ranking and filtering
This commit adds ranking and quick filtering to type-based search, improving performance and having it order results based on their type signatures. Motivation ---------- If I write a query like `str -> String`, a lot of functions come up. That's to be expected, but `String::from_str` should come up on top, and it doesn't right now. This is because the sorting algorithm is based on the functions name, and doesn't consider the type signature at all. `slice::join` even comes up above it! To fix this, the sorting should take into account the function's signature, and the closer match should come up on top. Guide-level description ----------------------- When searching by type signature, types with a "closer" match will show up above types that match less precisely. Reference-level explanation --------------------------- Functions signature search works in three major phases: * A compact "fingerprint," based on the [bloom filter] technique, is used to check for matches and to estimate the distance. It sometimes has false positive matches, but it also operates on 128 bit contiguous memory and requires no backtracking, so it performs a lot better than real unification. The fingerprint represents the set of items in the type signature, but it does not represent nesting, and it ignores when the same item appears more than once. The result is rejected if any query bits are absent in the function, or if the distance is higher than the current maximum and 200 results have already been found. * The second step performs unification. This is where nesting and true bag semantics are taken into account, and it has no false positives. It uses a recursive, backtracking algorithm. The result is rejected if any query elements are absent in the function. [bloom filter]: https://en.wikipedia.org/wiki/Bloom_filter Drawbacks --------- This makes the code bigger. More than that, this design is a subtle trade-off. It makes the cases I've tested against measurably faster, but it's not clear how well this extends to other crates with potentially more functions and fewer types. The more complex things get, the more important it is to gather a good set of data to test with (this is arguably more important than the actual benchmarking ifrastructure right now). Rationale and alternatives -------------------------- Throwing a bloom filter in front makes it faster. More than that, it tries to take a tactic where the system can not only check for potential matches, but also gets an accurate distance function without needing to do unification. That way it can skip unification even on items that have the needed elems, as long as they have more items than the currently found maximum. If I didn't want to be able to cheaply do set operations on the fingerprint, a [cuckoo filter] is supposed to have better performance. But the nice bit-banging set intersection doesn't work AFAIK. I also looked into [minhashing], but since it's actually an unbiased estimate of the similarity coefficient, I'm not sure how it could be used to skip unification (I wouldn't know if the estimate was too low or too high). This function actually uses the number of distinct items as its "distance function." This should give the same results that it would have gotten from a Jaccard Distance $1-\frac{|F\cap{}Q|}{|F\cup{}Q|}$, while being cheaper to compute. This is because: * The function $F$ must be a superset of the query $Q$, so their union is just $F$ and the intersection is $Q$ and it can be reduced to $1-\frac{|Q|}{|F|}. * There are no magic thresholds. These values are only being used to compare against each other while sorting (and, if 200 results are found, to compare with the maximum match). This means we only care if one value is bigger than the other, not what it's actual value is, and since $Q$ is the same for everything, it can be safely left out, reducing the formula to $1-\frac{1}{|F|} = \frac{|F|}{|F|}-\frac{1}{|F|} = |F|-1$. And, since the values are only being compared with each other, $|F|$ is fine. Prior art --------- This is significantly different from how Hoogle does it. It doesn't account for order, and it has no special account for nesting, though `Box<t>` is still two items, while `t` is only one. This should give the same results that it would have gotten from a Jaccard Distance $1-\frac{|A\cap{}B|}{|A\cup{}B|}$, while being cheaper to compute. Unresolved questions -------------------- `[]` and `()`, the slice/array and tuple/union operators, are ignored while building the signature for the query. This is because they match more than one thing, making them ambiguous. Unfortunately, this also makes them a performance cliff. Is this likely to be a problem? Right now, the system just stashes the type distance into the same field that levenshtein distance normally goes in. This means exact query matches show up on top (for example, if you have a function like `fn nothing(a: Nothing, b: i32)`, then searching for `nothing` will show it on top even if there's another function with `fn bar(x: Nothing)` that's technically a closer match in type signature. Future possibilities -------------------- It should be possible to adopt more sorting criteria to act as a tie breaker, which could be determined during unification. [cuckoo filter]: https://en.wikipedia.org/wiki/Cuckoo_filter [minhashing]: https://en.wikipedia.org/wiki/MinHash
1 parent a5b2de4 commit c56f632

File tree

9 files changed

+315
-72
lines changed

9 files changed

+315
-72
lines changed

src/librustdoc/html/static/js/externs.js

+2-1
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ function initSearch(searchIndex){}
1414
* pathWithoutLast: Array<string>,
1515
* pathLast: string,
1616
* generics: Array<QueryElement>,
17-
* bindings: Map<(string|integer), Array<QueryElement>>,
17+
* bindings: Map<integer, Array<QueryElement>>,
1818
* }}
1919
*/
2020
let QueryElement;
@@ -42,6 +42,7 @@ let ParserState;
4242
* totalElems: number,
4343
* literalSearch: boolean,
4444
* corrections: Array<{from: string, to: integer}>,
45+
* typeFingerprint: Uint32Array,
4546
* }}
4647
*/
4748
let ParsedQuery;

src/librustdoc/html/static/js/search.js

+197-57
Original file line numberDiff line numberDiff line change
@@ -238,6 +238,10 @@ function initSearch(rawSearchIndex) {
238238
* @type {Array<Row>}
239239
*/
240240
let searchIndex;
241+
/**
242+
* @type {Uint32Array}
243+
*/
244+
let functionTypeFingerprint;
241245
let currentResults;
242246
/**
243247
* Map from normalized type names to integers. Used to make type search
@@ -1049,6 +1053,8 @@ function initSearch(rawSearchIndex) {
10491053
correction: null,
10501054
proposeCorrectionFrom: null,
10511055
proposeCorrectionTo: null,
1056+
// bloom filter build from type ids
1057+
typeFingerprint: new Uint32Array(4),
10521058
};
10531059
}
10541060

@@ -1144,7 +1150,6 @@ function initSearch(rawSearchIndex) {
11441150
query.error = err;
11451151
return query;
11461152
}
1147-
11481153
if (!query.literalSearch) {
11491154
// If there is more than one element in the query, we switch to literalSearch in any
11501155
// case.
@@ -1952,8 +1957,7 @@ function initSearch(rawSearchIndex) {
19521957
* @param {integer} path_dist
19531958
*/
19541959
function addIntoResults(results, fullId, id, index, dist, path_dist, maxEditDistance) {
1955-
const inBounds = dist <= maxEditDistance || index !== -1;
1956-
if (dist === 0 || (!parsedQuery.literalSearch && inBounds)) {
1960+
if (dist <= maxEditDistance || index !== -1) {
19571961
if (results.has(fullId)) {
19581962
const result = results.get(fullId);
19591963
if (result.dontValidate || result.dist <= dist) {
@@ -2001,17 +2005,37 @@ function initSearch(rawSearchIndex) {
20012005
const fullId = row.id;
20022006
const searchWord = searchWords[pos];
20032007

2004-
const in_args = row.type && row.type.inputs
2005-
&& checkIfInList(row.type.inputs, elem, row.type.where_clause);
2006-
if (in_args) {
2007-
// path_dist is 0 because no parent path information is currently stored
2008-
// in the search index
2009-
addIntoResults(results_in_args, fullId, pos, -1, 0, 0, maxEditDistance);
2010-
}
2011-
const returned = row.type && row.type.output
2012-
&& checkIfInList(row.type.output, elem, row.type.where_clause);
2013-
if (returned) {
2014-
addIntoResults(results_returned, fullId, pos, -1, 0, 0, maxEditDistance);
2008+
// fpDist is a minimum possible type distance, where "type distance" is the number of
2009+
// atoms in the function not present in the query
2010+
const tfpDist = compareTypeFingerprints(
2011+
fullId,
2012+
parsedQuery.typeFingerprint
2013+
);
2014+
if (tfpDist !== false &&
2015+
!(results_in_args.size >= MAX_RESULTS && tfpDist > results_in_args.max_dist)
2016+
) {
2017+
const in_args = row.type && row.type.inputs
2018+
&& checkIfInList(row.type.inputs, elem, row.type.where_clause);
2019+
if (in_args) {
2020+
results_in_args.max_dist = Math.max(results_in_args.max_dist || 0, tfpDist);
2021+
const maxDist = results_in_args.size < MAX_RESULTS ?
2022+
(tfpDist + 1) :
2023+
results_in_args.max_dist;
2024+
addIntoResults(results_in_args, fullId, pos, -1, tfpDist, 0, maxDist);
2025+
}
2026+
}
2027+
if (tfpDist !== false &&
2028+
!(results_returned.size >= MAX_RESULTS && tfpDist > results_returned.max_dist)
2029+
) {
2030+
const returned = row.type && row.type.output
2031+
&& checkIfInList(row.type.output, elem, row.type.where_clause);
2032+
if (returned) {
2033+
results_returned.max_dist = Math.max(results_returned.max_dist || 0, tfpDist);
2034+
const maxDist = results_returned.size < MAX_RESULTS ?
2035+
(tfpDist + 1) :
2036+
results_returned.max_dist;
2037+
addIntoResults(results_returned, fullId, pos, -1, tfpDist, 0, maxDist);
2038+
}
20152039
}
20162040

20172041
if (!typePassesFilter(elem.typeFilter, row.ty)) {
@@ -2070,6 +2094,17 @@ function initSearch(rawSearchIndex) {
20702094
return;
20712095
}
20722096

2097+
const tfpDist = compareTypeFingerprints(
2098+
row.id,
2099+
parsedQuery.typeFingerprint
2100+
);
2101+
if (tfpDist === false) {
2102+
return;
2103+
}
2104+
if (results.size >= MAX_RESULTS && tfpDist > results.max_dist) {
2105+
return;
2106+
}
2107+
20732108
// If the result is too "bad", we return false and it ends this search.
20742109
if (!unifyFunctionTypes(
20752110
row.type.inputs,
@@ -2088,11 +2123,12 @@ function initSearch(rawSearchIndex) {
20882123
return;
20892124
}
20902125

2091-
addIntoResults(results, row.id, pos, 0, 0, 0, Number.MAX_VALUE);
2126+
results.max_dist = Math.max(results.max_dist || 0, tfpDist);
2127+
addIntoResults(results, row.id, pos, 0, tfpDist, 0, Number.MAX_VALUE);
20922128
}
20932129

20942130
function innerRunQuery() {
2095-
let elem, i, nSearchWords, in_returned, row;
2131+
let elem, i, nSearchWords;
20962132

20972133
let queryLen = 0;
20982134
for (const elem of parsedQuery.elems) {
@@ -2206,50 +2242,30 @@ function initSearch(rawSearchIndex) {
22062242
);
22072243
}
22082244

2245+
const fps = new Set();
22092246
for (const elem of parsedQuery.elems) {
22102247
convertNameToId(elem);
2248+
buildFunctionTypeFingerprint(elem, parsedQuery.typeFingerprint, fps);
22112249
}
22122250
for (const elem of parsedQuery.returned) {
22132251
convertNameToId(elem);
2252+
buildFunctionTypeFingerprint(elem, parsedQuery.typeFingerprint, fps);
22142253
}
22152254

2216-
if (parsedQuery.foundElems === 1) {
2217-
if (parsedQuery.elems.length === 1) {
2218-
elem = parsedQuery.elems[0];
2219-
for (i = 0, nSearchWords = searchWords.length; i < nSearchWords; ++i) {
2220-
// It means we want to check for this element everywhere (in names, args and
2221-
// returned).
2222-
handleSingleArg(
2223-
searchIndex[i],
2224-
i,
2225-
elem,
2226-
results_others,
2227-
results_in_args,
2228-
results_returned,
2229-
maxEditDistance
2230-
);
2231-
}
2232-
} else if (parsedQuery.returned.length === 1) {
2233-
// We received one returned argument to check, so looking into returned values.
2234-
elem = parsedQuery.returned[0];
2235-
for (i = 0, nSearchWords = searchWords.length; i < nSearchWords; ++i) {
2236-
row = searchIndex[i];
2237-
in_returned = row.type && unifyFunctionTypes(
2238-
row.type.output,
2239-
parsedQuery.returned,
2240-
row.type.where_clause
2241-
);
2242-
if (in_returned) {
2243-
addIntoResults(
2244-
results_others,
2245-
row.id,
2246-
i,
2247-
-1,
2248-
0,
2249-
Number.MAX_VALUE
2250-
);
2251-
}
2252-
}
2255+
if (parsedQuery.foundElems === 1 && parsedQuery.returned.length === 0) {
2256+
elem = parsedQuery.elems[0];
2257+
for (i = 0, nSearchWords = searchWords.length; i < nSearchWords; ++i) {
2258+
// It means we want to check for this element everywhere (in names, args and
2259+
// returned).
2260+
handleSingleArg(
2261+
searchIndex[i],
2262+
i,
2263+
elem,
2264+
results_others,
2265+
results_in_args,
2266+
results_returned,
2267+
maxEditDistance
2268+
);
22532269
}
22542270
} else if (parsedQuery.foundElems > 0) {
22552271
for (i = 0, nSearchWords = searchWords.length; i < nSearchWords; ++i) {
@@ -2796,6 +2812,95 @@ ${item.displayPath}<span class="${type}">${name}</span>\
27962812
};
27972813
}
27982814

2815+
/**
2816+
* Type fingerprints allow fast, approximate matching of types.
2817+
*
2818+
* This algo creates a compact representation of the type set using a Bloom filter.
2819+
* This fingerprint is used three ways:
2820+
*
2821+
* - It accelerates the matching algorithm by checking the function fingerprint against the
2822+
* query fingerprint. If any bits are set in the query but not in the function, it can't
2823+
* match.
2824+
*
2825+
* - The fourth section has the number of distinct items in the set.
2826+
* This is the distance function, used for filtering and for sorting.
2827+
*
2828+
* [^1]: Distance is the relatively naive metric of counting the number of distinct items in
2829+
* the function that are not present in the query.
2830+
*
2831+
* @param {FunctionType|QueryElement} type - a single type
2832+
* @param {Uint32Array} output - write the fingerprint to this data structure: uses 128 bits
2833+
* @param {Set<number>} fps - Set of distinct items
2834+
*/
2835+
function buildFunctionTypeFingerprint(type, output, fps) {
2836+
2837+
const input = type.id;
2838+
// `[]` doesn't go in the bloom filter because it's special-cased to match things
2839+
// other than itself.
2840+
if (input !== null && input !== typeNameIdOfArrayOrSlice) {
2841+
// https://docs.rs/rustc-hash/1.1.0/src/rustc_hash/lib.rs.html#60
2842+
// Rotate is skipped because we're only doing one cycle anyway.
2843+
const h0 = Math.imul(input, 0x9e3779b9);
2844+
const h1 = Math.imul(479001599 ^ input, 0x9e3779b9);
2845+
const h2 = Math.imul(433494437 ^ input, 0x9e3779b9);
2846+
output[0] |= 1 << (h0 % 32);
2847+
output[1] |= 1 << (h1 % 32);
2848+
output[2] |= 1 << (h2 % 32);
2849+
fps.add(input);
2850+
}
2851+
for (const g of type.generics) {
2852+
buildFunctionTypeFingerprint(g, output, fps);
2853+
}
2854+
const fb = {
2855+
id: null,
2856+
ty: 0,
2857+
generics: [],
2858+
bindings: new Map(),
2859+
};
2860+
for (const [k, v] of type.bindings.entries()) {
2861+
fb.id = k;
2862+
fb.generics = v;
2863+
buildFunctionTypeFingerprint(fb, output, fps);
2864+
}
2865+
output[3] = fps.size;
2866+
}
2867+
2868+
/**
2869+
* Compare the query fingerprint with the function fingerprint.
2870+
*
2871+
* @param {{number}} fullId - The function
2872+
* @param {{Uint32Array}} queryFingerprint - The query
2873+
* @returns {number|false} - False if non-match, number if distance
2874+
* This function might return 0!
2875+
*/
2876+
function compareTypeFingerprints(fullId, queryFingerprint) {
2877+
2878+
const fh0 = functionTypeFingerprint[fullId * 4];
2879+
const fh1 = functionTypeFingerprint[(fullId * 4) + 1];
2880+
const fh2 = functionTypeFingerprint[(fullId * 4) + 2];
2881+
const qh0 = queryFingerprint[0];
2882+
const qh1 = queryFingerprint[1];
2883+
const qh2 = queryFingerprint[2];
2884+
// Approximate set intersection with bloom filters.
2885+
// This can be larger than reality, not smaller, because hashes have
2886+
// the property that if they've got the same value, they hash to the
2887+
// same thing. False positives exist, but not false negatives.
2888+
const [in0, in1, in2, in3] = [fh0 & qh0, fh1 & qh1, fh2 & qh2];
2889+
// Approximate the set of items in the query but not the function.
2890+
// This might be smaller than reality, but cannot be bigger.
2891+
//
2892+
// | in_ | qh_ | XOR | Meaning |
2893+
// | --- | --- | --- | ------------------------------------------------ |
2894+
// | 0 | 0 | 0 | Not present |
2895+
// | 1 | 0 | 1 | IMPOSSIBLE because `in_` is `fh_ & qh_` |
2896+
// | 1 | 1 | 0 | If one or both is false positive, false negative |
2897+
// | 0 | 1 | 1 | Since in_ has no false negatives, must be real |
2898+
if ((in0 ^ qh0) || (in1 ^ qh1) || (in2 ^ qh2)) {
2899+
return false;
2900+
}
2901+
return functionTypeFingerprint[(fullId * 4) + 3];
2902+
}
2903+
27992904
function buildIndex(rawSearchIndex) {
28002905
searchIndex = [];
28012906
/**
@@ -2815,6 +2920,22 @@ ${item.displayPath}<span class="${type}">${name}</span>\
28152920
typeNameIdOfSlice = buildTypeMapIndex("slice");
28162921
typeNameIdOfArrayOrSlice = buildTypeMapIndex("[]");
28172922

2923+
// Function type fingerprints are 128-bit bloom filters that are used to
2924+
// estimate the distance between function and query.
2925+
// This loop counts the number of items to allocate a fingerprint for.
2926+
for (const crate in rawSearchIndex) {
2927+
if (!hasOwnPropertyRustdoc(rawSearchIndex, crate)) {
2928+
continue;
2929+
}
2930+
// Each item gets an entry in the fingerprint array, and the crate
2931+
// does, too
2932+
id += rawSearchIndex[crate].t.length + 1;
2933+
}
2934+
functionTypeFingerprint = new Uint32Array((id + 1) * 4);
2935+
2936+
// This loop actually generates the search item indexes, including
2937+
// normalized names, type signature objects and fingerprints, and aliases.
2938+
id = 0;
28182939
for (const crate in rawSearchIndex) {
28192940
if (!hasOwnPropertyRustdoc(rawSearchIndex, crate)) {
28202941
continue;
@@ -2964,17 +3085,36 @@ ${item.displayPath}<span class="${type}">${name}</span>\
29643085
}
29653086
searchWords.push(word);
29663087
const path = itemPaths.has(i) ? itemPaths.get(i) : lastPath;
3088+
let type = null;
3089+
if (itemFunctionSearchTypes[i] !== 0) {
3090+
type = buildFunctionSearchType(
3091+
itemFunctionSearchTypes[i],
3092+
lowercasePaths
3093+
);
3094+
if (type) {
3095+
const fp = functionTypeFingerprint.subarray(id * 4, (id + 1) * 4);
3096+
const fps = new Set();
3097+
for (const t of type.inputs) {
3098+
buildFunctionTypeFingerprint(t, fp, fps);
3099+
}
3100+
for (const t of type.output) {
3101+
buildFunctionTypeFingerprint(t, fp, fps);
3102+
}
3103+
for (const w of type.where_clause) {
3104+
for (const t of w) {
3105+
buildFunctionTypeFingerprint(t, fp, fps);
3106+
}
3107+
}
3108+
}
3109+
}
29673110
const row = {
29683111
crate: crate,
29693112
ty: itemTypes.charCodeAt(i) - charA,
29703113
name: itemNames[i],
29713114
path: path,
29723115
desc: itemDescs[i],
29733116
parent: itemParentIdxs[i] > 0 ? paths[itemParentIdxs[i] - 1] : undefined,
2974-
type: buildFunctionSearchType(
2975-
itemFunctionSearchTypes[i],
2976-
lowercasePaths
2977-
),
3117+
type,
29783118
id: id,
29793119
normalizedName: word.indexOf("_") === -1 ? word : word.replace(/_/g, ""),
29803120
deprecated: deprecatedItems.has(i),

tests/rustdoc-js/assoc-type.js

+2-2
Original file line numberDiff line numberDiff line change
@@ -7,16 +7,16 @@ const EXPECTED = [
77
'query': 'iterator<something> -> u32',
88
'correction': null,
99
'others': [
10-
{ 'path': 'assoc_type', 'name': 'my_fn' },
1110
{ 'path': 'assoc_type::my', 'name': 'other_fn' },
11+
{ 'path': 'assoc_type', 'name': 'my_fn' },
1212
],
1313
},
1414
{
1515
'query': 'iterator<something>',
1616
'correction': null,
1717
'in_args': [
18-
{ 'path': 'assoc_type', 'name': 'my_fn' },
1918
{ 'path': 'assoc_type::my', 'name': 'other_fn' },
19+
{ 'path': 'assoc_type', 'name': 'my_fn' },
2020
],
2121
},
2222
// if I write an explicit binding, only it shows up

0 commit comments

Comments
 (0)