Extended search does not provide all matched tokens in matches

[chengyin]

Describe the bug

When the query has multiple tokens, only the matches from the last token is included in returned matches.

Version

5.1.0

Is this a regression?

Yes, although 3.6.2 has a different tokenize API. The results in 3.6.2 would include all matches.

🔬Minimal Reproduction

https://runkit.com/embed/p96hle7l44k6

const Fuse = require("fuse.js");
const objects = [{ title: 'Authors of "Fuse Proof of Concept"' }];

const fuse = new Fuse(objects, {
  includeMatches: true,
  useExtendedSearch: true,
  findAllMatches: true,
  threshold: 0.001,
  distance: 10000000,
  keys: ["title"],
});

const getHighlights = results => results.map((result) =>
  result.matches.map(({ value, indices }) =>
    indices.map(([start, end]) => value.substr(start, end - start + 1))
  )
);

const results = fuse.search("authors fuse proof");
console.log(getHighlights(results));
// → [[["or", "of", "F", "Proof", "of", "o", "p"]]]

(Expecting to see Authors and Fuse in the matches).

[chengyin]

Seems like GH parsed my commit message wrong in our private repo. Reopening here.