feat: integrate Jaccard similarity ranking into autocomplete snippet selection

[markijbema]

Summary

This PR integrates the existing jaccardSimilarity function into the autocomplete snippet selection process to improve context quality by prioritizing snippets that share more symbols with the code around the cursor position.

Problem

The jaccardSimilarity function existed in src/services/continuedev/core/autocomplete/context/ranking/index.ts but was not being used in the actual snippet gathering flow. The getSnippets() function in filtering.ts was using random shuffling for base snippets instead of similarity-based ranking.

Solution

1. Exported jaccardSimilarity from the ranking module to make it available for use elsewhere
2. Created rankSnippetsBySimilarity() function that ranks snippets by their Jaccard similarity to the cursor context
3. Replaced random shuffling with similarity-based ranking for all snippet categories:
   • Clipboard snippets
   • Recently visited ranges
   • Recently edited ranges
   • Diff snippets
   • Recently opened files
   • Base snippets (root path, import definitions, static)

The cursor context is computed using the sliding window around the cursor position (same approach used in the original rankAndOrderSnippets function).

Changes

• src/services/continuedev/core/autocomplete/context/ranking/index.ts: Export jaccardSimilarity function
• src/services/continuedev/core/autocomplete/templating/filtering.ts:
  • Import and use jaccardSimilarity
  • Add rankSnippetsBySimilarity() helper function
  • Replace shuffleArray with similarity-based ranking
• src/services/continuedev/core/autocomplete/templating/__tests__/filtering.spec.ts: New test file with comprehensive tests

Testing

• Added 6 new tests for the ranking behavior
• All existing ranking tests continue to pass
• Tests verify:
  • Snippets are ranked by similarity to cursor context
  • Clipboard snippets are ranked when enabled
  • Snippets already in caret window are filtered out
  • Empty and single snippet arrays are handled correctly
  • Token limits are respected while prioritizing similar snippets

[changeset-bot]

⚠️ No Changeset found

Latest commit: 380612b

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR