Marketplace search improvements

[isidorn]

Hi VS Code PM here 👋

I understand that there are already search issues opened in this repository, however I wanted to have one super issue which has all the items linked.
Improving search is the number one thing we would want for the Marketplace team to improve 💯

If needed we can provide many more examples where search can be improved, but here's a list of issues to start. These examples are mostly from the Extensions view in VS Code which uses Marketplace search API.

• Advanced extension search (for example excluding certain words) #20
• Search results for specific language debugger are very unspecific #68
• Search relevance doesn't seem great #71
• Microsoft Team needs to Clean Up The VS-Code Extensions Market-Place Please...!! #105
• Shouldn't there be a restriction on displayName length and other fields in the extension manifest?  #107
• VS Code: search for "@category:"debuggers" CoreCLR" does not show C# extension #113
• Extensions filter for publisher should be exact vscode#151192
• Marketplace: searching by an extension's name is practically broken vscode#42890
• When searching for an extension by it's canonical name, a different extension is provided. vscode#73182
• No extension results returned for simple misspelling  vscode#106418
• Poor extension search results when including space in search term vscode#102193
• Extension widget keeps resorting  vscode#29779
• Default to AND rather than OR in marketplace search vscode#51207
• Ranking for "night-owl-vscode-theme" search in marketplace doesn't return obvious results vscode#69810
• Search with multiple words is sub-optimal Marketplace search improvements #154 (comment)
• Category in the search field is not reflected in the category filter in the results page #413
• "spell checker" vs "spell checking" #506

[jeff-hykin]

As one of the last people who commented on one of the main downstream issues 2 years ago: I'm really glad to see this being opened! 👍

[christianshay]

There are special characters (for example "/") that are not needed and not documented that are messing up the tags.

This tag link is bringing up any extension that includes "PL" or "SQL" instead of those tagged "PL/SQL":
https://marketplace.visualstudio.com/search?term=tag%3APL%2FSQL&target=VSCode&category=All%20categories&sortBy=Relevance

[isidorn]

One more example:

[bpasero]

Just to clarify on the above example: even if I put a 100% matching search for the name of the extension, the extension I search for is not on the top:

[prashantvc]

We at VS Marketplace started the work to improve the search relevancy.
As part of this effort, we need your help to curate the search results to define baseline KPI

How can you help?

1. Visit https://ms-extensions.azurewebsites.net/
2. Search for an extension (any of your favourite extensions)
3. Observe the search results, do they make sense?
4. Rearrange results, you can drag and drop, any item need to be on the top? Ranked lower
5. Fill in the feedback, rational behind the rearranging the results
6. Hit submit and repeat for different search terms

[jeff-hykin]

There's lots of examples from previous posts that are still unfixed but sure I'll oblige and redo the search/screenshots

Example 1

Prolog Syntax

Want to see & Justification

(The "Prolog" extension being at the top is fine)

1. My C++ syntax extension shouldn't be the 2nd option
2. A theme called "syntax" shouldn't be the 3rd option
3. My "Better Prolog Syntax" extension should be above them considering it has:

• a title with 100% of the search query
• keywords that match 100% of the search query
• been updated more recently
• and isn't being drug-down by missing or low ratings (5/5 stars)

Me wanting the the "Prolog Language" extension higher needs more justification though:

Justification/Expected Feature: Uncommon Word Importance

If someone searches for "Code Entschuldigung" showing the most-popular extension with "Code" in the name is like showing an the most-popular extension with the word "The" in the name, because "Code" is absurdly common. In contrast, the other term "Entschuldigung" is extremely uncommon. If there is an extension with an extremely uncommon word that matches, then its almost certain thats what the user is looking for. This applies to "Prolog Syntax" search, "Prolog" is uncommon "Syntax" is very common.

The equation for this is simple, its just Bayes Rule:

• How many times does a word appear across all extensions?
  • "Language", "Code", "Syntax" will have really high numbers
  • terms like "Prolog" or "Entschuldigung" will have extremely low
• Inversely weight a word based on its frequency across all extensions

I mean come on this is search-101 methodology

Example 2

Prolog Language

Want to see

The one with arrow above Angular

Justification/Expected Feature: Negative Relevance

• If an extension title contains niche topics (high Bayes score) like "Japanese" and "Angular" that are NOT in the search, that should be penalty.
  => If someone was searching for Angular or Japanese they would type "Angular" or "Japanese" not "Prolog"

Example 3

code eol

(Keywords of the "Code-eol (Line Endings)" extension)

Want to see

I'm going to assume the reader is intelligent and gets the idea.

As a side note I would expect word2vec similarity metrics, but I think VS Code Marketplace needs to start with the basics before I can request that.

[prashantvc]

@jeff-hykin Thanks a lot for taking time to write up a detailed answer. We are doing data driven, and iterative process to improve the search relevancy. Yes, we are starting with basics; you can expect better tokanisation and word2vec will eventually make into the marketpalce

[lramos15]

Here's a gif of searching for Azure Repos. I need to type the full name before it is even in view.

[benibenj]

Here a comparison between the old and new Search results. I entered the exact same name of an extension. In the old version it shows up at second place which is fine (I think it should be first on exact match). In the new version it's placed a lot further down!

Old results:

New results:

[prashantvc]

@benibenj thanks for a lot for the report! We will investigate it.
But generally, how do you like the new search service?

[benibenj]

It seems that smaller extensions (less downloads) are harder to find if they have multiple words in their name (Python C++ Debugger for example).

[hediet]

GitLens should be first when searching for Git Lens:

Version Lens should be below, as it does not even mention git.

[kj0171]

We have done some bug fixes and improvements on the marketplace search service. The changes are live in vscode insider. Do try them out and share feedback :)

Search.Relevancy.Update.Slow.mp4

[hediet]

Nice work on the update!

However, I find this problematic:

The GitLens extension does not show up here.

[isidorn]

@hediet thanks for the feedback, we are looking into this exact case. The issue is that we are doing prefix matching instead of fuzzy matching.

[lramos15]

Maybe an edge case but when searching by publisher i.e. Matt Bierner the top result doesn't make sense and the third result is a bit of a weaker match I'd say.

[isidorn]

@lramos15 good catch, thanks for reporting this.
@SaiKanth007 @kj0171 let's double check this one once we do the improvements we agreed on.

[pcjmfranken]

Simple filtering options for statistical properties such as last updated date, download count, verification status, etc. Preferably multiple of such filters could be applied simultaneously.

This would, for example, allow me to display only extensions updated within the last 30 days, with a download count of at least 2500, and only by verified publishers.

This data is already available to the marketplace search results page (took a quick peek at the devtools network panel), so why not use it?

[kj0171]

Simple filtering options for statistical properties such as last updated date, download count, verification status, etc. Preferably multiple of such filters could be applied simultaneously.

This would, for example, allow me to display only extensions updated within the last 30 days, with a download count of at least 2500, and only by verified publishers.

This data is already available to the marketplace search results page (took a quick peek at the devtools network panel), so why not use it?

Thanks for the feedback. We are aware of this. Will try to address this in near future.

[kj0171]

We have rolled out search enhancements (Details here: #154 (comment)).

Please share feedback and reopen if necessary.

[isidorn]

Thanks @kj0171

To clarify, most of the improvements can be seen in VS Code Insiders.
While we plan to have these improvements in VS Code Stable soon.
Try them out in VS Code Insiders and let us know what you think.

[alefragnani]

Just noticed something on Insiders release that could lead to an impersonating issue discussed here https://vscode-dev-community.slack.com/archives/C74CB59NE/p1673358662096609, based on a post in https://blog.aquasec.com/can-you-trust-your-vscode-extensions

The marketplace search does not respect exact match if you use the publisher field.

I search for an extension, clicked in the publisher name to see other extensions of that publisher (myself), but the new search also returns extensions from authors with similar names.

Based on this comment it seems only well-known publishers are handled, but I would argue that this change should be revisited.

Also, if you misstype the search, using publisher:"microsoft instead (yes, just missing the final double quotes), the result is not good at all. This error, on the other hand, happens on both, Stable and Insiders releases.

Thank you

[kj0171]

Just noticed something on Insiders release that could lead to an impersonating issue discussed here https://vscode-dev-community.slack.com/archives/C74CB59NE/p1673358662096609, based on a post in https://blog.aquasec.com/can-you-trust-your-vscode-extensions

The marketplace search does not respect exact match if you use the publisher field.

I search for an extension, clicked in the publisher name to see other extensions of that publisher (myself), but the new search also returns extensions from authors with similar names.

Based on this comment it seems only well-known publishers are handled, but I would argue that this change should be revisited.

Also, if you misstype the search, using publisher:"microsoft instead (yes, just missing the final double quotes), the result is not good at all. This error, on the other hand, happens on both, Stable and Insiders releases.

Thank you

Thank you for pointing out the issue. For a few reasons, we aren't able to take this fix forward. We are aware of this issue and are working towards this. It will soon be fixed. We will keep you updated.

[prashantvc]

We deployed the search improvements.
Note: This is an ongoing effort, and more improvements will follow. We appriciate if you have any feedback for the team

You will start seeing more relevant results, and support for following features:

• Inclusion and Exclusion (+, -)
• Enforce the search results to include or exclude search terms
• Special Word Support (AND, OR, NOT)
• Overall relevancy improvements

Light Theme - Previous Search
Dark Theme - New and Improved Search

Inclusion and Exclusion (+, -) [#20]

Multi-word search

AND/OR
You can now use AND/OR operation with multiword searches

NOT
Exclude unwanted search terms

[prashantvc]

The marketplace search does not respect exact match if you use the publisher field.

@alefragnani This is quite an unique problem, thanks a lot for reporting. There are two opinions within the team/VS Code users.
We are still figuring out the intent behind the query. Did user mean to apply a filter? Did they try search extensions by publisher name!? It may take awhile conclude this.

The missing double quote can be handled better, it's on our list to fix it.

[prashantvc]

Hey All,

We have deployed (7th Feb) number of changes to the VS Code Insiders improving relevancy, especially for multi-word searches and overrall search experience. The improvements will make their way into VS Code Stable in couple of weeks

Please give it try and let us know what you think. We will continue to work on improving search in VS Code as well as in the Marketplace; your continued support and feedback will help us make it better for the community.

Thank you all for participating in the discusstion. Please feel free to leave comments or contact me directly, we can chat about ideas and possible improvents (Booking Link)

I am closing this issue, and continue discussion in the open issues listed in the description

[isidorn]

I created this follow up issue to make sure @alefragnani publisher bug is still captured
#580

@SaiKanth007 mentioned to me that this should be fixed end of March.

[gerroon]

This is not working for me. Is this implemented?

@enabled NOT @category:"themes"