"nuget.exe list [search terms]" - behavior changed with nuget.org search improvements of August 2019

[Falconne]

nuget list [search terms] functionality allows searching for NuGet packages from the command line. This worked well upto 5.0.2, but in the version after that the list method ignores the search terms and will go on to list every single package in every package source.

Details about Problem

Product: NuGet.exe

NuGet version: 5.3.0

OS version: Windows 10

Worked before? If so, with which NuGet version:
Yes, last worked in v5.0.2.

Detailed repro steps so we can see the same problem

1. nuget.exe list Microsoft.Extensions.Configuration.Json

2. With v5.0.2 that will return immediately with about a dozen hits on the given package name. With any later version, this will list every package in the public repo, ignoring the search term.

[alphaleonis]

Not sure this has to do with the nuget.exe version. I downloaded a couple of older versions, and the same behavior exists there now. So maybe a problem with the search API at nuget.org?

[Falconne]

@alphaleonis Hmm, possibly, though that's not what I'm getting. On my dev and build machines I can consistently reproduce it after 5.0.2 but 5.0.2 is fine. My build machines automatically fetch an updated nuget.exe at staggered intervals and they all failed after 5.0.2. Since I've locked them down to that version it's been ok.

But yeah, it may be a coincidence that it appears consistent for me.

[alphaleonis]

It's strange... nuget.exe list sdaf only lists one package. But most other searches I try seems to return pretty much all packages from nuget.org. Not sure what's going on, but something isn't working the way it should. And this is when I use version 4.9.4.5839 of nuget.exe.

Accessing the API URL displayed when running in verbose mode also seems to return pretty much all packages, so my bet would still be on the nuget.org site:

Example (skip 1000 first result, get 30):

https://www.nuget.org/api/v2/Search()?$filter=IsLatestVersion&$orderby=Id&searchTerm=%27microsoft.extensions.configuration.json%27&targetFramework=%27%27&includePrerelease=false&$skip=1000&$top=30&semVerLevel=2.0.0

[jairbubbles]

I can confirm this too.

The other side effects is that my job became a lot slower :

[jairbubbles]

nuget list sdaf -source https://api.nuget.org/v3/index.json => sdaf 11.0.0

nuget list Newtonsoft.Json -source https://api.nuget.org/v3/index.json => many many packages

(nuget 4.8.1.5435)

[mparent]

I've also noticed this problem. This seems to be specifically related to nuget.org. Our office's repo doesn't seem to be affected.

[rrelyea]

@joelverhagen / @karann-msft - can you take a quick look at this. Known issue? Plan?

[joelverhagen]

tl;dr: if you want to find a single package on nuget.org using nuget.exe list then you can use nuget.exe list packageid:{your package ID} -source https://api.nuget.org/v3/index.json. This essentially issues a search where the package ID must exactly match something.

Request for people in this thread

Could you respond back to this issue and tell us what purpose you are using nuget.exe list for? It's a very generic command and hasn't been updated much in a long time. If you could help us understand why you are using it then we as a team can come to a better understanding of what you need! Thanks!

Longer explanation

nuget.exe list essentially uses the search API to enumerate packages. The result set will be very similar to https://www.nuget.org/packages but in a different order (sorted by title/ID instead of popularity). The nuget.exe list -h help text hints at this with [search terms] but it's not obvious from the help text that the results are sorted in title/ID order.

When we revamped our search algorithm, we started included a lot more things in the search. Because of the sorting, this means for a package ID with a generic token in it, you will get a lot of matches and it's very likely your specific package is not on the first many pages.

One extreme example which hopefully can illustrate the change ("microsoft" and "aspnet" get tokenized and match some results albeit with a rather low score compared to the proper "microsoft.aspnet.mvc" query):

To know how many results nuget.exe list would return, you can just do the same query in the search box on the website: https://www.nuget.org/packages.

Microsoft.Extensions.Configuration.Json - 20,052 results (a lot of generic tokens in there)
sdaf - 1 result
newtonsoft.json - 2,888 results (basically a lot of things have the term "json" in them)

So seeing many more packages in the results of search is expected. And, as @alphaleonis mentioned, nuget.exe list does a search internally.

There is a search syntax on nuget.org that allows you to do more specific searches. They work in nuget.exe list too. The one you most likely want is packageid:{ID} which means to only return packages with exactly the specified ID. In short, when you use this it will only ever return one or zero results.

I totally agree for nuget list [search terms] this can cause an explosion of paging now. If the packageid:{ID} trick doesn't work for you, could you explain your scenario in detail? Maybe together we can come up with another approach.

Note that when you type nuget.exe list without a search parameter it will enumerate all packages on a feed (up to a limit restricted on the server side). This is not a very useful behavior but this works the same before and after our search changes on nuget.org.

[rrelyea]

Awesome. Thanks Joel.
All- let us know if you have any concerns. Closing.

[matthewjones555]

Thanks @joelverhagen.

If the packageid:{ID} trick doesn't work for you, could you explain your scenario in detail? Maybe together we can come up with another approach.

The packageid trick works fine unless you're using multiple sources which include a v2 feed, or a file system source, in which case it doesn't work. It would be nice to have some kind of -packageid switch available directly in the list command, so we don't have to use api specific syntax.

I'm trying to use nuget list to get a list of available versions for a specific package. This needs aggregating from all sources on the machine, in the same way that the VS integration does it. These package sources are as follows, a nuget.org v3 api, a nuget gallery v2 api, and a file system package feed.

Please let me know if you need more details.

[joelverhagen]

@matthewjones555, this helps a lot, thanks!.

I think for the short term the best option for the mutli-feed environment is to run multiple invocations of nuget.exe list:

# one for nuget.org
nuget.exe list packageid:{ID} -Source https://api.nuget.org/v3/index.json

# one for everything else
nuget.exe list {ID} -Source C:\my\local\feed -Source http://myorg-gallery/api/v2

This is not elegant but perhaps it's a workaround you can use for now so I thought I would call it out.

It would be nice to have some kind of -packageid switch available directly in the list command, so we don't have to use api specific syntax.

Yes, this sounds like a good idea. There is an documented API today for looking up individual packages on V3 feed:
https://docs.microsoft.com/en-us/nuget/api/registration-base-url-resource

This has a V2 API equivalent (/api/v2/FindPackagesById()) and a file-based feed equivalent (directory enumeration).

It would just be a matter of lighting up this option in nuget.exe list (or perhaps a new top level command) and calling this API instead of the search resource. In NuGet client code, there is one resource that has an implementation on V2, V3, and file-based sources that could handle this case very easily:
https://github.com/NuGet/NuGet.Client/blob/ca008ff53cd941d31cba2052c1fcf4c31770b294/src/NuGet.Core/NuGet.Protocol/Resources/PackageMetadataResource.cs#L11

One ongoing effort is to put new experiences in the .NET CLI (in addition or in lieu of nuget.exe) so it's not clear to me where a new switch should go moving forward. My expertise is on the nuget.org server side so I'll defer to people that focus on NuGet client:

@rrelyea, @aortiz-msft - should we re-open this issue to track a different issue for command-line based package ID lookup?

[LostBeard]

Not sure why the issue is closed and Microsoft is closing issues that mention this as if it is fixed. 2023-11-15 and I still cannot search for a package like "OpenCvSharp4.runtime" in Visual Studio 17.9.0 Preview 1.0 without getting every single package
that contains "runtime" or "opencvsharp4" as a result instead of packages that have that exact string.

This seems to be an obvious bug that clearly still exists. Is it being tracked somewhere else with intentions to fix?

My issue is exactly the one reported on this Visual Studio Developer Community which got marked as a duplicate for this issue and was closed (locked even.)

When looking for the exact package name “Microsoft.VisualStudio.TestPlatform.TestFramework.Extensions” it seems to split the string on “.” and searches on the individual results rather than an exact match. The results are too large which is why I was using search to begin with.

That leaves me to scroll through lots of packages that are clearly not matches.