Machine readable output for dotnet list package

[adamdriscoll]

Details about Problem

It is difficult to parse the output of dotnet list package. We have requirements in our organization to audit packages that we are using in our projects. Without having a good way to parse the output of this command, it makes it difficult to structure the list of packages in a way that can be fed into another auditing system.

Running the following command produces the following output:

dotnet list package --include-transitive

Project 'UniversalDashboard' has the following package references
   [net472]:
   Top-level Package                                           Requested          Resolved
   > Microsoft.AspNetCore                                      2.1.3              2.1.3
   > Microsoft.AspNetCore.Authentication.Cookies               2.1.2              2.1.2
   > Microsoft.AspNetCore.Authentication.Facebook              2.1.2              2.1.2
   > Microsoft.AspNetCore.Authentication.Google                2.1.2              2.1.2
   > Microsoft.AspNetCore.Authentication.JwtBearer             2.1.2              2.1.2
   > Microsoft.AspNetCore.Authentication.MicrosoftAccount      2.1.2              2.1.2
   > Microsoft.AspNetCore.Authentication.Twitter               2.1.2              2.1.2
   > Microsoft.AspNetCore.Mvc                                  2.1.2              2.1.2
   > Microsoft.AspNetCore.ResponseCompression                  2.1.1              2.1.1
   > Microsoft.AspNetCore.Routing                              2.1.1              2.1.1
   > Microsoft.AspNetCore.Server.Kestrel.Https                 2.1.3              2.1.3
   > Microsoft.AspNetCore.Server.Kestrel.Transport.Libuv       2.1.3              2.1.3
   > Microsoft.AspNetCore.Session                              2.1.1              2.1.1
   > Microsoft.AspNetCore.SignalR                              1.0.3              1.0.3
   > Microsoft.AspNetCore.StaticFiles                          2.1.1              2.1.1
   > Microsoft.AspNetCore.WebSockets                           2.1.1              2.1.1
   > Microsoft.DotNet.PlatformAbstractions                     2.1.0              2.1.0
   > Microsoft.Extensions.Caching.Memory                       2.1.2              2.1.2
   > Microsoft.Extensions.Logging.Debug                        2.1.1              2.1.1
   > Microsoft.Extensions.Logging.Filter                       1.1.2              1.1.2
   > Newtonsoft.Json                                           11.0.2             11.0.2
   > NLog                                                      4.5.8              4.5.8
   > NLog.Web.AspNetCore                                       4.6.0              4.6.0
   > PowerShellStandard.Library                                5.1.0-preview-06   5.1.0-preview-06
   > Quartz                                                    3.0.6              3.0.6
   > System.Net.NameResolution                                 4.3.0              4.3.0
   > System.Reflection.Emit                                    4.3.0              4.3.0
   > System.Runtime.Loader                                     4.3.0              4.3.0
   > System.Security.Principal                                 4.3.0              4.3.0

   Transitive Package                                                Resolved
   > Libuv                                                           1.10.0
   > Microsoft.AspNetCore.Antiforgery                                2.1.1
   > Microsoft.AspNetCore.Authentication                             2.1.2
   > Microsoft.AspNetCore.Authentication.Abstractions                2.1.1
   > Microsoft.AspNetCore.Authentication.Core                        2.1.1
   > Microsoft.AspNetCore.Authentication.OAuth                       2.1.2
   > Microsoft.AspNetCore.Authorization                              2.1.1
   > Microsoft.AspNetCore.Authorization.Policy                       2.1.1
   > Microsoft.AspNetCore.Connections.Abstractions                   2.1.3
   > Microsoft.AspNetCore.Cors                                       2.1.1
   > Microsoft.AspNetCore.Cryptography.Internal                      2.1.1
   > Microsoft.AspNetCore.DataProtection                             2.1.1
   > Microsoft.AspNetCore.DataProtection.Abstractions                2.1.1
   > Microsoft.AspNetCore.Diagnostics                                2.1.1
   > Microsoft.AspNetCore.Diagnostics.Abstractions                   2.1.1
   > Microsoft.AspNetCore.HostFiltering                              2.1.1
   > Microsoft.AspNetCore.Hosting                                    2.1.1
   > Microsoft.AspNetCore.Hosting.Abstractions                       2.1.1
   > Microsoft.AspNetCore.Hosting.Server.Abstractions                2.1.1
   > Microsoft.AspNetCore.Html.Abstractions                          2.1.1
   > Microsoft.AspNetCore.Http                                       2.1.1
   > Microsoft.AspNetCore.Http.Abstractions                          2.1.1
   > Microsoft.AspNetCore.Http.Connections                           1.0.3
   > Microsoft.AspNetCore.Http.Connections.Common                    1.0.3
   > Microsoft.AspNetCore.Http.Extensions                            2.1.1
   > Microsoft.AspNetCore.Http.Features                              2.1.1
   > Microsoft.AspNetCore.HttpOverrides                              2.1.1
   > Microsoft.AspNetCore.JsonPatch                                  2.1.1
   > Microsoft.AspNetCore.Localization                               2.1.1
   > Microsoft.AspNetCore.Mvc.Abstractions                           2.1.2
   > Microsoft.AspNetCore.Mvc.ApiExplorer                            2.1.2
   > Microsoft.AspNetCore.Mvc.Core                                   2.1.2
   > Microsoft.AspNetCore.Mvc.Cors                                   2.1.2
   > Microsoft.AspNetCore.Mvc.DataAnnotations                        2.1.2
   > Microsoft.AspNetCore.Mvc.Formatters.Json                        2.1.2
   > Microsoft.AspNetCore.Mvc.Localization                           2.1.2
   > Microsoft.AspNetCore.Mvc.Razor                                  2.1.2
   > Microsoft.AspNetCore.Mvc.Razor.Extensions                       2.1.1
   > Microsoft.AspNetCore.Mvc.RazorPages                             2.1.2
   > Microsoft.AspNetCore.Mvc.TagHelpers                             2.1.2
   > Microsoft.AspNetCore.Mvc.ViewFeatures                           2.1.2
   > Microsoft.AspNetCore.Razor                                      2.1.1
   > Microsoft.AspNetCore.Razor.Design                               2.1.1
   > Microsoft.AspNetCore.Razor.Language                             2.1.1
   > Microsoft.AspNetCore.Razor.Runtime                              2.1.1
   > Microsoft.AspNetCore.ResponseCaching.Abstractions               2.1.1
   > Microsoft.AspNetCore.Routing.Abstractions                       2.1.1
   > Microsoft.AspNetCore.Server.IISIntegration                      2.1.1
   > Microsoft.AspNetCore.Server.Kestrel                             2.1.3
   > Microsoft.AspNetCore.Server.Kestrel.Core                        2.1.3
   > Microsoft.AspNetCore.Server.Kestrel.Transport.Abstractions      2.1.3
   > Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets           2.1.3
   > Microsoft.AspNetCore.SignalR.Common                             1.0.3
   > Microsoft.AspNetCore.SignalR.Core                               1.0.3
   > Microsoft.AspNetCore.SignalR.Protocols.Json                     1.0.3
   > Microsoft.AspNetCore.WebUtilities                               2.1.1
   > Microsoft.CodeAnalysis.Analyzers                                1.1.0
   > Microsoft.CodeAnalysis.Common                                   2.8.0
   > Microsoft.CodeAnalysis.CSharp                                   2.8.0
   > Microsoft.CodeAnalysis.Razor                                    2.1.1
   > Microsoft.CSharp                                                4.5.0
   > Microsoft.DiaSymReader.Native                                   1.7.0

The problem with this type of output is that it is difficult to parse. npm provides two options that make this easier when listing packages. They include both a -parsable and a -json parameter. I think dotnet list package should provide something similar.

I think that dotnet list package should do something similar.

F:\universal-dashboard\src\UniversalDashboard>dotnet --version
2.2.102

I'm happy to work on a PR for this. I've found the responsible code and could implement a "parse-friendly" version but wanted feedback from the community\team before doing anything.

https://github.com/NuGet/NuGet.Client/blob/dev/src/NuGet.Core/NuGet.CommandLine.XPlat/Utility/ProjectPackagesPrintUtility.cs

---

Update 12/10/2021:
After diving into this issue, it's quite large issue, it looks none of dotnet command implemented it, this is 1st time. Let's divide and conquer this issue, here're sub issues:

• Machine readable json output for dotnet list package: Create spec for machine readable json output for "dotnet list package" #11449
  Spec proposal PR: Spec for machine readable json output for dotnet list package #11446

[nkolev92]

Hey @adamdriscoll
Thanks for opening this issue.
It is something we considered initially, but wasn't added for V1.

@rrelyea @anangaur to chip in on how they'd want this done.

I personally prefer a json option.

[anangaur]

Json seems fine as most of NuGet generated artifacts are json too.

@adamdriscoll , If you could start with a one pager that includes the proposal on output i.e. the structure of the data and how sample output would look like with different options, that can be a good start. We can review the proposal and then suggest/make changes as required. Since this is not on our priority list, it would be difficult for us to propose a full spec on this in near term.

[ignatandrei]

json will be good to have.

[xavierdecoster]

Having a --json option for dotnet list package would be very convenient indeed, especially when --outdated and --deprecated come into the picture. Making the output machine-readable unlocks additional tooling and automation scenarios.

Based on today's (and planned) console output, here's a suggestion.
Would love feedback/input and make sure we understand what data and format would be desired for this JSON output.

outdated.json

Command: dotnet list package --outdated --include-transitive --json

{
  "outdated": [
    {
      "framework": "netcoreapp2.0",
      "topLevelPackages": [
        {
          "id": "NuGet.Common",
          "requestedVersion": "4.8.0-preview1.5146",
          "resolvedVersion": "4.8.0-preview1.5146",
          "latestVersion": "4.9.1-zlocal.52990"
        }
      ],
      "transitivePackages": [
        {
          "id": "NuGet.Frameworks",
          "requestedVersion": "4.8.0-preview3.5278",
          "latestVersion": "4.9.1-zlocal.52990"
        }
      ]
    }
  ]
}

deprecated.json

Command: dotnet list package --deprecated --json

{
  "deprecated": [
    {
      "framework": "netcoreapp2.0",
      "topLevelPackages": [
        {
          "id": "My.Legacy.Package",
          "resolvedVersion": "2.0.0",
          "reasons": ["Legacy"],
          "alternativePackage": {
            "id": "My.Awesome.Package",
            "versionRange": "[3.0.0,)"
          }
        },
        {
          "id": "My.Buggy.Package",
          "resolvedVersion": "1.1.0",
          "reasons": ["Critical Bugs"],
          "alternativePackage": {
            "id": "My.NotBuggy.Package",
            "versionRange": "*"
          }
        },
        {
          "id": "My.CompletelyBroken.Package",
          "resolvedVersion": "0.9.0",
          "reasons": ["Legacy", "Critical Bugs"],
          "alternativePackage": null
        }
      ],
      "transitivePackages": null
    }
  ]
}

installed-packages.json

Command: dotnet list package --json

This would be very similar to the outdated.json example above but simply listing all installed packages.

[anangaur]

Should it be null or empty object list for no transitivePackages?

For multi-targetting, there is a trade-off (size of the file) between listing the packages separately for each framework vs. having this info as an attribute for each package. I like the current approach better though.

[xavierdecoster]

Should it be null or empty object list for no transitivePackages?

No strong opinion :)

For multi-targetting, there is a trade-off (size of the file) between listing the packages separately for each framework vs. having this info as an attribute for each package. I like the current approach better though.

The current model happens to correspond with the in-memory model used for console printing, which is rather convenient.

[anangaur]

Announcement for the spec review: NuGet/Announcements#35
Spec: Machine readable output for dotnet list package

[nkolev92]

@anangaur @xavierdecoster Can you please put the spec in a PR to the https://github.com/NuGet/Client.Engineering repo?

It's very difficult to leave comments on a spec that's in the wiki.

[zivkan]

I agree with moving the spec to a PR so feedback can be provided more easily.

In addition, the json format needs to take into account that it's possible to run dotnet list package on a solution, in which case it needs to show each project separately. The proposed output only looks suitable for single projects.

[anangaur]

Agree with having the specs as files than wiki. I will try to find a place, for next specs. A common place than having client vs. server places.

[nkolev92]

A couple of comments from a first look (please PR it!).

• TransitivePackage should not be null. Just do an empty array.
• How about framework as the key rather than having a framework kvp?
• How do we handle multiple projects?

[xavierdecoster]

Agree with having the specs as files than wiki. I will try to find a place, for next specs. A common place than having client vs. server places.

A couple of comments from a first look (please PR it!).

Thanks, will update. Where should the spec files go (and the PR)? :)

[nkolev92]

Start with https://github.com/NuGet/Client.Engineering, unless @anangaur is ready with a new repo? :)

[anangaur]

Home/wiki, for now :)

[nkolev92]

Another dup in #9524

[Vandersteen]

Is there any progress on this issue ?

[erdembayar]

Is there any progress on this issue ?

I'm working on implementation, since this's the 1st ever json output for .net cli tooling still discovering new thing.
I need community input for below spec revision: #11983

[dburriss]

Awesome to see the progress on this @erdembayar !
Will this be assigned to a milestone at any point? I have a usecase and I was going to parse the text but if it is feasible to wait for this to ship I would prefer that.

[erdembayar]

Awesome to see the progress on this @erdembayar Erick Yondon FTE ! Will this be assigned to a milestone at any point? I have a usecase and I was going to parse the text but if it is feasible to wait for this to ship I would prefer that.

Thank you for your interest, I just merged the PR for json output, in that PR I refactored the coupled code to MVC style code so next time someone can add yaml,xml outputs with little effort.
Most likely this would be in NuGet 6.5 (VS17.5), I still need to complete dotnet/sdk#28564 by next week.

[kaylumah]

Very cool :) hoop to see this soon :)

[erdembayar]

If anyone want to test out, it's already in public nightly build: https://github.com/dotnet/installer#table
We'll update the doc soon: https://github.com/NuGet/docs.microsoft.com-nuget/issues/2922
In case of there is any issue with generating json output then it will return 1 instead of 0 if you're using it for CI pipeline automation, so you can check what is going on.

[erdembayar]

Alternatively, you can install latest binary from here .\dotnet-install.ps1 -Channel 7.0.2xx -Quality daily

[Postlagerkarte]

@erdembayar Thanks for the great work! However, I am a bit confused. The documentation has been updated a few days ago (https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-list-package) - so I was assuming that this feature is ready to use in production. Sadly the latest .NET 7.0.2 from January 10, 2023 contains .NET SDK 7.0.102 and is missing the feature. I know that I can manually install the latest build but I want to use it within an Azure Pipeline. Can you advise?

[erdembayar]

@erdembayar Thanks for the great work! However, I am a bit confused. The documentation has been updated a few days ago (https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-list-package) - so I was assuming that this feature is ready to use in production. Sadly the latest .NET 7.0.2 from January 10, 2023 contains .NET SDK 7.0.102 and is missing the feature. I know that I can manually install the latest build but I want to use it within an Azure Pipeline. Can you advise?

Here is the PR#32855 for doc change. I explicitly asked not to merge until it's GA, but someone merged it, considering it was for main branch not our live branch of docs, I didn't ask to revert.
But it looks like they moved all changes from main to live 2 days ago even though not all new features are GA yet. I'll ask why it happened, by the way this change is in NET SDK 7.0.2xxx, not in 7.0.102.
Sorry for the confusion, most likely we just revert the change until NET SDK 7.0.2xxx is GA.

[Postlagerkarte]

Thanks for clarifying! Any idea when NET SDK 7.0.2xxx will be GA?

[erdembayar]

Thanks for clarifying! Any idea when NET SDK 7.0.2xxx will be GA?

Sorry, I couldn't tell exactly, but looking at https://dotnet.microsoft.com/en-us/download/dotnet/7.0 release dates, it usually happens every 1 month, so I estimate sometime in Feb,2023.

[erdembayar]

Thanks for clarifying! Any idea when NET SDK 7.0.2xxx will be GA?

Sorry, I couldn't tell exactly, but looking at https://dotnet.microsoft.com/en-us/download/dotnet/7.0 release dates, it usually happens every 1 month, so I estimate sometime in Feb,2023.

This feature is GA now with .NET SDK 7.0.200 🥳, please download from https://dotnet.microsoft.com/en-us/download/dotnet/7.0