Display progress bar on "interactive" (not CI) command line restore

[natemcmaster]

From @natemcmaster on January 20, 2017 17:33

Steps to reproduce

Get slow internet.
Clear nuget caches.
dotnet new.
Add dependencies to lots of packages not in the first-run cache warmup. This can be done in a simple app by adding a bunch of rids to <RuntimeIdentifiers>.
dotnet restore.

Expected behavior

Show some sign of progress while downloading from the network. A spinner would be great, but even just confirmation that restore has begun would be helpful.

Actual behavior

No console output appears until NuGet is almost done with the restore. In the screenshot below, restore took over a minute (85 seconds). There was not console output until about 75 seconds after restore began

Environment data

dotnet --info output: using 1.0.0-rc3

cc @shanselman @blackdwarf

Copied from original issue: dotnet/cli#5406

[emgarten]

Restore previously solved this by writing out GET messages to let the user know what activity was taking place, however this is now hidden by default which causes users to once again think that NuGet is hung.

Previous issue: https://github.com/dotnet/cli/issues/1623

The logging for dotnet restore is done by an MSBuild task. I don't know if it is possible to create a spinner by logging through the MSBuild logger.

[blackdwarf]

Spinners will not work for CLI scenarios, however ASCII progress bars would. 😄 @natemcmaster do you see enough info if you pass in -v normal?

[shanselman]

@blackdwarf I see spinners all the time. https://github.com/sindresorhus/cli-spinners Or at the least, yes, progress like NPM.

[blackdwarf]

@shanselman that was a joke on my part. But, anyway, the main point is, does changing the verbosity influence this?

[shanselman]

I would have to test again. Point is, under fairly typical networking conditions (this was 3Mbs, consistently) we still saw a LONG pause with no feedback. You gotta show something. ...... dots....something.

[shanselman]

Just checked it @blackdwarf. I have 35Mbs here. Did -v but still waited around for a WHILE with no feedback.

[rrelyea]

can we log a dot, w/ no "\n", for each get? or something like that?

[blackdwarf]

The fact that normal verbosity doesn't influence this is pretty bad. @rrelyea back in the day, nuget client used to have a verbosity setting for this. Did we remove that?

In lack of that, I think that any output would be good. We could, I think, do a progress bar? Unless there is some limitation in the MSBuild loggers?

[emgarten]

nuget client used to have a verbosity setting for this. Did we remove that?

The verbosity level is still there. dotnet restore is explicitly calling msbuild /t:Restore with a less verbose level.

A progress bar is difficult because restore downloads/discovers packages as it walks to be as efficient as possible, it doesn't know if there will be zero or a hundred packages downloaded until the end.

Restore should have a great progress experience from the console, I'm just not sure what is possible given the way restore works and the limitations of the MSBuild logger.

[blackdwarf]

@emgarten I see...is there a mapping between restore verbosity levels and MSBuild ones?

OK, so nix the progress bar idea. A series of dots as @rrelyea suggested would be good then I guess?

[emgarten]

Dots could work. In the restore task NuGet knows the msbuild verbosity. It could convert lower level messages that it isn't going to display into dots.

[rrelyea]

feedback from @DoRonMotter

• keep diag output a superset of normal build output...so log parsers don't get confused.
• pushing to 4.0.1 as this doesn't currently meet the bar.

[shanselman]

I'd also point out that "it's hard to do async progress bars" isn't really fair. Docker and npm and Yarn have nice ones.

[ghost]

A docker like progress bar with transmission rate / total size is actually quite useful. Here is a sample code to make a simple spinner https://blogs.msdn.microsoft.com/brada/2005/06/11/showing-progress-in-a-console-windows/.

that or windows 98 drumsticks

[gbaydin]

Hello, I was wondering whether there has been any progress in adding a simple progress bar to dotnet restore. We're currently working on machine learning code (DiffSharp, TorchSharp) that depend on very large nuget packages (some GBs) that hold CUDA binaries, for example the following package and its dependencies:

https://www.nuget.org/packages/libtorch-cuda-10.2/

and the user experience of dotnet restore is not great because it "hangs" for long periods of time (say, half an hour in my average broadband home connection in the UK) without showing any indication of what is going on. Similar workflows in other ecosystems (say, Python pip, conda etc., or Docker) show progress information during package download showing the progress for the file currently being downloaded, download speed, ETA, etc. I think dotnet restore really needs such basic progress information in the CLI out of the box with the default settings.

I don't know if this is the correct project to report this problem (this existing issue seemed relevant). Please let me know if I should be opening an issue somewhere else, for example https://github.com/dotnet/sdk

Also tagging @dsyme @interesaaat

I'm using dotnet CLI tools on linux. This is the output of dotnet --info if it helps:

.NET Core SDK (reflecting any global.json):
 Version:   3.1.201
 Commit:    b1768b4ae7

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  20.04
 OS Platform: Linux
 RID:         linux-x64
 Base Path:   /usr/share/dotnet/sdk/3.1.201/

Host (useful for support):
  Version: 3.1.3
  Commit:  4a9f85e9f8

.NET Core SDKs installed:
  3.1.201 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.App 3.1.3 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 3.1.3 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

[zivkan]

For Microsoft's 2021 hackathon, I started working on this, however, now that NuGet is integrated in MSBuild, I found that MSBuild itself is blocking this from being possible to implement. Specifically, after an MSBuild log message is written, I need to be able to write the progress bar to the new last line on the console. However, MSBuild's Log*() methods are non-blocking, meaking that most of the time my code writes the progress bar before MSBuild writes the message, making output quite unpleasant.

• Ability for Tasks (such as NuGet) to display progress, have fine-grained cursor position control dotnet/msbuild#6944

For now, I feel like this issue is blocked.

[gbaydin]

Hi @zivkan I'm really happy to hear you considered to work on this.

It's easily the biggest problem with the user experience in dotnet package management for me. All other ecosystems have some way of indicating when the user is waiting for a package download to finish. In my case, still in 2021, I'm sometimes waiting more than 30 minutes for a dotnet restore to finish with absolutely no indication of what is going on, due to the large nature of packages I need to use for a machine learning project (multi-GB nuget packages). The command line seems blocked during this time.

I understand the difficulty you faced with new lines in the MSBuild design. Even in that case, I think nuget/msbuild can as a minimum do something like "if the package download seems to take more than 10 seconds, print a new line saying that nuget is downloading package, this might take a few minutes..." so there is at least an indication of what is going on.

I'm actually really frustrated that nuget/msbuild project managers haven't addressed something so basic this for so long.

[zivkan]

FWIW, here's where we got in the hackathon.

but it uses new MSBuild APIs to avoid the issue in the linked msbuild repo. Therefore, NuGet can't ship this prototype until/unless MSBuild ship the new API.

[xoofx]

We are running dotnet restore on large projects with lots of RuntimeIdentifiers, my network connection not being always fast can make the restore take more than 10min and while I'm not seeing any progress. This is quite a frustrating experience to say the least.

@zivkan it seems you had a prototype on showing some progress. What should be done to have this upstream? Do you have this on a branch somewhere?

[baronfel]

@xoofx we on the MSBuild side are also interested in progress reporting, potentially including it into the Logger APIs the engine provides to Tasks so that loggers like Terminal Logger can report progress - this API shape is something we're hoping to work on partners with over the next few months. We'd prefer to do something in a unified way rather than each Task having to reimplement the wheel.

[xoofx]

@baronfel That's great to hear!

I can see that dotnet/msbuild#6944 is opened. Unless someone is already working on it, and if this can help, I can make a proposal API with a PR to start the discussion?

[tkreindler]

Has there been any progress on this? I work on a large monorepo within Microsoft and long NuGet restores of 15+ minutes are an everyday occurrence - and easily over an hour if you're doing some cross-cutting refactor. A progress bar would be great.

[baronfel]

We're working on this in MSBuild itself right now. The MSBuild engine will gain APIs for reporting progress, then Restore and other Tasks can use those APIs to tell the MSBuild engine what the status of their operations are.

Once we have these APIs, MSBuild loggers like the new Terminal Logger can render these progress messages using things like progress bars.