Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dotnet restore stalls and prints a lot of things #6168

Closed
smoothdeveloper opened this issue May 19, 2016 · 5 comments
Closed

dotnet restore stalls and prints a lot of things #6168

smoothdeveloper opened this issue May 19, 2016 · 5 comments
Labels
question
Milestone
1.1.0

Comments

@smoothdeveloper
Copy link

Steps to reproduce

  • checkout microsoft/visualfsharp
  • cd into lkg folder
  • call dotnet restore
  • wait for eternity

Expected behavior

  • top level summary of packages that needs to be downloaded
  • download packages with a reasonable degree of concurrency
  • report progress after each package being downloaded

Actual behavior

  • pages and pages (and pages) of output with 0 value to end user dotnetrestore.txt (this is 280kb, incomplete output since the process stalled)
  • lots of http requests done without any concern about bandwidth, throttling
  • eventually process stalls completely, 20 min after launching the command, no output, no cpu usage, no network activity

Environment data

dotnet --info output:

>tools\dotnetcli\dotnet --info
.NET Command Line Tools (1.0.0-rc2-002665)

Product Information:
 Version:     1.0.0-rc2-002665
 Commit Sha:  6eea734681

Runtime Environment:
 OS Name:     Windows
 OS Version:  6.1.7601
 OS Platform: Windows
 RID:         win7-x64

Just for your information, to compare user experience I created this text file with edlin.exe (format is so simple that I don't need anything better):

paket.dependencies

source https://www.myget.org/F/fsharp-daily/api/v3/index.json
source https://www.myget.org/F/dotnet-core/api/v3/index.json
source https://www.myget.org/F/dotnet-buildtools/api/v3/index.json
source https://api.nuget.org/v3/index.json
source https://www.nuget.org/api/v2/

nuget Microsoft.FSharp.Compiler.Host.netcore = 1.0.0-alpha-160406

called paket.exe install next to it

and here is the (entire) output that got printed, I just paste it because unlike the other one I've attached, it can be printed without scaring :

Paket version 2.64.0.0
Resolving packages for group Main:
 - Microsoft.FSharp.Compiler.Host.netcore is pinned to 1.0.0-alpha-160406
 - Microsoft.NETCore 5.0.1-rc3-24117-00
 - Microsoft.NETCore.Runtime.CoreCLR 1.0.2-rc3-24117-00
 - Microsoft.NETCore.Platforms 1.0.1-rc3-24117-00
 - Microsoft.NETCore.Windows.ApiSets 1.0.1-rc3-24117-00
 - Microsoft.NETCore.ConsoleHost 1.0.0-rc3-24117-00
 - Microsoft.FSharp.Compiler.NetCore 1.0.0-alpha-160509
 - Microsoft.VisualBasic 10.0.1-rc3-24117-00
 - System.Threading.Tasks.Dataflow 4.6.0-rc3-24117-00
 - System.Numerics.Vectors 4.1.1-rc3-24117-00
 - System.AppContext 4.1.0-rc3-24117-00
 - System.ComponentModel.Annotations 4.1.0-rc3-24117-00
 - System.Diagnostics.Tracing 4.1.0-rc3-24117-00
 - System.IO 4.1.0-rc3-24117-00
 - System.IO.Compression 4.1.0-rc3-24117-00
 - System.Linq 4.1.0-rc3-24117-00
 - System.Linq.Expressions 4.1.0-rc3-24117-00
 - System.Net.Http 4.1.0-rc3-24117-00
 - System.Net.NetworkInformation 4.1.0-rc3-24117-00
 - System.Reflection 4.1.0-rc3-24117-00
 - System.Reflection.TypeExtensions 4.1.0-rc3-24117-00
 - System.Runtime 4.1.0-rc3-24117-00
 - System.Runtime.Extensions 4.1.0-rc3-24117-00
 - System.Runtime.InteropServices 4.1.0-rc3-24117-00
 - System.Text.RegularExpressions 4.1.0-rc3-24117-00
 - System.Diagnostics.Process 4.1.0-rc3-24117-00
 - System.Security.Cryptography.Algorithms 4.2.0-rc3-24117-00
 - System.Collections.Concurrent 4.0.12-rc3-24117-00
 - System.ObjectModel 4.0.12-rc3-24117-00
 - System.Collections 4.0.11-rc3-24117-00
 - System.Diagnostics.Debug 4.0.11-rc3-24117-00
 - System.Dynamic.Runtime 4.0.11-rc3-24117-00
 - System.Globalization 4.0.11-rc3-24117-00
 - System.Net.Primitives 4.0.11-rc3-24117-00
 - System.Text.Encoding 4.0.11-rc3-24117-00
 - System.Text.Encoding.Extensions 4.0.11-rc3-24117-00
 - System.Threading 4.0.11-rc3-24117-00
 - System.Threading.Tasks 4.0.11-rc3-24117-00
 - System.Xml.ReaderWriter 4.0.11-rc3-24117-00
 - System.Xml.XDocument 4.0.11-rc3-24117-00
 - System.Net.Requests 4.0.11-rc3-24117-00
 - System.Threading.ThreadPool 4.0.10-rc3-24117-00
 - Microsoft.CSharp 4.0.1-rc3-24117-00
 - System.ComponentModel 4.0.1-rc3-24117-00
 - System.Diagnostics.Tools 4.0.1-rc3-24117-00
 - System.Globalization.Calendars 4.0.1-rc3-24117-00
 - System.Globalization.Extensions 4.0.1-rc3-24117-00
 - System.IO.Compression.ZipFile 4.0.1-rc3-24117-00
 - System.IO.FileSystem 4.0.1-rc3-24117-00
 - System.IO.FileSystem.Primitives 4.0.1-rc3-24117-00
 - System.IO.UnmanagedMemoryStream 4.0.1-rc3-24117-00
 - System.Linq.Parallel 4.0.1-rc3-24117-00
 - System.Linq.Queryable 4.0.1-rc3-24117-00
 - System.Reflection.DispatchProxy 4.0.1-rc3-24117-00
 - System.Reflection.Extensions 4.0.1-rc3-24117-00
 - System.Reflection.Primitives 4.0.1-rc3-24117-00
 - System.Resources.ResourceManager 4.0.1-rc3-24117-00
 - System.Runtime.Handles 4.0.1-rc3-24117-00
 - System.Runtime.Numerics 4.0.1-rc3-24117-00
 - System.Security.Claims 4.0.1-rc3-24117-00
 - System.Security.Principal 4.0.1-rc3-24117-00
 - System.Threading.Tasks.Parallel 4.0.1-rc3-24117-00
 - System.Threading.Timer 4.0.1-rc3-24117-00
 - System.Reflection.Emit 4.0.1-rc3-24117-00
 - System.Reflection.Emit.ILGeneration 4.0.1-rc3-24117-00
 - System.Diagnostics.TraceSource 4.0.0-rc3-24117-00
 - System.Runtime.InteropServices.PInvoke 4.0.0-rc3-24117-00
 - System.Runtime.Loader 4.0.0-rc3-24117-00
 - System.Security.Cryptography.Primitives 4.0.0-rc3-24117-00
 - System.Threading.Thread 4.0.0-rc3-24117-00
 - NETStandard.Library 1.5.0-rc3-24117-00
 - System.Reflection.Metadata 1.3.0-rc3-24117-00
 - System.Collections.Immutable 1.2.0-rc3-24117-00
 - Microsoft.DiaSymReader 1.0.7
 - Microsoft.NETCore.Targets 1.0.1-rc3-24117-00
 - Microsoft.DiaSymReader.PortablePdb 1.0.0-rc-60301
 - Microsoft.FSharp.Core.netcore 1.0.0-alpha-160509
 - System.Runtime.Serialization.Primitives 4.1.1-rc3-24117-00
 - System.Security.Cryptography.X509Certificates 4.1.0-rc3-24117-00
 - runtime.native.System.IO.Compression 4.1.0-rc3-24117-00
 - System.Net.Sockets 4.1.0-rc3-24117-00
 - System.Runtime.WindowsRuntime 4.0.11-rc3-24117-00
 - System.Runtime.InteropServices.WindowsRuntime 4.0.1-rc3-24117-00
 - System.Diagnostics.Contracts 4.0.1-rc3-24117-00
 - Microsoft.Win32.Primitives 4.0.1-rc3-24117-00
 - System.Reflection.Emit.Lightweight 4.0.1-rc3-24117-00
 - System.Threading.Overlapped 4.0.1-rc3-24117-00
 - runtime.native.System.Net.Http 4.0.1-rc3-24117-00
 - System.Net.WebHeaderCollection 4.0.1-rc3-24117-00
 - System.Threading.Tasks.Extensions 4.0.0-rc3-24117-00
 - System.Diagnostics.DiagnosticSource 4.0.0-rc3-24117-00
 - runtime.native.System 4.0.0-rc3-24117-00
 - System.Buffers 4.0.0-rc3-24117-00
 - System.Security.Principal.Windows 4.0.0-rc3-24117-00
 - runtime.native.System.Security.Cryptography 4.0.0-rc3-24117-00
 - System.Security.Cryptography.Encoding 4.0.0-rc3-24117-00
 - System.Security.Cryptography.OpenSsl 4.0.0-rc3-24117-00
 - System.Console 4.0.0-rc3-24117-00
 - System.Runtime.InteropServices.RuntimeInformation 4.0.0-rc3-24117-00
 - Microsoft.Win32.Registry 4.0.0-rc3-24117-00
 - System.Security.Cryptography.Cng 4.1.0-rc3-24117-00
 - System.Security.Cryptography.Csp 4.0.0-rc3-24117-00
Locked version resolution written to C:\dev\src\projects\github.com\visualfsharp\.paket\test\paket.lock
Downloading Microsoft.CSharp 4.0.1-rc3-24117-00
Downloading Microsoft.DiaSymReader 1.0.7
Downloading Microsoft.FSharp.Compiler.Host.netcore 1.0.0-alpha-160406
Downloading Microsoft.DiaSymReader.PortablePdb 1.0.0-rc-60301
Downloading Microsoft.FSharp.Compiler.netcore 1.0.0-alpha-160509
Downloading Microsoft.FSharp.Core.netcore 1.0.0-alpha-160509
Downloading Microsoft.NETCore 5.0.1-rc3-24117-00
Downloading Microsoft.NETCore.ConsoleHost 1.0.0-rc3-24117-00
Downloading Microsoft.NETCore.Platforms 1.0.1-rc3-24117-00
Downloading Microsoft.NETCore.Runtime.CoreCLR 1.0.2-rc3-24117-00
Downloading Microsoft.NETCore.Targets 1.0.1-rc3-24117-00
Downloading Microsoft.NETCore.Windows.ApiSets 1.0.1-rc3-24117-00
Downloading Microsoft.VisualBasic 10.0.1-rc3-24117-00
Downloading Microsoft.Win32.Primitives 4.0.1-rc3-24117-00
Downloading Microsoft.Win32.Registry 4.0.0-rc3-24117-00
Downloading NETStandard.Library 1.5.0-rc3-24117-00
Downloading runtime.native.System 4.0.0-rc3-24117-00
Downloading runtime.native.System.IO.Compression 4.1.0-rc3-24117-00
Downloading runtime.native.System.Net.Http 4.0.1-rc3-24117-00
Downloading runtime.native.System.Security.Cryptography 4.0.0-rc3-24117-00
Downloading System.AppContext 4.1.0-rc3-24117-00
Downloading System.Buffers 4.0.0-rc3-24117-00
Downloading System.Collections 4.0.11-rc3-24117-00
Downloading System.Collections.Concurrent 4.0.12-rc3-24117-00
Downloading System.Collections.Immutable 1.2.0-rc3-24117-00
Downloading System.ComponentModel 4.0.1-rc3-24117-00
Downloading System.ComponentModel.Annotations 4.1.0-rc3-24117-00
Downloading System.Console 4.0.0-rc3-24117-00
Downloading System.Diagnostics.Contracts 4.0.1-rc3-24117-00
Downloading System.Diagnostics.Debug 4.0.11-rc3-24117-00
Downloading System.Diagnostics.DiagnosticSource 4.0.0-rc3-24117-00
Downloading System.Diagnostics.Process 4.1.0-rc3-24117-00
Downloading System.Diagnostics.Tools 4.0.1-rc3-24117-00
Downloading System.Diagnostics.TraceSource 4.0.0-rc3-24117-00
Downloading System.Diagnostics.Tracing 4.1.0-rc3-24117-00
Downloading System.Dynamic.Runtime 4.0.11-rc3-24117-00
Downloading System.Globalization 4.0.11-rc3-24117-00
Downloading System.Globalization.Calendars 4.0.1-rc3-24117-00
Downloading System.Globalization.Extensions 4.0.1-rc3-24117-00
Downloading System.IO 4.1.0-rc3-24117-00
Downloading System.IO.Compression 4.1.0-rc3-24117-00
Downloading System.IO.Compression.ZipFile 4.0.1-rc3-24117-00
Downloading System.IO.FileSystem 4.0.1-rc3-24117-00
Downloading System.IO.FileSystem.Primitives 4.0.1-rc3-24117-00
Downloading System.IO.UnmanagedMemoryStream 4.0.1-rc3-24117-00
Downloading System.Linq 4.1.0-rc3-24117-00
Downloading System.Linq.Expressions 4.1.0-rc3-24117-00
Downloading System.Linq.Parallel 4.0.1-rc3-24117-00
Downloading System.Linq.Queryable 4.0.1-rc3-24117-00
Downloading System.Net.Http 4.1.0-rc3-24117-00
Downloading System.Net.NetworkInformation 4.1.0-rc3-24117-00
Downloading System.Net.Primitives 4.0.11-rc3-24117-00
Downloading System.Net.Requests 4.0.11-rc3-24117-00
Downloading System.Net.Sockets 4.1.0-rc3-24117-00
Downloading System.Net.WebHeaderCollection 4.0.1-rc3-24117-00
Downloading System.Numerics.Vectors 4.1.1-rc3-24117-00
Downloading System.ObjectModel 4.0.12-rc3-24117-00
Downloading System.Reflection 4.1.0-rc3-24117-00
Downloading System.Reflection.DispatchProxy 4.0.1-rc3-24117-00
Downloading System.Reflection.Emit 4.0.1-rc3-24117-00
Downloading System.Reflection.Emit.ILGeneration 4.0.1-rc3-24117-00
Downloading System.Reflection.Emit.Lightweight 4.0.1-rc3-24117-00
Downloading System.Reflection.Extensions 4.0.1-rc3-24117-00
Downloading System.Reflection.Metadata 1.3.0-rc3-24117-00
Downloading System.Reflection.Primitives 4.0.1-rc3-24117-00
Downloading System.Reflection.TypeExtensions 4.1.0-rc3-24117-00
Downloading System.Resources.ResourceManager 4.0.1-rc3-24117-00
Downloading System.Runtime 4.1.0-rc3-24117-00
Downloading System.Runtime.Extensions 4.1.0-rc3-24117-00
Downloading System.Runtime.Handles 4.0.1-rc3-24117-00
Downloading System.Runtime.InteropServices 4.1.0-rc3-24117-00
Downloading System.Runtime.InteropServices.PInvoke 4.0.0-rc3-24117-00
Downloading System.Runtime.InteropServices.RuntimeInformation 4.0.0-rc3-24117-00
Downloading System.Runtime.InteropServices.WindowsRuntime 4.0.1-rc3-24117-00
Downloading System.Runtime.Loader 4.0.0-rc3-24117-00
Downloading System.Runtime.Numerics 4.0.1-rc3-24117-00
Downloading System.Runtime.Serialization.Primitives 4.1.1-rc3-24117-00
Downloading System.Runtime.WindowsRuntime 4.0.11-rc3-24117-00
Downloading System.Security.Claims 4.0.1-rc3-24117-00
Downloading System.Security.Cryptography.Algorithms 4.2.0-rc3-24117-00
Downloading System.Security.Cryptography.Cng 4.1.0-rc3-24117-00
Downloading System.Security.Cryptography.Csp 4.0.0-rc3-24117-00
Downloading System.Security.Cryptography.Encoding 4.0.0-rc3-24117-00
Downloading System.Security.Cryptography.OpenSsl 4.0.0-rc3-24117-00
Downloading System.Security.Cryptography.Primitives 4.0.0-rc3-24117-00
Downloading System.Security.Cryptography.X509Certificates 4.1.0-rc3-24117-00
Downloading System.Security.Principal 4.0.1-rc3-24117-00
Downloading System.Security.Principal.Windows 4.0.0-rc3-24117-00
Downloading System.Text.Encoding 4.0.11-rc3-24117-00
Downloading System.Text.Encoding.Extensions 4.0.11-rc3-24117-00
Downloading System.Text.RegularExpressions 4.1.0-rc3-24117-00
Downloading System.Threading 4.0.11-rc3-24117-00
Downloading System.Threading.Overlapped 4.0.1-rc3-24117-00
Downloading System.Threading.Tasks 4.0.11-rc3-24117-00
Downloading System.Threading.Tasks.Dataflow 4.6.0-rc3-24117-00
Downloading System.Threading.Tasks.Extensions 4.0.0-rc3-24117-00
Downloading System.Threading.Tasks.Parallel 4.0.1-rc3-24117-00
Downloading System.Threading.Thread 4.0.0-rc3-24117-00
Downloading System.Threading.ThreadPool 4.0.10-rc3-24117-00
Downloading System.Threading.Timer 4.0.1-rc3-24117-00
Downloading System.Xml.ReaderWriter 4.0.11-rc3-24117-00
Downloading System.Xml.XDocument 4.0.11-rc3-24117-00
23 minutes, 22 seconds - ready.

paket is a community project, maintained by volunteers on their free time, you might consider contributing to it and integrate it in the dotnet restore story; so far it does the job better than the dotnet restore I tried.

Paket has a lots of other goodies too.
@blackdwarf
Copy link

@smoothdeveloper you seem to be running on Win7. Do you have Virtual Box installed? Or similar? If so, you are probably running into this (known) issue: #5311 . Can you check to see if that solves your problem?

@smoothdeveloper
Copy link
Author

@blackdwarf thanks for insight.

I do have VMWare installed, which indeed creates extra network adapters.

Are we talking about a networking issue low in the dotnet core stack (HAL or managed network socket lib)?

In order to reproduce the issue, I need to clean up my nuget cache, but I have no clue where dotnet restore is actually caching stuff? It seems you didn't get this simple answer to the same question in your issue.

@blackdwarf
Copy link

@smoothdeveloper could be, the Nuget team is still investigating, as you can see in the linked issue. To see if this solves your issue, just disable those virtual adapters and re-run dotnet restore.

The NuGet cache is in your home directory on Windows under .nuget in most cases. There is a packages dir below it that you delete.

@smoothdeveloper
Copy link
Author

@blackdwarf, any idea of which upstream ticket on NuGet/NuGet.Client (make sure the capital letters are consistent to make it work on linux, although GitHub Inc. is lenient on that) we should track?

  • In dotnet restore consistently times out on Win7x64 #5311 I see a link to some code file in NuGet/NuGet.Client but no issue n°?
  • I don't see any Issues tab at the same spot
  • in this repository I see 770 open issues but they might collude a lots of different topics so it is easy to get lost / create duplicates / miss an update

We should "trickle-up" to the source of truth (NuGet.Client), and have the maintainers work on :

  • better console output (284kb for my use case is too much to read for most maintainers)
  • not doing so many http request at once
  • figure out a optimized path (less chatty protocol with server? I have no clue) to actually restore the complete package as a single archive, even if that means downloading more but reducing a lot number of requests (firewall and proxies will be happy, and I assume NuGet.Server too).

Thanks!

@TheRealPiotrP
Copy link
Contributor

TheRealPiotrP commented May 26, 2016 •
edited
Loading

dotnet/cli#3229 reduces the verbosity of NuGet during dotnet restore. The other NuGet behaviors described are not under CLI control so should go to https://github.com/nuget/home

@msftgits msftgits transferred this issue from dotnet/cli Jan 31, 2020
@msftgits msftgits added this to the 1.1.0 milestone Jan 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
1.1.0
Development

No branches or pull requests
4 participants
@smoothdeveloper @TheRealPiotrP @blackdwarf @msftgits