Unusually High CPU While Fetching Mirrors

[tresf]

I'm noticing 80-100% CPU utilization during the fetching process for mirrors (Server 2016 Core Eval).

Is anyone else experiencing this? Is there an inefficiency in the CUI download indicator, or is this a bug specific to the OS/VM technology that I am on?

It doesn't seem to cause any major problems in a development environment, but for a production system, this risks the chance of stealing the resources from elsewhere. May be all or part of the cause for the Anti-Malware executable acting up mid-install (e.g. possibly related to #1388).

[rrelmy]

I noticed that downloads are significantly slower with scoop than with a browser. I have ~10% CPU (maybe 1 of 8 HT cores?) usage on the powershell process.
Seems more than it should be. Need to investigate

[deevus]

Yeah I have noticed this as well. With the work I was doing with the progress bars I was working towards being able to switch out the downloader for something else if needed. Perhaps this could be user configurable.

…

On Tue, Mar 21, 2017 at 8:13 AM Rémy ***@***.***> wrote: I noticed that downloads are significantly slower with scoop than with a browser. I have ~10% CPU (maybe 1 of 8 HT cores?) usage on the powershell process. Seems more than it should be. Need to investigate — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1389 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA0QgjI6CsS2xmrDlBhszwcXlJmgqDeKks5rnuvcgaJpZM4MiCGF> .

[tresf]

I've found that commenting out the download cursor logic fixes the CPU spikes introduced with the [========> 23%] download indicator. There are still a few other spikes, but this is a good start.

The cursor draw routine must have an expensive call in it although I think it can be improved anyway.
https://github.com/lukesampson/scoop/blob/master/lib/install.ps1#L215-L252

If I'm reading this correctly:

• The download indicator is updated with the buffer; each 2,048 B
• If my math is correct... A 100 MB package = yields 48,828 executions (1e+8 / 2048) of the draw routine.
• Assuming we're interested in 0%-100% indicator, this means on average, each percent of a 100MB download is drawn about 500 times.

I've never done CUI/CLI interface coding before, so I'm not prepared to write up a PR just yet, but I think drawing the indicator less often will help remedy this problem. There are likely other indicators that could benefit from less frequent updates as well if the CPU spikes correlate/indicate ("other indicators" is purely speculative, sorry).

[deevus]

Since I wrote the progress logic I'll have a look at optimising it a bit.

…

On Tue, Mar 21, 2017 at 2:45 PM Tres Finocchiaro ***@***.***> wrote: I've found that commenting out the download cursor logic <https://github.com/lukesampson/scoop/blob/master/lib/install.ps1#L167> fixes the CPU spikes introduced with the download indicator. There are still a few other spikes, but this helps. The cursor draw routine must have an expensive call in it although I think it can be improved. https://github.com/lukesampson/scoop/blob/master/lib/install.ps1#L215-L252 If I'm reading this correctly: - The download indicator is updated with the buffer; each 2,048 B - If my math is correct... 100 MB = 48,828 executions of the draw routine. - Assuming we're interested in 1-100% indicator, this means on average, each percent is drawn about 500 times. I've never done CUI interface before, so I'm not prepared to write up a PR just yet, but I think drawing the indicator less often will help remedy this problem. There are likely other indicators that could benefit from less frequent updates. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1389 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA0Qgn5UXAdtMFUjVDsab8X8E-hwE67Bks5rn0f0gaJpZM4MiCGF> .

[tresf]

There are still some serious performance problems with the download process.

I've upgraded and confirmed I have 01d9b95 in lib\install.ps1

Please see right-most edge...

With L201 $last = dl_onProgress $totalRead $last

Without L201 $last = dl_onProgress $totalRead $last

[deevus]

Can you try increasing the timeout to see if that helps?

…

On Thu, Mar 23, 2017 at 12:36 PM Tres Finocchiaro ***@***.***> wrote: There are still some serious performance problems with the download process. Please see right-most edge... With $last = dl_onProgress $totalRead $last [image: image] <https://cloud.githubusercontent.com/assets/6345473/24227939/47d93880-0f47-11e7-88be-b3a839fd0652.png> Without $last = dl_onProgress $totalRead $last [image: image] <https://cloud.githubusercontent.com/assets/6345473/24227969/7b2511c8-0f47-11e7-8519-2e2e97251aea.png> — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#1389 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA0QglYhtvuoRosG2NUhJuIxAH5k89bRks5rocyqgaJpZM4MiCGF> .

[tresf]

I changed it to 1 second (1000 ms) same problem.

[deevus]

Perhaps it has to do with the number of times the event is thrown, regardless of how often we draw to the screen.

[deevus]

I'm probably not the best person to test this, because my machine is a beast (i7-5820k /w 64GB RAM) whereas you're running a laptop i7

[deevus]

What if you do call $last = dl_onProgress $totalRead $last but you make the function just return?

[tresf]

Perhaps it has to do with the number of times the event is thrown, regardless of how often we draw to the screen.

I think it's a factor, but it's not a smoking gun. Yes, this still exposes the performance problems, but so does moving the $delta directly into the while {...}.

I was able to eradicate it by using a KB counter (e.g. byte.length * 100), but on slow connections with very small packages (e.g. awk), it doesn't really show enough progress (the timer is the ideal solution).

Still trying things....

[tresf]

@deevus I've created a PR #1394 which seems to fix the problem. It appears many things can spike the CPU of powershell.exe, namely

1. New date instantiations/millis math.
2. Frequent function calls -- even if empty
3. Frequent screen updates.

By using a stopwatch directly within the while { ... } loop, it seems to fix the problem.

I blindly reverted your 01d9b95 but there is a formatting change that gets reverted. Let me know if you need that back in.

For a visual...

Before #1394

After #1394

[ghost]

Oh this ticket is tree years old and I have exact two issues.
i installed it today, slow downloading and hight cpu usage.