[Bug] Can not install with url encoded in utf-8

[yi-Xu-0100]

Bug Report

Current Behavior

Expected Behavior

The file is encoding with utf-8, but the Webclient use default encoding, and it is not recommended.

See more in system.net.webclient.encoding and system.text.encoding.default.

Additional context/output

Possible Solution

We can solve this problem by set the Encoding for Webclient.

System details

Windows version: [e.g. 7, 8, 10]

win10

OS architecture: [e.g. 32bit, 64bit]

64bit

PowerShell version: [output of "$($PSVersionTable.PSVersion)"]

5.1.19041.1645

Additional software: [(optional) e.g. ConEmu, Git]

Scoop Configuration

default setting.

[yi-Xu-0100]

@rashil2000 I get the character encoding from the Content-Type header in the response with the help of #4324 (comment).

$wc = New-Object Net.Webclient
$wc.Headers.Add('User-Agent', (Get-UserAgent))
$rawData = $wc.downloadData($url)
if($wc.ResponseHeaders["Content-Type"] -match 'charset=([^;]*)') {
  $encoding = [System.Text.Encoding]::GetEncoding($Matches[1])
} else {
  $encoding = [System.Text.Encoding]::GetEncoding('utf-8')
}
$str = $encoding.GetString($rawData)

And I found many webclient.downloadstring statements were used in different files.

May we add a function about it in files that include the utils? Or create a new file to provide this function?

[rashil2000]

May we add a function about it in files that include the utils?

We can do this

[yi-Xu-0100]

@rashil2000 I did not find a file that includes common utils. Or did I miss it?

[rashil2000]

lib/core.ps1