Update-Help not working with https?

[iricigor]

Steps to reproduce

I have created updatable help system according to docs.microsoft.com pages

You may see the module here. Module info and content files are inside cab directory.

It is working fine in Windows PowerShell, but not in PowerShell Core v.6.2

Update-Help -Module InstallModuleFromGit -Verbose -Force

Expected behavior

This is the output from Windows PowerShell session.

VERBOSE: Your connection has been redirected to the following URI: "http://github.com/iricigor/InstallModuleFromGit/raw/master/cab/"
VERBOSE: Performing the operation "Update-Help" on target "InstallModuleFromGit, Current Version: 0.0.0.0, Available Version: 1.0.0.0, UICulture:
en-US".
VERBOSE: InstallModuleFromGit: Updated W:\PowerShell\GitHub\InstallModuleFromGit\en-US\InstallGitModule-help.xml. Culture en-US Version 1.0.0.0
VERBOSE: InstallModuleFromGit: Updated W:\PowerShell\GitHub\InstallModuleFromGit\en-US\InstallModuleFromGit-help.xml. Culture en-US Version 1.0.0.0

Actual behavior

And this is the output from PowerShell Core.

VERBOSE: Your connection has been redirected to the following URI: "http://github.com/iricigor/InstallModuleFromGit/raw/master/cab/"
VERBOSE: Performing the operation "Update-Help" on target "InstallModuleFromGit, Current Version: 0.0.0.0, Available Version: 1.0.0.0, UICulture: en-US".
VERBOSE: Your connection has been redirected to the following URI: "https://github.com/iricigor/InstallModuleFromGit/raw/master/cab/"
VERBOSE: Performing the operation "Update-Help" on target "InstallModuleFromGit, Current Version: 0.0.0.0, Available Version: 1.0.0.0, UICulture: en-US".
update-help : Failed to update Help for the module(s) 'InstallModuleFromGit' with UI culture(s) {en-US} : Help content cannot be found. Make sure the server is available and the help content location is properly defined in the HelpInfo XML.
At line:1 char:1
+ update-help -Module InstallModuleFromGit -Verbose -Force
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : ResourceUnavailable: (:) [Update-Help], Exception
+ FullyQualifiedErrorId : HelpContentNotFound,Microsoft.PowerShell.Commands.UpdateHelpCommand

Environment data

Name                           Value
----                           -----
PSVersion                      6.2.0
PSEdition                      Core
GitCommitId                    6.2.0
OS                             Microsoft Windows 10.0.17763
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Comment 1

I have compared my setup and PowerShell built-in modules setup and the only difference I noticed was https (my setup) vs. http protocol (PowerShell modules). So, I tried to use http to prove it in new branch, but I could not give proof as GitHub seems to redirect download requests, as visible in the PS Core output above. Notice that first redirect points to http, and 2nd one to https and then it fails.

Comment 2

I have tried also to simulate really missing files and in that case I receive one more error before the one above. This new error says:

Response status code does not indicate success: 404 (Not Found)

Comment 3

Even if this turns out to be my setup problem, I think such generic error messages should be improved. The error HelpContentNotFound was being thrown if anything fails, without real control if content is found or not. Also, at least it my experience it looks like InnerException is just dismissed, as I was not able to see it in any error.

[iSazonov]

Do you see the issue with PowerShell 7.0?

[iricigor]

Hi @iSazonov , Sorry for delayed response.

Yes, the same behavior can be observed in PowerShell 7.0.2, the error is still HelpContentNotFound,Microsoft.PowerShell.Commands.UpdateHelpCommand

┌[iiric@IIRIC1]
└[~]> find-module InstallModuleFromGit | install-module -Scope CurrentUser
┌[iiric@IIRIC1]
└[~]> $ErrorView = 'NormalView'
┌[iiric@IIRIC1]
└[~]> Update-Help -Module InstallModuleFromGit -Verbose -Force
VERBOSE: Your connection has been redirected to the following URI: "https://github.com/iricigor/InstallModuleFromGit/raw/master/cab/"
VERBOSE: Performing the operation "Update-Help" on target "InstallModuleFromGit, Current Version: 0.0.0.0, Available Version: 1.0.0.0, UICulture: en-US".
Update-Help : Failed to update Help for the module(s) 'InstallModuleFromGit' with UI culture(s) {en-US} : Help content cannot be found. Make sure the server is available and the help content location is properly defined in the HelpInfo XML..
English-US help content is available and can be installed using: Update-Help -UICulture en-US.
At line:1 char:1
+ Update-Help -Module InstallModuleFromGit -Verbose -Force
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : ResourceUnavailable: (:) [Update-Help], Exception
+ FullyQualifiedErrorId : HelpContentNotFound,Microsoft.PowerShell.Commands.UpdateHelpCommand
┌[iiric@IIRIC1] [x]
└[~]> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.0.2
PSEdition                      Core
GitCommitId                    7.0.2
OS                             Microsoft Windows 10.0.19042
Platform                       Win32NT
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

It is failing also on WSL

PS /mnt/c/Users/iiric> Update-Help -Module InstallModuleFromGit -Verbose -Force
VERBOSE: Your connection has been redirected to the following URI: "https://github.com/iricigor/InstallModuleFromGit/raw/master/cab/"
VERBOSE: Performing the operation "Update-Help" on target "InstallModuleFromGit, Current Version: 0.0.0.0, Available Version: 1.0.0.0, UICulture: en-US".
Update-Help: Failed to update Help for the module(s) 'InstallModuleFromGit' with UI culture(s) {en-US} : Help content cannot be found. Make sure the server is available and the help content location is properly defined in the HelpInfo XML..
English-US help content is available and can be installed using: Update-Help -UICulture en-US.
PS /mnt/c/Users/iiric> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      7.0.2
PSEdition                      Core
GitCommitId                    7.0.2
OS                             Linux 4.19.104-microsoft-standard #1 SMP Wed Feb 19 06:37:35 UTC 2020
Platform                       Unix
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1
WSManStackVersion              3.0

Just for reference, here is successful output from Windows PowerShell 5.1

PS C:\Users\iiric> find-module InstallModuleFromGit | install-module -Scope CurrentUser
PS C:\Users\iiric> Update-Help -Module InstallModuleFromGit -Verbose -Force
VERBOSE: Your connection has been redirected to the following URI:
"https://github.com/iricigor/InstallModuleFromGit/raw/master/cab/"
VERBOSE: Performing the operation "Update-Help" on target "InstallModuleFromGit, Current Version: 0.0.0.0, Available Version:
1.0.0.0, UICulture: en-US".
VERBOSE: InstallModuleFromGit: Updated
C:\Users\iiric\Documents\WindowsPowerShell\Modules\InstallModuleFromGit\1.1.0\en-US\InstallModuleFromGit-help.xml. Culture en-US
 Version 1.0.0.0
VERBOSE: Your connection has been redirected to the following URI:
"https://github.com/iricigor/InstallModuleFromGit/raw/master/cab/"
VERBOSE: Performing the operation "Update-Help" on target "InstallModuleFromGit, Current Version: 0.0.0.0, Available Version:
1.0.0.0, UICulture: en-US".
VERBOSE: InstallModuleFromGit: Updated
C:\Users\iiric\Documents\WindowsPowerShell\Modules\InstallModuleFromGit\1.0.1\en-US\InstallModuleFromGit-help.xml. Culture en-US
 Version 1.0.0.0
PS C:\Users\iiric> $PSVersionTable

Name                           Value
----                           -----
PSVersion                      5.1.19041.1
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.19041.1
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

[iSazonov]

/cc @sdwheeler Have you any thoughts about the issue?

[iricigor]

Sorry, I just noticed that error message is not the same! Error category is the same, but descrription is now updated. There is this additional line:

English-US help content is available and can be installed using: Update-Help -UICulture en-US.

However, running update with that parameter is not helping. It is visible from original command that it is trying already to download en-US help.

Update-Help: Failed to update Help for the module(s) 'InstallModuleFromGit' with UI culture(s) {en-US} : Help content cannot be found. Make sure the server is available and the help content location is properly defined in the HelpInfo XML..
English-US help content is available and can be installed using: Update-Help -UICulture en-US.

That error description points to this piece of code.

If I specify some unsupported culture, I will get proper message.

Update-Help -Module InstallModuleFromGit -verbose -UICulture cs-CZ -force
VERBOSE: Your connection has been redirected to the following URI: "https://github.com/iricigor/InstallModuleFromGit/raw/master/cab/"
Update-Help: Failed to update Help for the module(s) 'InstallModuleFromGit' with UI culture(s) {cs-CZ} : The specified culture is not supported: cs-CZ. Specify a culture from the following list: {en-US}..
English-US help content is available and can be installed using: Update-Help -UICulture en-US.

Note that message now says The specified culture is not supported instead of Help content cannot be found

If I use the same files as published, but from local folder, Update-Help is successful.

┌[iiric@IIRIC1] [ master ≣]
└[~\repos\GitHub\InstallModuleFromGit]> Update-Help -module InstallModuleFromGit -Verbose -Force -Path '.\cab'
VERBOSE: Performing the operation "Update-Help" on target "InstallModuleFromGit, Current Version: 1.0.0.0, Available Version: 1.0.0.0, UICulture: en-US".
VERBOSE: InstallModuleFromGit: Updated C:\Users\iiric\Documents\PowerShell\Help\InstallModuleFromGit\1.1.0\en-US\InstallModuleFromGit-help.xml. Culture en-US Version 1.0.0.0

[iricigor]

So, I seems to have found the root cause.

I am using this url for help https://github.com/iricigor/InstallModuleFromGit/raw/master/cab/ which seems to be OK

(iwr 'https://github.com/iricigor/InstallModuleFromGit/raw/master/cab/').StatusCode
200

But, if I run it like this, it shows the problem:

(iwr 'https://github.com/iricigor/InstallModuleFromGit/raw/master/cab/' -MaximumRedirection 0).StatusCode
Invoke-WebRequest: You are being redirected.

So, the problem is this piece of code. It is not allowing redirects.

private bool DownloadHelpContentHttpClient(string uri, string fileName, UpdatableHelpCommandType commandType)
{
    // TODO: Was it intentional for them to remove IDisposable from Task?
    using (HttpClientHandler handler = new HttpClientHandler())
    {
        handler.AllowAutoRedirect = false;
        handler.UseDefaultCredentials = WebClient.UseDefaultCredentials;
        using (HttpClient client = new HttpClient(handler))
        {
            client.Timeout = _defaultTimeout;
            Task<HttpResponseMessage> responseMsg = client.GetAsync(new Uri(uri), _cancelTokenSource.Token);

Conclusion and Final Question

Updates on my side is simple, but this is different behavior in Windows PowerShell and PowerShell 7. Windows PowerShell is allowing redirects during Update-Help command, while PowerShell 7 is not allowing them. Was this intentional or should it be fixed?

[iSazonov]

@iricigor The code you referenced comes from Windows PowerShell.

/cc @adityapatwardhan for conclusion.

[sdwheeler]

I think the real problem is GitHub. They don't allow raw access to the files in the repo they way Update-Help expects to get them. You might have better results if you published the help files (xml/cab/zip) in the release assests. Then GitHub provides a direct download URL https://github.com/iricigor/releases/download/InstallModuleFromGit/<release-tag>/.

[iricigor]

@sdwheeler I managed to update my module to use direct (not redirected) url for help files, and I am not getting HelpContentNotFound error anymore.

@iSazonov Yeap, I figured out that mentioned code comes from monad times, so I am not sure why it works in Windows PowerShell. I tried to recompile DownloadHelpContentHttpClient code on my machine, and it is failing on this line a few lines later in the code

if (response.IsSuccessStatusCode)
{
	WriteResponseToFile(response, fileName);
}
else
{
	Errors.Add(new UpdatableHelpSystemException("HelpContentNotFound",
		StringUtil.Format(HelpDisplayStrings.HelpContentNotFound),
		ErrorCategory.ResourceUnavailable, null, responseMsg.Exception));
}

Variable response gets status code 301 and it fails. If recompiled with handler.AllowAutoRedirect = true; it returns success status.

[iSazonov]

@iricigor You could compile a debug build and use debugger to find a place where the fix could be. This makes sense only if @adityapatwardhan confirms.

[adityapatwardhan]

I am sorry, working on a release now. I can only have a look at this after the release.

[iricigor]

@adityapatwardhan Take your time, nothing urgent here. I will just summarize the above here.

Summary

Command Update-Help behaves differently in Windows PowerShell and PowerShell Core / PowerShell 7. If target url is redirected, Windows PowerShell follows the url and returns success. In case of pwsh, it returns an error HelpContentNotFound.

Root cause

In current PowerShell, the code explicitly disallows redirects. It can be easily updated, if needed. The mentioned code is coming from Windows PowerShell time and its not modified during PowerShell OSS era.

Questions

1. I am not sure if it is OK to allow redirects in the code. Anyone sees some security considerations?
2. Even if we keep "no redirects" setting, is it needed to at least update error handling with new error description?

[iSazonov]

I am not sure if it is OK to allow redirects in the code. Anyone sees some security considerations?

/cc @TravisEz13

[sdwheeler]

Redirects are required because we use our FWLink redirection service for these URLs.

[TravisEz13]

Redirection is allowed for non-executable content (for Microsoft services HTTPS is required)

[microsoft-github-policy-service]

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

[microsoft-github-policy-service]

This issue has not had any activity in 6 months, if this is a bug please try to reproduce on the latest version of PowerShell and reopen a new issue and reference this issue if this is still a blocker for you.

[microsoft-github-policy-service]

This issue has been marked as "No Activity" as there has been no activity for 6 months. It has been closed for housekeeping purposes.