[QUESTION] How (and why) does Pester package C# NuGet Packages?

[jzabroski]

Hi @dlwyatt

How (and why) does Pester package C# NuGet Packages the way it does?

We were looking at the most popular PSGallery packages, and, of course, your fantastic library is one of the most popular packages! We unzipped the raw nupkg and peeked inside to try to understand how you wrote your psd1 and how you were bundling your C# NuGet Packages like Newtonsoft.Json.dll and Gherkin.dll, but were surprised to see you don't declare these dependencies anywhere.

In fact, when I searched for Add-Type, I see a curious way of loading the Gherkin.dll I have not seen in anyone else's PowerShell code, and so I'm curious why you're calling Add-Type this way:

Pester/Functions/Gherkin.ps1

Line 2 in 3e77d00

& $SafeCommands["Add-Type"] -Path "${Script:PesterRoot}/lib/Gherkin/core/Gherkin.dll"

- I am guessing this is so people can mock Add-Type, and to ensure you're calling the "non-mocked" version of Add-Type.

Curiously, Newtonsoft.Json.dll doesn't appear to be loaded at all. Presumably, this is because you're using an older version of Gherkin.dll - but this doesn't really address my question, because ideally I want to load Nuget packages, not .NET assemblies, since they contain richer metadata than the assemblies themselves and are the actual artifact sources I want to include.

[jzabroski]

Possibly related, @mklement0 wrote this issue: PowerShell/PowerShell#6724

[nohwnd]

Hi,

I am guessing this is so people can mock Add-Type, and to ensure you're calling the "non-mocked" version of Add-Type.

Yes that is exactly the reason :)

Curiously, Newtonsoft.Json.dll doesn't appear to be loaded at all.

Yes that is also true, it uses Gherkin that needs it as a dependency and is compiled against older .NET framework so it runs up to PowerShell v2.

I think it is packaged this way because that is how choco package was packaged, and then someone asked for nuget package as well so they can use and manage it by VisualStudio when big VS and PowerShell tools were all the rage. Nuget and choco are pretty much the same thing, so the same package is pushed to both places. Since that time a lot have changed, but the package is still packaged the same way.

Do you have any suggestions how to do it better? :)

[jzabroski]

We're discussing it internally. We have four main ideas on how things can be done for developing large scale PowerShell projects. Before I explain those, understand that it is easier for us to higher productive C# engineers than IT people who know PowerShell and have business analyst-like mindset.

Approaches

I've come up with 4 possible approaches, and given them all nicknames internally that I'll share here (emboldened), because it's hard to generate good ideas about best practices without giving back a little.

1. C# First
   1. Cmdlet Binding: Reference nuget package Microsoft.PowerShell.5.ReferenceAssemblies or Microsoft.PowerShell.SDK and just code C# Cmdlets.
2. PowerShell First
   1. Pester-like: Use your directory structure. Statically drop dll's into a lib folder
   2. PoshTools-like: PowerShell Tools for Visual Studio supports two different ways in its "PowerShell Pro Tools packager" component:
      • Since v1.10.0: Merge-Script for bundling of referenced assemblies via searching PowerShell code for [System.Reflection.Assembly]::LoadFrom()
      • Since v4.0.31: When creating packages, auto-bundle assemblies referenced via [System.Reflection.Assembly]::LoadFrom
   3. .NET SDK-like: Use a csproj file type formatted using .NET SDK project format, and <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies> to force any PackageReferences to be included in the bin directory. Then use dotnet publish to publish the bin directory as an all-encompassing package.

I guess there is a fifth style (that I've tried and liked, but found wanting more), which is @RamblingCookieMonster 's WFTools approach of a "dumb psd1 PowerShell Manifest file" that contains no information about what functions it exports (unless your build process dynamically builds the psd1 was well). The further downside of WFTools I've discovered (which is the real dealbreaker for me) is that the way it loads itself as a module causes really nasty stack traces where the line numbers are way off and hard to track down.

[jzabroski]

@nohwnd So far, I've found the ".NET SDK-like" approach to be the most desirable approach. The downside is it creates an extra dll, the Dummy.dll for the Dummy.csproj class library. The other slightly annoying behavior is that if a Nuget package supports internationalization, dotnet publish will publish all satellite assemblies available. The other annoying problem is I'm not sure how to deal with "assembling binding redirect" hell.

Our approach looks like this:

<Project Sdk="Microsoft.NET.Sdk">
 
  <PropertyGroup>
    <TargetFramework>net461;netstandard2.0</TargetFramework>
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
  </PropertyGroup>
 
  <ItemGroup>
    <PackageReference Include="SomeNugetPackage" Version="2.11.0" /> // reference the concrete dll's in RequiredAssemblies psd1 key
    <PackageReference Include="SomeOtherNugetPackage" Version="3.3.1" /> // reference the concrete dll's in RequiredAssemblies psd1 key
  </ItemGroup>
 
  <ItemGroup>
    <Content Include="MyModule.psd1" CopyToPublishDirectory="PreserveNewest"/>
    <Content Include="MyModule.psm1" CopyToPublishDirectory="PreserveNewest"/>
  </ItemGroup>
</Project>

In theory, this also allows targeting both PowerShell v5 and PowerShell Core. However, I don't think it can support older versions of .NET Framework like you mention Pester supporting .NET Framework 2 (is that even supported by Microsoft Premium Support contract?) Also, while theory is theory, in practice we discovered today that the lack of PowerShell's support for assemblyBinding redirect can also cause the need for hooking into the AppDomain's System.ResolveEventHandler (see [1] [2]). - I still haven't delivered a fully working proof-of-concept for this, and I suspect in this scenario, RequiredAssemblies key won't work, because Import-Module will load RequiredAssemblies before I can hook any assembly System.ResolveEventHandler. In this case, I might need to create a C# wrapper dll which implements a hacky DirtyAssemblyResolveHelper similar to what MSBuild and BenchmarkDotNet does. In the MSBuild case, it applies MSBuild's redirect logic.

In theory, since psd1 is a walkable file format, it should even be possible to auto-generate the RequiredAssemblies from the deps.json file generated by MSBuild when it computes the transitive closure of all assemblies to generate the lock file. At the end of the day, the RequiredAssemblies key is just that: a flattened list of the Nuget lock file.

[jzabroski]

@nohwnd We made significant progress here:

1. We write all our PowerShell commands using Binary Cmdlets, in C#.
2. We only support PowerShell 6 or greater.
3. We created a "DebugHelper.ps1" shim that takes a $AssemblyPath, $WindowTitle, and $Task of Start or Stop. Start sets the current pwsh.exe [Console]::Title = $WindowTitle, then calls Import-Module -Verbose $AssemblyPath. Stop calls Get-Process pwsh, finds the pwsh.exe with the matching $WindowTitle, and calls Stop-Process on it. Because of how assembly locking works, this guarantees you can't debug the same assembly in two separate pwsh.exe sessions, which makes stepping through C# code in Visual Studio trivial.
4. We use .NET Core launchSettings.json to configure a "profile" to launch a special pwsh.exe process, which in turn calls Import-Module $assemblyPath - this same trick can be used for debugging dlls
5. We use IModuleAssemblyInitializer to bootstrap loading our module with goodies, like Inversion of Control (NuGet package Autofac), and our Logging framework (which is a wrapper around NLog and Stackify). This allows us to collect errors with internal users using our modules, throughout the organization.
6. Not yet proof-of-concepted but exciting implication: We can use AutoFixture to auto-generate "pester combinators". e.g., Test that the function composition Do(Set-Foo, Get-Foo) is reflexive, test that the function composition Do(New-Foo, Get-Foo) is reflexive. This property-based testing allows to scrap a lot of boilerplate writing PowerShell tests.
7. Not yet proof-of-concepted but annoying: PowerShell does not have a first-class notion of semantic logging, because the notion of logging in PowerShell does not map one-to-one with how application developers think of semantic logging, since application developers typically don't have a Verbose mode but scripters do. Also, application developers tend to use namespaces to statically structure and filter or enable/disable log message output, whereas shell scripters don't use that configuration because it's interactive and dynamic. Here is a brief summary of how PowerShell thinks of logging:

   Verbose logging:

   • is aimed at end users
   • to provide additional information on demand; in PowerShell, the end users requests this information by using the -Verbose common parameter or setting the $VerbosePreference preference variable to ‘Continue’.
   • example: a script that targets multiple machines may by default (Info logging) log / display only the count of machines that are targeted; with verbose logging / display turned on, the individual machine names are output too.

   Debug logging:

   • is aimed at developers
   • to help them diagnose problems and fix bugs; thus, it is for technical information generally not of interest (or intelligible to) end users.

So far, we have written a workstation/server reboot scheduler/executor/whatif utility, an ultrafast parallel system ping utility, and file share permissions auditing utility. I can not understate how much nicer this is than writing PowerShell directly for "system's engineering".

[jzabroski]

@nohwnd One further improvement. Our launchSettings.json now looks like this:

{
  "profiles": {
    "Debug Interactively": {
      "commandName": "Executable",
      "executablePath": "pwsh.exe",
      // Note: $(TargetPath) is a built-in VS property referring to the project's output binary.
      "commandLineArgs": "-noexit -command [console]::Title = 'Cmdlet Debugging - $(MSBuildProjectName)'; Import-Module -Verbose ('$(TargetPath)' -replace '[.]dll$', '.psd1')"
    },
    "Debug via Tests": {
      "commandName": "Executable",
      "executablePath": "pwsh.exe",
      "commandLineArgs": "-NoExit -Command Import-Module ('$(TargetPath)' -replace '[.]dll$', '.psd1'); Set-Location '$(MSBuildProjectDirectory)/../$(MSBuildProjectName).Tests'; Invoke-Pester"
    }
  }
}

This second profile target lets you open a powershell window, auto-invoke Invoke-Pester with your assembly under test, and then if you have to edit the ps1 pester tests (due to a bug in the tests), you can without needing to reload the Visual Studio debug session.

The one limitation is it only lets you do one test assembly to one Binary Cmdlets assembly. Can you think of a better solution?