Enhancement: Install helpers - Parse-Args PS function for parsing Package Parameters

[sitano]

I am heavily using -ia parser for a bunch of packages i've developed.

Use cases:

1. Almost always chocolateyInstall.ps1 script wants some advanced control options to be passes with predefined default values. Actually, it is sad that choco requires additional flag to pass anything inside, but ok.
2. When a package installs service, it is usual some parameters to be passed to the service args. Still parser required to process input args with predefined imports.

What i have:

1. Parse-Args -Arg $env:ChocolateyAdvanceArgs -Default @{ option = "default value" } -> [hashtable]
2. Merge-Args -To $A -From $B

Args format:

-something => flag, argv[something] = ""
-key=name => kv, argv[key] = name
-key=name1 -key=name2 => kv, argv[key] = @( name1, name2 )
-key="name" => kv, argv[key] = name
-key="name "" name" => kv, argv[key] = name " name

Example:

choco install package -y -ia '-flag -key=value -key2="value 123 -key3=1 -key3=2"'

[ferventcoder]

Actually, it is sad that choco requires additional flag to pass anything inside, but ok.

What would you suggest as an alternative?

[ferventcoder]

Not to throw this off too much, but you should be using -params (Package Parameters), not Install Args. Install Args are meant for the native installer, where Package Parameters have a more general purpose use.

[sitano]

hm, ok - i will try

[ferventcoder]

You run into the same thing, it's just that package params do not get passed automatically through to any native installers.

[ferventcoder]

This is the article we have right now on how to parse package parameters - https://github.com/chocolatey/choco/wiki/How-To-Parse-PackageParameters-Argument

And agreed the process of parsing should become helpers.

[sitano]

got it - thanks for info, i will fix it in my scripts

[sitano]

yeah, i saw the article - the point i denied to use this method - there is no '/"" escaping inside values. + /key:value is too different from what i used to with unix tools, even PS have different syntax

[ferventcoder]

/ is the norm for Windows native installer args so it is very familiar for folks who are dealing with that. If you want to use -, go for it, since you have to add that example to your package script anyway. Just change the regex to look for - instead of /.

[sitano]

ok. i just found that c-style parser is a little bit more convenient here, coz regex for " "" " requires look ahead feature and it makes this regexp too complex to easy understand. thats why, i did my own thing :;):

[DarwinJS]

@ferventcoder - I would like to grab this one.

I was hoping to do the following as well:

• Allow "=" as well as colon for a separator (please comment if there are known issues with this)
• Allow "_" in variable names to improve readability and to conform to generally acceptable variable name conventions.
• Automatically create powershell variables (rather than a hash) $valuename = "value" and $switchargument = $true
  This is mainly to make it easy to specify default parameters at the top of the code and have args automatically override. No need for a bunch of sample code on reassigning values of a hash to yet another variable.

[ferventcoder]

@DarwinJS I await your proposal. :)

[DarwinJS]

OK - no known problems with "=" as far as you know?

Proposal = Pull Request?

Psuedo code for ChocolateyInstall.ps1:

$MyVariable = "yep"
$MyOtherVariable = "hi"
$MySwitch = $False

Import-ChocolateyPackageParams #Warning - overwrites variables of the same name.

[ferventcoder]

There are known problems with "=" :)

Proposal is what you just sent.

I'm thinking maybe simpler?

$params = Get-ChocolateyPackageParameters

[DarwinJS]

Well, if you didn't need to set defaults, then what I am proposing would be this simple:

Import-ChocolateyPackageParams

Any variables in -Params are now variables in the script - not even any fusing with a hash table.

If someone wants to manipulate settings or do checking on them, then they could make the names on the -Params parameter something other than what they use in their script.

This is also a little more like the powershell parameter patterns in that parameters defined for a script simply become variables available in your code.

[ferventcoder]

The problem is in understanding Chocolatey, you almost always need to set the defaults.

[ferventcoder]

This is also a little more like the powershell parameter patterns in that parameters defined for a script simply become variables available in your code.

The difference is that when you have a script you know about, you get to define those variables for use. When it is automatically created for you, you are trusting in documentation and magic. I'm not sure we want something magical. It needs to be straightforward and easily understandable. Simplicity is the essence of Chocolatey. And to quote this again, "Simple ain't easy."

Simple requires much more thought. It's super easy to come up with a solution to a problem. It is generally much more difficult to come up with a simple, elegant solution. It is usually one that requires much more thought.

[ferventcoder]

With RoundhousE, it took us months to come up with how database versioning should be done. We have it in there now and it is simple and makes complete sense, we use the same versioning concept as DLLs and code. But we spent a couple of months thinking about it prior to the light bulb going off and us saying "Duh!".

So long story short, I want you to find the simple, elegant solution to the problem. Which may not necessarily be the first one that comes to mind.

[DarwinJS]

Sure, I always say "Getting to simple is complex." On one contract the client seemed to have the persistent though that keeping it simple meant that the process of simplifying should also be simple - nada.

Well, whether something is "first to mind" versus not is also a function of how many times one has grappled with the problem and created solutions for the problem (or a similar one) in the past and whether there are significant patterns to key off. Before VB Script supported named parameters I had created an argument processing engine that was almost identical.

Following the powershell pattern would also be familiar to powershell coders.

I think I told you in another thread how repeatedly dealing with the versioned resources problem caused me to eventually create "Get-Versionstring" which could pull a version string from any of an EXE, a DLL, an MSI GUID or a Programs and Features friendly name. Returns 0.0.0.0 for does not exist and 0.0.0.1 for "exists with no version resource" and of course the actual version number for versioned resources. Was enhanced to strip text crap from the end (check the iexplore.exe version). Ended up being much more flexible than I originally intended.

I would suggest that any mysteries of something like "Import-ChocolateyPackagePararms" be combated by:

• a comment in the choco templates (like there is for explaining such things now)
• it's name (if there is a better one - I'm up for it. Maybe Get-ChocoPackageParamsToVariables or ConvertFrom-ChocoPackageParamsToVariables -String $env:chocolateyPackageParameters)
• it's output. In this case it could output to the session (and there for the log) something like the below (or maybe only if the verbose switch is flipped does it do this)

Get-ChocoPackageParamsToVariables found -Params and is converting it to variables...
Variable "Parameter1" is set to "cool data" (previous value: "some default value")
Variable "Parameter2" is set to "uncool data" (previous value: none)
Variable "RunFasterThanNormal" is set to $True (previous value: none)

[ferventcoder]

Yeah, naming is also hard. :)

[ferventcoder]

Get-VersionString sounds kind of interesting. :)

[DarwinJS]

Late edit to the above: ConvertFrom-ChocoPackageParamsToVariables -String $env:chocolateyPackageParameters

[DarwinJS]

Yes - it really rocks. Was part of an idempotent, DSC like engine for Desktop software deployment automation that had to handle many more complex issues than we typically do in server.

[DarwinJS]

Get-VersionString automatically figures out what type of resource you have given it as well - I like to do the hard thinking when I build the code - not everytime I go to use it ;)

[DarwinJS]

Maybe it's possible to just let PowerShell / .NET do the dirty work:

[management.automation.psparser]::Tokenize('/ACTION=Install /Q /INDICATEPROGRESS /IACCEPTSQLSERVERLICENSETERMS /FEATURES="SQLENGINE,FULLTEXT,CONN,IS,BC,SSMS,ADV_SSMS" /TCPENABLED=1 /INSTANCENAME=MSSQLSERVER /BROWSERSVCSTARTUPTYPE=Automatic /SQLSVCSTARTUPTYPE=Automatic /SQLSVCACCOUNT="NT AUTHORITY\Network Service" /SQLSYSADMINACCOUNTS="BUILTIN\Administrators" /AGTSVCACCOUNT="NT AUTHORITY\Network Service" /SECURITYMODE=SQL /SAPWD="ABC123"',[ref]$null)

Wasn't having much luck passing this in to Choco's built-in support.

From: https://blogs.technet.microsoft.com/heyscriptingguy/2011/09/20/solve-problems-with-external-command-lines-in-powershell/

[DarwinJS]

I have been playing with prototyping this and I can't read nor write variables in the scope of ChocolateyInstall.ps1. (even when using "Get-Variable -scope")

Are helper functions sandboxed in some way?

[ferventcoder]

They are sandboxed, but only in the way that the module that loads them is not in the module path and only loaded by Chocolatey.

[DarwinJS]

Ok - so then my variable scoping in a helper function is very different than the regular CMDLet / Module implementation?

I guess, my helper does not even have a shared scope with ChocolateyInstall.ps1 then?

[DarwinJS]

I would also like to add all the special characters that are valid password characters to the values that can match the value data. AWS EC2 assigned passwords seem to try to use the whole list of special chars.

[ferventcoder]

Ok - so then my variable scoping in a helper function is very different than the regular CMDLet / Module implementation?

Maybe?

I guess, my helper does not even have a shared scope with ChocolateyInstall.ps1 then?

The chocolateyInstall can call the helpers, so there is some sharing of scope. The functions are available like they would be for a regular powershell module.

[jberezanski]

I'd love to see this feature implemented, but I'd suggest a different approach to the chocolateyInstall.ps1 side of this. PowerShell already has a rich, robust, well-understood mechanism of passing arguments to scripts - why not use it?

Example chocolateyInstall.ps1 (which could be used for Visual Studio 2013):

# declare parameters using standard PowerShell syntax and facilities, such as default values, data types and validation
Param (
    [Parameter(HelpMessage = 'The optional features to be installed')]
    [ValidateSet('Blend', 'LightSwitch', 'VC_MFC_Libraries', 'SQL', 'WebTools')]
    [string[]] $Features = @('WebTools', 'SQL'),
    [Parameter(HelpMessage = 'The product key - if not provided, a 90-day trial version will be installed')]
    [ValidatePattern('^([a-zA-Z0-9]{5}-){4}([a-zA-Z0-9]){5}$')]    # ABCDE-FG123-H4567-QWERT-09876
    [string] $ProductKey
)
# use the parameters
. $PSScriptRoot\VisualStudioInstallHelpers.ps1
Install-VisualStudio -Features $Features -ProductKey $ProductKey

The parameter declaration syntax is declarative and rich in metadata, enabling scenarios such as:

• input validation: allowed values, pattern matching, data type restrictions
• handling arrays and objects convertible from string, e.g. Uri
• providing default values
• mandatory parameters ([Parameter(Mandatory = $true)])
• enabling choco to display an auto-generated package parameter help, similar to what Get-Help does for cmdlets and scripts

The syntax is also terse - there is no imperative boilerplate code and nearly all text is actually meaningful. The parameter declarations are quite readable, so chocolateyInstall.ps1 becomes very much self-documenting, perhaps even eliminating the need for the package author to write separate package parameter documentation in the package description.

Implementation considerations:
The packageParameters parsing logic would be executed directly inside choco.exe, prior to calling the script. Choco could then prepare a hashtable with parameter values and call chocolateyInstall.ps1 passing the parameters simply by splatting.
To support existing packages with custom parameter syntax and parsing logic, choco could perform the above only after examining the beginning of chocolateyInstall.ps1 and finding the parameter declaration block.
Chocolatey entry points other than choco.exe (the future PowerShell module, custom applications based on chocolatey.dll) could also be allowed to pass package parameters in a hashtable, completely avoiding the (sometimes brittle) string parsing problem. GUI applications could generate and display a form for entering the parameters, similar to what the IIS console does for WebDeploy package parameters.
Chocolatey could also provide a few predefined parameters, which scripts could opt-in to receive by declaring them. One of them could be $RawPackageParameters (for packages insisting on doing the parsing themselves), others could replace the environment variables that are currently being set by Chocolatey.

One last point - unlike adding a new helper (which is a breaking change), this approach would allow packages to function even on legacy versions of Chocolatey, with graceful degradation of functionality (package parameters always set to default values) instead of hard installation errors due to missing helpers.

[DarwinJS]

I would love nothing more than to use PowerShell's native parameters - but I'm not understanding how this would have backward compatibility - wouldn't chocolatey need to take the current -params and pass it along properly formatted -which means a future version of Chocolatey?

I also don't understand how a new helper is a breaking change - if someone isn't using it, they just keep using the older template code that has been provided.

• There are also already ways to get the raw parameters in PowerShell:
  $MyInvocation.BoundParameters.GetEnumerator()
  $MyInvocation.UnboundArguments
• To keep excess parameters from causing a syntax error I have generally used an argument that receives the remaining undeclared arguments:[Parameter(Mandatory=$False,ValueFromPipeline=$True,ValueFromRemainingArguments=$True)][string[]]$ExtraArgs

[jberezanski]

I'm not understanding how this would have backward compatibility - wouldn't chocolatey need to take the current -params and pass it along properly formatted -which means a future version of Chocolatey?

Taking full advantage of this feature would certainly require support from Chocolatey, so it does mean a new version, yes. Let me describe the compatibility scenarios:

1. Future Chocolatey, future package with parameter declarations

Chocolatey parses the parameters and passes them to the package. No surprise there.

2. Future Chocolatey, unmodified existing package (which does not declare parameters) with custom $Env:chocolateyInstallArguments parsing logic

Chocolatey recognizes the package as a legacy one and does not parse packageParameters, but passes the string unmodified to the package via $Env:chocolateyInstallArguments, like current Chocolatey does.

3. Future Chocolatey, future package which declares a [string] $ChocolateyRawPackageParameters parameter

Chocolatey recognizes the package as one that takes parameter parsing responsibility upon itself. Chocolatey does not parse packageParameters, but passes the string unmodified to the package as the value of the $ChocolateyRawPackageParameters parameter. This provides an easy upgrade path for existing packages with custom parsing logic.

4. Current/earlier Chocolatey, unmodified existing package with custom packageParameters parsing logic

No change from current behavior (obviously). Chocolatey passes the parameter string unmodified to the package.

5. Current/earlier Chocolatey, future package with parameter declarations and no code looking at $Env:chocolateyInstallArguments

Chocolatey passes packageParameters as $Env:chocolateyInstallArguments. The package installs using default parameter values.

6. Current/earlier Chocolatey, future package with parameter declarations and additional backward compatiblity code parsing $Env:chocolateyInstallArguments

Chocolatey passes packageParameters as $Env:chocolateyInstallArguments. The package recognizes it is being installed by a legacy Chocolatey and parses the parameter values out of $Env:chocolateyInstallArguments. This only makes sense if the parsing code in the package is fully compatible with the logic used by future Chocolatey. Under that condition, it would be possible to create parameterized packages supporting both current/legacy and future parameter passing models, which might be useful in some private scenarios (however, I think most folks would be okay with simply upgrading Chocolatey).

I also don't understand how a new helper is a breaking change - if someone isn't using it, they just keep using the older template code that has been provided.

True, however, that's not what I meant (I've been really unclear). When a package does use a new helper, it ceases to be compatible with earlier versions of Chocolatey - the missing helper will cause the package installation to fail. On the other hand, with parameter declarations, as stated in 5. above, the package will install under earlier Chocolatey (using default parameter values), making it a graceful reduction of functionality instead of a complete breakage.

To keep excess parameters from causing a syntax error I have generally used an argument that receives the remaining undeclared arguments

They only cause a syntax error if the script is decorated with [CmdletBinding()], but in those cases - yes, an elegant technique.

There are also already ways to get the raw parameters in PowerShell

Right, a.k.a. $PSBoundParameters and $args, respectively.

[mattheyan]

Hey Guys,

I see this hasn't been updated in a while, but I'd like to weigh in on this, since I feel that package parameters are an area where Chocolatey could make things a lot easier on package authors. I think the --params option already is a nice foundation provided by Chocolatey, but it would be nice if there was a "happy path". :)

First, I'd like to point out that it isn't completely necessary that a parameter parsing helper be included in Chocolatey out of the box.

1. Chocolatey supports extension packages
2. Command-line parameter formats are not universally agreed upon
3. The --params arg is already out there in the wild (legacy factor)

Personally I vote for working on an extension package or packages, at least initially until some level of consensus develops. We can experiment with formats and approaches, and theoretically the best approach(es) will bubble up to the top via real-world usage. But, I also think it would be useful if the conversation about the way the params are parsed could continue rather than just dissipating into the void, i.e. there are factors that someone like Rob can provide insight and guidance on, especially if the goal would be to put something in choco at some point...

I really like the idea from @jberezanski about supported PowerShell script parameters in chocolateyInstall.ps1 (and I'd love to hear Rob's take on it). However, I suspect it would be best to treat this as separate from the '--params' argument. Here's one idea... It isn't all that uncommon for a command-line tool that will call other tools or scripts to have to deal with passing through arguments. One approach out there is the linux form of using -- to designate that the remaining arguments should be passed directly to an inner command.

PS> choco install -y VS2013 -- -Features Blend,LightSwitch

In addition to the benefits already laid out:

• This is not re-inventing the wheel, but is based on an existing convention.
• Testing install scripts could be slightly easier using this approach.
• In terms of discoverability, someone can look at the chocolateyinstall.ps script on the gallery site and immediately know how to pass args.

This would have to be something that chocolatey supported, but wouldn't involve the --params argument at all. Also, theoretically non-PS based packages could accept different format of args and/or parse them in a different way (just thinking out loud).

In conclusion, it would be nice if both approaches, --params and PS script params, could be supported.

Thoughts?

-Bryan

[ferventcoder]

I have not forgotten about this - I'm just focused on 0.9.10.0 and getting it out the door. This will come after that, so I will be back to discuss then. 👍

[mrpullen]

I have been using a settings.psd1 file to store all my defaults parameter settings, then overriding the defaults with the --params. But if we define the "expected" settings or parameters couldn't you just parse the choco install {{packagename}} command string to grab the values defined? That way we don't have to change the existing convention, we just need to store the choco install command into an $env variable then parse it using a helper and providing a settings.psd1 file.. And if that file doesn't exist, you just don't go looking..

settings.psd1 file
@{
InstallPath = "C:\Program Files\mypackage"
AddDesktopShortcut = $true
AnotherDefaultSetting = "somevalue"
}

choco install mypackage -InstallPath "C:\mypackage" -AddDesktopShortcut false

so that we would just parse the "choco install mypackage -InstallPath "C:\mypackage" -AddDesktopShortcut false" string looking for key values from settings.psd1 and a regular expression that supported -key "value" | -key 'value' | key value or any other desired formatting.

[123BLiN]

Hello, just a proposition if method from here https://github.com/chocolatey/choco/wiki/How-To-Parse-PackageParameters-Argument will stay for a long time.
I propose to use list of tuples instead of hashtable.
Something like:

 $arguments = New-Object System.Collections.Generic.List[System.Object]

  # Let's assume that the input string is something like this, and we will use a Regular Expression to parse the values
  # /Port:81 /Edition:LicenseKey /AdditionalTools

  # Now we can use the $env:chocolateyPackageParameters inside the Chocolatey package
  $packageParameters = $env:chocolateyPackageParameters

  # Default the values
  $port = "81"
  $edition = "LicenseKey"
  $additionalTools = $false
  $installationPath = "c:\temp"

  # Now parse the packageParameters using good old regular expression
  if ($packageParameters) {
      $match_pattern = "\/(?<option>([a-zA-Z]+)):(?<value>([`"'])?([a-zA-Z0-9- _\\:\.]+)([`"'])?)|\/(?<option>([a-zA-Z]+))"
      $option_name = 'option'
      $value_name = 'value'

      if ($packageParameters -match $match_pattern ){
          $results = $packageParameters | Select-String $match_pattern -AllMatches
          $results.matches | % {
            $tuple = New-Object "tuple[String, String]"  $_.Groups[$option_name].Value.Trim(), $_.Groups[$value_name].Value.Trim())
            $arguments.Add($tuple)
        }
      }
      else
      {
          Throw "Package Parameters were found but were invalid (REGEX Failure)"
      }

This should allow to have more than one variable with the same name, for example consul -config-dir option. (https://www.consul.io/docs/agent/options.html#_config_dir)

[onyxhat]

It seems like a lot of complexity would be removed if $packageParameters wasn't cast as a string; allowing for structured data - which seems like what everyone is trying to work with as an end result.

[ferventcoder]

@onyxhat agreed. it's the technical details that get you though. Part of what this is doing is taking that string passed to a CLI application (choco.exe) and providing it in the same format, so that folks can decide what to do with it in packaging. We are looking to add a helper to take that string and convert it to structured data for folks.

One of the other aspects is passing the parameter in as structured data as a script parameter.

[onyxhat]

Ended up implementing the following. Didn't want to be opinionated about the style of switching - so just reduced the parser to take the string apart into k/v pairs that create locally scoped variables. That way you can pass it around however you like however your installer or process needs to use them.

#$packageParameters = $env:chocolateyPackageParameters
$packageParameters = "var1 = val1;var2=`"val2 with spaces -`""
$match_pattern = "(?<key>(\w+))\s*=\s*(?<value>([`"'])?([\w- _\\:\.]+)([`"'])?)"

# Now parse the packageParameters using good old regular expression
if ($packageParameters -match $match_pattern ){
    $results = $packageParameters | Select-String $match_pattern -AllMatches
    $results.matches | % {
        Set-Variable -Name $_.Groups['key'].Value.Trim() -Value $_.Groups['value'].Value.Trim()
    }
}

[ferventcoder]

@onyxhat very similar to https://github.com/chocolatey/choco/wiki/How-To-Parse-PackageParameters-Argument, except you are doing this with "=" as the delimiter. 👍

[ferventcoder]

Although, you are creating variables with yours... hmmm interesting.

[onyxhat]

@ferventcoder Yeah, that's precisely what I started from as a template; just trimmed it down to the bare essentials and modified just to spin up variables. On the off chance someone prefers to pad declarations with spaces I allowed for that as well.

[dragon788]

Definitely finding this discussion interesting as I look at modifying a package installer to handle parameters that modify how it installs.

Is PowerShell's switch statement compatible with PS2 or whatever Chocolatey targets?

Just curious because in that case you could simply set the values in a switch statement instead of all the if statements in the wiki example.

[ferventcoder]

Just an update to this discussion:

• Get-PackageParameters was implemented in Chocolatey-Core.extension a long while back.
• Function - Get-PackageParameters #1393 brought the same to straight Chocolatey, allowing also for = and any value without a space or : or = to be the item key
• Get-PackageParameters should handle urls #1459 will provide further fixes to allow for urls are parameter values (e.g. /url:https://somewhere.com)

[ferventcoder]

There is also a bug in the way that the Chocolatey core extension module loads up, in that it unconditionally overrides the built-in one. chocolatey-community/chocolatey-packages#898

[ferventcoder]

• Get-PackageParameters: force built-in method to be preferred over chocolatey-core.extension method #1476 fixes the bug for the override
• Get-PackageParameters - Resolve differences between chocolatey.core-extension and built-in method #1490 was another fix to be sure that the built-in is at least compatible and draws on more.

@jberezanski as this conversation went all kinds of ways - the core of what this issue was about has been covered and this can be closed. I would like to capture what you are hoping for - PowerShell parameters for Chocolatey automation scripts into another issue. Would you be able to split that out and reference the points here (@mattheyan and @DarwinJS had some really good thoughts in here as well to expand on your notes).