Map variables experiment

[pd93]

Warning

All experimental features are subject to breaking changes and/or removal at any time. We strongly recommend that you do not use these features in a production environment. They are intended for testing and feedback only.

Context

This experiment attempts to solve the problems originally described by #140. It is a follow-up to the "Any Variables" experiment (#1415) which was merged without support for maps.

Currently, all variable types are allowed except for maps. This is because there is some debate around the syntax that should be used to define them. The proposals for these syntaxes are described below.

Proposal 1

Warning

This experiment proposal breaks the following functionality:

• Dynamically defined variables (using the sh keyword)

Note

To enable this experiment, set the environment variable: TASK_X_MAP_VARIABLES=1. Check out [our guide to enabling experiments][enabling-experiments] for more information.

This proposal removes support for the sh keyword in favour of a new syntax for
dynamically defined variables, This allows you to define a map directly as you
would for any other type:

version: 3

tasks:
  foo:
    vars:
      FOO: {a: 1, b: 2, c: 3} # <-- Directly defined map on the `FOO` key
    cmds:
      - 'echo {{.FOO.a}}'

Migration

Taskfiles with dynamically defined variables via the sh subkey will no longer
work with this experiment enabled. In order to keep using dynamically defined
variables, you will need to migrate your Taskfile to use the new syntax.

Previously, you might have defined a dynamic variable like this:

version: 3

tasks:
  foo:
    vars:
      CALCULATED_VAR:
        sh: 'echo hello'
    cmds:
      - 'echo {{.CALCULATED_VAR}}'

With this experiment enabled, you will need to remove the sh subkey and define
your command as a string that begins with a $. This will instruct Task to
interpret the string as a command instead of a literal value and the variable
will be populated with the output of the command. For example:

version: 3

tasks:
  foo:
    vars:
      CALCULATED_VAR: '$echo hello'
    cmds:
      - 'echo {{.CALCULATED_VAR}}'

If your current Taskfile contains a string variable that begins with a $, you
will now need to escape the $ with a backslash (\) to stop Task from
executing it as a command.

Proposal 2

Note

To enable this experiment, set the environment variable: TASK_X_MAP_VARIABLES=2. Check out [our guide to enabling experiments][enabling-experiments] for more information.

This proposal maintains backwards-compatibility and the sh subkey and adds another new map subkey for defining map variables:

version: 3

tasks:
  foo:
    vars:
      FOO:
        map: {a: 1, b: 2, c: 3} # <-- Defined using the `map' subkey instead of directly on 'FOO'
      BAR: true # <-- Other types of variables are still defined directly on the key
      BAZ:
        sh: 'echo Hello Task' # <-- The `sh` subkey is still supported
    cmds:
      - 'echo {{.FOO.a}}'

Parsing JSON and YAML

In addition to the new map keyword, this proposal also adds support for the json and yaml keywords for parsing JSON and YAML strings into real objects/arrays. This is similar to the fromJSON template function, but means that you only have to parse the JSON/YAML once when you declare the variable,
instead of every time you want to access a value.

Before:

version: 3

tasks:
  foo:
    vars:
      FOO: '{"a": 1, "b": 2, "c": 3}' # <-- JSON string
    cmds:
      - 'echo {{(fromJSON .FOO).a}}' # <-- Parse JSON string every time you want to access a value
      - 'echo {{(fromJSON .FOO).b}}'

After:

version: 3

tasks:
  foo:
    vars:
      FOO:
        json: '{"a": 1, "b": 2, "c": 3}' # <-- JSON string parsed once
    cmds:
      - 'echo {{.FOO.a}}' # <-- Access values directly
      - 'echo {{.FOO.b}}'

Variables by reference

Lastly, this proposal adds support for defining and passing variables by reference. This is really important now that variables can be types other than a string.

Previously, to send a variable from one task to another, you would have to use the templating system. Unfortunately, the templater always outputs a string and operations on the passed variable may not have behaved as expected. With this proposal, you can now pass variables by reference using the ref subkey:

Before:

version: 3

tasks:
  foo:
    vars:
      FOO: [A, B, C] # <-- FOO is defined as an array
    cmds:
      - task: bar
        vars:
          FOO: '{{.FOO}}' # <-- FOO gets converted to a string when passed to bar
  bar:
    cmds:
      - 'echo {{index .FOO 0}}' # <-- FOO is a string so the task outputs '91' which is the ASCII code for '[' instead of the expected 'A'

After:

version: 3

tasks:
  foo:
    vars:
      FOO: [A, B, C] # <-- FOO is defined as an array
    cmds:
      - task: bar
        vars:
          FOO:
            ref: .FOO # <-- FOO gets passed by reference to bar and maintains its type
  bar:
    cmds:
      - 'echo {{index .FOO 0}}' # <-- FOO is still a map so the task outputs 'A' as expected

This means that the type of the variable is maintained when it is passed to another Task. This also works the same way when calling deps and when defining a variable and can be used in any combination:

version: 3

tasks:
  foo:
    vars:
      FOO: [A, B, C] # <-- FOO is defined as an array
      BAR:
        ref: .FOO # <-- BAR is defined as a reference to FOO
    deps:
      - task: bar
        vars:
          BAR:
            ref: .BAR # <-- BAR gets passed by reference to bar and maintains its type
  bar:
    cmds:
      - 'echo {{index .BAR 0}}' # <-- BAR still refers to FOO so the task outputs 'A'

All references use the same templating syntax as regular templates, so in
addition to simply calling .FOO, you can also pass subkeys (.FOO.BAR) or
indexes (index .FOO 0) and use functions (len .FOO):

version: 3

tasks:
  foo:
    vars:
      FOO: [A, B, C] # <-- FOO is defined as an array
    cmds:
      - task: bar
        vars:
          FOO:
            ref: index .FOO 0 # <-- The element at index 0 is passed by reference to bar
  bar:
    cmds:
      - 'echo {{.MYVAR}}' # <-- FOO is just the letter 'A'

Looping over maps (Both proposals)

This experiment also adds support for looping over maps using the for keyword, just like arrays. In addition to the {{.ITEM}} variable being populated when looping over a map, we also make an additional {{.KEY}} variable available that holds the string value of the map key.

Proposal 1

version: 3

tasks:
  foo:
    vars:
      MAP: {a: 1, b: 2, c: 3}
    cmds:
      - for:
          var: MAP
        cmd: 'echo "{{.KEY}}: {{.ITEM}}"'

Proposal 2

version: 3

tasks:
  foo:
    vars:
      map:
        MAP: {a: 1, b: 2, c: 3}
    cmds:
      - for:
          var: MAP
        cmd: 'echo "{{.KEY}}: {{.ITEM}}"'

Note

Remember that maps are unordered, so the order in which the items are looped over is random.

[task-bot]

This issue has been marked as an experiment proposal! 🧪 It will now enter a period of consultation during which we encourage the community to provide feedback on the proposed design. Please see the experiment workflow documentation for more information on how we release experiments.

[simonrouse9461]

Is there a way to do templating in map variables?
For example,

version: 3

tasks:
  foo:
    vars:
      PREFIX: prefix_
      MAP:
        map: 
          a: '{{.PREFIX}}1'
          b: '{{.PREFIX}}2'
          c: '{{.PREFIX}}3'
    cmds:
      - echo {{index .MAP "a"}}

This will print {{.PREFIX}}1 instead of prefix_1.

[pd93]

@simonrouse9461 See #1526 and #1544. Your example works in the latest release. Also worth noting that TASK_X_MAP_VARIABLES won't replace TASK_X_ANY_VARIABLES until the next release.

[simonrouse9461]

@pd93 Thanks! Upgrading to the latest release solved the problem.

[simonrouse9461]

Suggestion:
For the ref variables, is it possible to reuse the template engine syntax? For example,

version: 3

tasks:
  foo:
    requires:
      vars: [VAR_NAME]
    vars:
      VAR_MAP: 
        map:
          FOO: [1, 2, 3]
          BAR: [4, 5, 6]
    cmds:
      - task: bar
        vars:
          VAR:
            ref: index .VAR_MAP .VAR_NAME
  bar:
    cmds:
      - echo {{index .VAR 0}}

Then,

• task foo VAR_NAME=FOO will print 1
• task foo VAR_NAME=BAR will print 4

This will make things more flexible and programmable.

[pd93]

is it possible to reuse the template engine syntax

@simonrouse9461 I was initially going to reply saying that this is quite difficult as the text/template package doesn't surface the ability to resolve references without converting the result to a string. However, after some thought, I have created #1612 to solve this.

[task-bot]

This experiment has been marked as a draft! ✨ This means that an initial implementation has been added to the latest release of Task! You can find information about this experiment and how to enable it in our experiments documentation. Please see the experiment workflow documentation for more information on how we release experiments.

[steffans]

Suggestion:

Like proposal 1 but use YAML custom tags in order to define sh or ref, instead of prefixes $ or #. Allows to also use sh or ref for nested map values. See example here:

version: 3

tasks:
  foo:
    vars:
      FOO: {a: 1, b: 2, c: 3} # <-- Directly defined map on the `FOO` key
      BAR: !sh 'echo Hello Task' # <-- Use a custom tag for a sh
      BAZ: !ref '.BAR' # <-- Use a custom tag for a ref
      MAP_WITH_TAGS: {a: 1, b: !ref '.BAR', c: !sh 'echo 2'} # <-- Use sh or ref in defined map
    cmds:
      - 'echo {{.FOO.a}}'

As an alternative to maintain backwards-compatibility also use YAML custom tags to define data types like map. This would still support tags like sh or ref see example here:

version: 3

tasks:
  foo:
    vars:
      FOO: !map {a: 1, b: 2, c: 3} # <-- Use a custom tag for map data type
      BAR: !sh 'echo Hello Task' # <-- Use a custom tag for a sh
      BAZ: !ref '.BAR' # <-- Use a custom tag for a ref
    cmds:
      - 'echo {{.FOO.a}}'

See also:

• https://github.com/eemeli/yaml/blob/main/docs/06_custom_tags.md
• https://symfony.com/doc/current/components/yaml.html#handling-invalid-types

For example Symfony YAML uses custom tags to add native PHP types like !php/object, !php/const, !php/enum, etc.

[simonrouse9461]

@pd93 I really like @steffans's idea. Is there a chance to have a Proposal 3 for this?

[pd93]

@steffans, @simonrouse9461, I took a look at this today. It's an interesting proposal. It definitely has some clear advantages over the other 2 proposals. However after some (very limited) research, I can see some drawbacks too:

1. Lack of familiarity with the syntax (This is the first time I've used it after many years of handling YAML files). YAML has a very low bar to entry. It's one of the giant benefits of using Task over make and other task tools IMO. Does this increase that bar?
2. JSON schema (also used by the very popular YAML VS Code extension) doesn't support custom tags as far as I can see - this leads to confusing errors in the Taskfile. To solve this users are required to add the custom tags to their VS Code settings.
3. !map is extremely similar to the already defined !!map (which we can't use for the same reason that we can't directly define a map in proposal 1). I don't fancy repeatedly trying to explain the difference to confused users.

[JonZeolla]

I've been taking a poke at this very briefly but it made me wonder if there would be interest in something that would support using the output of a sh: into a map (proposal 2). This would allow me to create complex tasks or scripts that generate valid JSON for consumption by task (something I already do regularly for GitHub Actions matrixes).

For instance:

---
version: 3

tasks:
  foo:
    vars:
      BAZ:
        sh: 'echo "{a: 1, b: 2, c: 3}"'
      FOO:
        map: '{{.BAZ}}'
    cmds:
      - 'echo {{.FOO.a}}'

Or, a more streamlined:

---
version: 3

tasks:
  foo:
    vars:
      FOO:
        map:
          sh: 'echo "{a: 1, b: 2, c: 3}"'
    cmds:
      - 'echo {{.FOO.a}}'

[quirin-buechner-mdctec]

I have another suggestion:
Introduce a new string-template function named "sh":

version: 3
vars:
  PACKAGE_VERSION_TAG: 'v{{sh "pwsh -C"}}(gc ./package.json | ConvertFrom-Json -AsHashtable).version{{end}}' 

tasks:
  git:tag:package-version:
    desc: 'Create a git tag based on the version defined in package.json. The version value is prepended with "v"'
    cmd: git tag '{{.PACKAGE_VERSION_TAG}}'

Advantages:

• Allows to specify the exact shell executable including parameters.
• No need for extra 'sh' key in a var entry. Map variables can be defined in yaml directly like in proposal 1.
• Only uses familiar syntax. (No extra !map or $... expansion)
• The output value can be used anywhere inside a string. (In the example above, the string from stdout is easily prepended with v)

Adressing @JonZeolla's post: also post-processing of the shell output seems quite simple using the pipe syntax.:

tasks:
  foo:
    vars:
      FOO: '{{sh | fromJson}}echo "{a: 1, b: 2, c: 3}"{{end}}'
    cmds:
      - 'echo {{.FOO.a}}'

@pd93 Is there a chance to have a Proposal 3 for this?

[pd93]

@JonZeolla What you propose in your comment is actually already possible today by using ref and fromJson:

version: 3

tasks:
  foo:
    vars:
      FOO: '{"a": 1, "b": 2, "c": 3}'
      BAR:
        ref: 'fromJson .FOO'
    cmds:
      - 'echo {{.BAR.a}}'

I don't think we need a new keyword to deal with this.

@quirin-buechner-mdctec the same thing applies to your comment but it is worth noting that it is impossible to output anything other than a string from a template. If your var contains {{}} the output will always be a string. This is why ref was created (so you could preserve the type of a templated variable).

Most of the time, users will want to define maps statically, not dynamically. As such, any new proposal for defining maps would need to allow a user to define a static map object in YAML or its not worth having IMO. Converting strings to a map dynamically is already possible as shown above.