`IncludeGroup` to include an external configuration

[SteveL-MSFT]

Summary of the new feature / enhancement

Not sure yet if this is a good idea as I can see the main downside that you don't get a complete view of the configuration in one doc, but there might be use cases where you define separate config docs for network, storage, compute (as examples) and you want a higher level doc that applies all of them. So it could look like:

resources:
- name: networking
  type: Microsoft.DSC/Include
  properties:
    configurationFile: ./networking.yaml
- name: storage
  type: Microsoft.DSC/Include
  properties:
    configurationFile: ./storage.yaml

Proposed technical implementation details (optional)

No response

[acangialosi]

One use case for this is configuring development environments like dev box where there are a common set of configurations used across multiple workloads a team or an individual developer has defined. Dev Teams often standardize some environment specific configurations like enabling dev mode, showing hidden files, enabling hyper-v, or installing VS Code across multiple project types.
Then they have workload or project specific environment requirements. One project focused on intelligent Apps needs a specific version of python installed and the Python extension installed.
Another project is a .net project that installs .NET Core and the C# DevKit extension for VS Code.
Being able to DependsOn across files also useful.

common-dev-env.yaml
    - resource: Microsoft.Windows.Developer/DeveloperMode
    - resource: Microsoft.Windows.Developer/WindowsExplorer
      directives:
        allowPrerelease: true
      settings:
        FileExtensions: Show
        HiddenFiles: Show
    - resource: Microsoft.WinGet.DSC/WinGetPackage
      id: vscode
      directives:
        allowPrerelease: true
      settings:
        id: Microsoft.VisualStudioCode

intelligent-app.yaml
    resources:
    - name: commondevenv
      type: Microsoft.DSC/Include
      properties:
        configurationFile: ./common-dev-env.yaml
    - resource: Microsoft.WinGet.DSC/WinGetPackage
      id: python311
      directives:
        allowPrerelease: true
      settings:
        id: Python.Python.3.11
    - resource: PSDscResources/Script
      id: VSCodemspython
      dependsOn:
        - python311
        - vscode
      settings:
        SetScript: |
          code --install-extension ms-python.python
        GetScript: |
          return $false
        TestScript: |
          return $false

dotnet-app.yaml
    resources:
    - name: commondevenv
      type: Microsoft.DSC/Include
      properties:
        configurationFile: ./common-dev-env.yaml
    - resource: Microsoft.WinGet.DSC/WinGetPackage
      id: dotnet8
      directives:
        allowPrerelease: true
      settings:
        id: Microsoft.DotNet.SDK.8
    - resource: PSDscResources/Script
      id: VSCodemspython
      dependsOn:
        - dotnet8
        - vscode
      settings:
        SetScript: |
          code --install-extension ms-dotnettools.csdevkit
        GetScript: |
          return $false
        TestScript: |
          return $false

[michaeltlombardi]

I think this is a highly functional group to add to DSC, but I would also like the ability to return the expanded node graph / configuration document, in addition to invoking a DSC operation against it.

Consider the following minimal example configuration documents:

---
# configs/baseline/security.dsc.config.yaml
resources:
- type: Test/Echo
  name: Security Setting
  properties: { output: 'Sets a security value' }
---
# configs/environment/test.dsc.config.yaml
resources:
- type: Microsoft.DSC/Include
  name: Security baseline
  properties:
    configurationFile: >-
      [concat(
        envvar('DSC_CONFIG_ROOT'),
        '/../baseline/security.dsc.config.yaml'
      )]
- type: Test/Echo
  name: Environmental config
  properties: { output: Sets a value for test machines }
  dependsOn:
    - "[resourceId('Microsoft.DSC/Include', 'Security baseline')]"
---
# configs/webserver.dsc.config.yaml
resources:
- type: Microsoft.DSC/Include
  name: Test environment
  properties:
    configurationFile: >-
      [concat(
        envvar('DSC_CONFIG_ROOT'),
        '/environment/test.dsc.config.yaml'
      )]
- type: Test/Echo
  name: Webserver config
  properties: { output: Sets a value for web servers }
  dependsOn:
    - "[resourceId('Microsoft.DSC/Include', 'Test environment')]"

I would want to both be able to invoke DSC operations against the resources in all three configurations, but I'd also want to be able to see what the expanded document looks like. Maybe I could run a command like this:

dsc config resolve --path ./configs/webserver.dsc.config.yaml

To get a response like this¹:

resources:
- type: Microsoft.DSC/Group
  name: Test environment
  properties:
    resources:
    - type: Microsoft.DSC/Group
      name: Security baseline
      properties:
        resources:
        - type: Test/Echo
          name: Security Setting
          properties:
            output: 'Sets a security value'
    - type: Test/Echo
      name: Environmental config
      properties:
        output: Sets a value for test machines
      dependsOn:
      - "[resourceId('Microsoft.DSC/Group', 'Security baseline')]"
- type: Test/Echo
  name: Webserver config
  properties:
    output: Sets a value for web servers

I could save that output, similar to the export command, to have a new configuration document to work from. It would be even better if that document included metadata for the available/required resources and contexts, a way for me to turn my general configuration document into a more reproducible, explicit document. This would also be highly useful for other tools, like a visualization tool to explore the node graph my configuration defines.

For other operations, even without resolve, I would expect a Microsoft.DSC/Include group to work just like a Microsoft.DSC/Group resource instance.

An alternate and possibly better UX/DevX, would be to extend the Microsoft.DSC/Group resource to support an configurationFile property (possibly mutually exclusive with resources), like:

resources:
- name: networking
  type: Microsoft.DSC/Group
  properties:
    configurationFile: ./networking.yaml
- name: storage
  type: Microsoft.DSC/Group
  properties:
    configurationFile: ./storage.yaml

So for a resolve command the resource type stays the same, but properties are updated to reflect the included configurations.

Footnotes

1. The example output leaves out the other properties for a configuration, like $schema, but more importantly, we may need to consider propagating variables, parameters, and metadata. ↩

[SteveL-MSFT]

I think --whatif will kind of give you what you want with --resolve, but I don't see any reason not to have --resolve output the uber config with inlined included configurations.

I don't think there's any real benefit of having this capability encapsulated within Microsoft.DSC/Group rather than a new Microsoft.DSC/include since it'll be implemented by dsc itself. Personally, I prefer the Microsoft.DSC/Include as it's more explicit about what's happening. You can always put an Include within a Group if there's a need for it.

Finally, dependsOn would only work within an included configuration and not across, but you can have an entire included config depend on another group or resource at the parent level.

[SteveL-MSFT]

Created separate issue for resolve as it can be done later

[SteveL-MSFT]

DSC_CONFIG_ROOT won't be required as paths will be relative to that root. In the case the input is not a file, both DSC_CONFIG_ROOT and CWD will be the current directory. We should also explicitly disallow traversing up the parent so .. will not be allowed at all and will error.