Dev container feature ordering, multiple features in same repository

[edgonmsft]

Exploring more widespread usage of features as they are implemented today a couple of interesting scenarios came up:

• Feature writing feels a little more complicated than it feels it should be. Some features that exist today have checks or code to guard against the features not executing in a predictable manner. This also complicates having features like user creation that a user could define as the first or last part of the execution.
• The more atomic a feature was the easier it was to reuse it in different scenarios.
• Allow features to be referenced multiple times in a devcontainer.json so that it's possible to execute them multiple times with different parameters. Situations came up where it would have been useful to execute a feature multiple times to install multiple versions of an sdk. This would also simplify the feature itself, avoiding the need to write it in a way that can loop.
• Having the feature source code in a repository also allowed for faster iteration and can even help with using features to implement repo specific scripts.

Proposal

Thinking of this situation the proposal is:

• Convert features in devcontainer.json to an array so that it's possible to control the order of execution and to define multiple executions of the same feature.
• Add a source marker to the feature, that would allow the source code of the feature be part of the repository or the filesystem.
• Make features more independent, meaning all code is in the same folder (or subfolders).
• Define a feature.json, similar to how GitHub Actions work, where we could eventually support different ways features could be executed besides the install.sh supported today.

Questions

I think the biggest question is to define how we want to handle backwards compatibility with the way features are implemented and defined today.

[Chuxel]

For future reference: I think this was partly broken out from #8, correct?

Add a source marker to the feature, that would allow the source code of the feature be part of the repository or the filesystem.

By this, do you mean that you do not need to implement install.sh when you are dealing with multiple features in a collection? Assuming so, there's also #21 which defines multiple potential scripts down the road. Would the pointer to the source here be to a folder? (Or put another way, what are you thinking the folder structure would look like?)

I think the biggest question is to define how we want to handle backwards compatibility with the way features are implemented and defined today.

By this do you mean, the current object-based syntax (non-array) in devcontainer.json? If so, I think the question is whether we just convert to an array behind the scenes and keep it with an assumed alphabetical order, or have tools / services that support the old format do a "squiggle" to indicate its deprecated and will be removed somewhere along the way.

[edgonmsft]

@Chuxel Yeah I just felt that some are not as explicit and I thought a discussion on the specific points.

By this, do you mean that you do not need to implement install.sh when you are dealing with multiple features in a collection? Assuming so, there's also #21 which defines multiple potential scripts down the road. Would the pointer to the source here be to a folder? (Or put another way, what are you thinking the folder structure would look like?)

For this I was thinking something closer to how Actions work. Meaning that we can have a feature.json file in the root of the feature that defines what scripts or applications are to be used for each of the lifecycle scripts. It would still be a good idea to say that if that file is not present the default is install.sh, etc. That way the configuration is only needed if users need to do something different than the default.

For the file structure I was thinking something like:

collectionFolder
+-- features-collection.json
+-- common (or similar name)
|    +-- (library files)
+-- feature1
|    +-- feature.json
|    +-- install.sh (or others)
|    +-- (other files)
+-- feature2
(etc)

Like proposed in https://github.com/microsoft/dev-container-spec //issues/7 the idea would be that we can still distribute groups of features as a single collection, on 1 tar file. This way library scripts or data common to a group of features would still be distributed with them. But from the point of the spec each feature would be parsed/executed separately, in its own layer and is considered independent. The 'features-collection.json' file is more of a UX help on what features exist at the moment of searching and where they are located. 'feature.json' would be the file that we would use to execute a particular feature referenced in 'devcontainer.json'

By this do you mean, the current object-based syntax (non-array) in devcontainer.json? If so, I think the question is whether we just convert to an array behind the scenes and keep it with an assumed alphabetical order, or have tools / services that support the old format do a "squiggle" to indicate its deprecated and will be removed somewhere along the way.

I like the second idea since it maintains things working as today and supports the array syntax moving forward.

[Chuxel]

Got it - that makes sense and would be compatible with #21. Having one giant file right now does get cumbersome for editing. features.json then could be closer to devcontainer.json in its format without that getting out of hand. e.g. extensions and settings should probably be under customizations to mirror #1. Thoughts @chrmarti @bamurtaugh ?

[bamurtaugh]

Using an array in the way described above makes sense, as it seems to make config feel more manageable.

I'm trying to discern all of the features files that could exist in a project. I have the following understanding:

• devcontainer-features.json: list of all features in a project, with metadata for each
• install.sh: entry point to install features

How do feature.json and features-collection.json tie in? Is features-collection.json an alternative option to or new name for devcontainer-features.json? Do you have an example of what a feature.json would look like compared to the other files here?

[edgonmsft]

The way I think about it is that we have files required to execute each feature and files for organization, UX and search of the available features.

The special case is when we have libraries of scripts that are shared by different features.

Concept

Feature

• A feature is a folder with or without a feature.json file in it.
• It is considered a single unit from the point of view of its execution, with one entrypoint for each lifecycle event.
• Each feature would be executed in a single layer of the build process.
• Features should be as simple as possible to facilitate reuse and composition.
• If there is a feature.json it should contain everything required to execute it and display it to the user.
• In principle this should be all that is needed for an application that implements the spec to successfully execute it.

Structure

+-- feature
| +-- feature.json
| +-- validate.sh (default)
| +-- aquire.sh (default)
| +-- install.sh (default)
| +-- (other files)

The filenames would be the default names if a feature.json is not present or does not contain that information.

feature.json

Property	Type	Description
id	string	Id of the feature/definition. The ids should be unique in the contest of the repository where they exist.
name	string	Name of the feature/definition
description	string	Description of the feature/definition
metadata	any	Freeform data added by users for their own purposes.
keywords	array	List of keywords relevant to a user that would search for this definition/feature.
validate.app	string	App to execute. (Optional, defaults to /bin/sh)
validate.parameter	string	Parameters/script to execute. (Defaults to validate.sh)
aquire.app	string	App to execute. (Optional, defaults to /bin/sh)
aquire.parameter	string	Parameters/script to execute. (Defaults to aquire.sh)
configure.app	string	App to execute. (Optional, defaults to /bin/sh)
configure.parameter	string	Parameters/script to execute. (Defaults to install.sh)
options	object	Possible options to be passed as environment variables to the execution of the scripts

Options

Property	Type	Description
id	string	Id of the option
id.type	string	Type of the option
id.enum	string array	possible values
id.default	string	default value for the option
id.description	string	Description for the option

In this way feature.json becomes the complete information about the feature itself.

Feature Collection

In the case where the intent is to have a library of scripts common to a group of features or the intent is to distribute them as a single package then the structure can be one folder up and include a feature-collection.json file.

Structure

collectionFolder
+-- features-collection.json
+-- common (or similar name)
| +-- (library files)
+-- feature1
| +-- feature.json
| +-- install.sh (or others)
| +-- (other files)
+-- feature2
(etc)

feature-collection.json

The structure of this file could be as proposed here #15 and in a way it can be generated automatically by appending each feature.json file for each feature.

Strictly speaking its not required for execution, since the application, through the id can identify the particular feature its is looking for and execute it, but it can serve as the information included with a release so that applications can show this information to the users prior to downloading the package.

devcontainer.json

For devcontainer.json the changes would be to support the features tag as an object or as an array.

In the case of an array, then the structure would be:

Property	Type	Description
id	string	reference to the particular feature to be included.
options	object	Type of the option

ID can be any of the following:

Type	Description
feature	Included features with the application.
organization/repository/{feature or collection}/@Version	Reference to a particular release
https://<../URI/..>/devcontainer-features.tgz#{feature}	Direct reference to a download.
./{local-path} -or- ../{local-path}	Path relative to devcontainer.json

If we agree I can update the proposal with this changes.

[bamurtaugh]

The explanation above was super helpful, thanks so much for writing it up @edgonmsft.

If we agree I can update the proposal with this changes.

Meaning we'd adopt the bullets under the Proposal header in the original issue here? I think that makes sense.

[edgonmsft]

Yeah I'm transforming it into more of a spec document and merging it with #15 to do 1 document

[joshspicer]

Generally sounds like a good plan to me! I have a few smaller comments, which i'll leave for PR #15 when that is updated :)

[edgonmsft]

Added the proposal here: #27

[bamurtaugh]

Should we close this issue now that #27 is merged?

[joshspicer]

Agreed this should be closed given that:

• We have already merged this proposal as Proposal for Features functionality and spec. #27
• There is further breakout discussion in:
  • Feature installation order #43
  • Installing a dev container feature more than once in a devcontainer.json #44