-
Notifications
You must be signed in to change notification settings - Fork 951
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Composite Run Steps ADR #554
Changes from 42 commits
edbe74a
185ef8c
d30db96
02b6823
91bb0a9
8e42609
45c1514
bc8488e
c712d5d
ef811a4
474f239
0741388
d4a1c00
ac9354f
0d8e0f0
a48fc79
02020f1
bcaed7a
d74a9ab
7e432a5
c5ddde9
dc63a51
225ef90
4170900
292dab3
45458e9
de80742
51f06d4
bba6d10
1caedf1
8ae6dd9
22c8c27
43cd477
82079ae
580cd08
c9eb5df
3ea4aed
5383305
4bc9ce4
d2843e8
b2199ac
9051dfc
c88a953
3166328
e3489c5
9664871
53da2f5
f9d6481
18124bd
1be703b
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,346 @@ | ||
TODO: Change file name to represent the correct PR number (PR is not created yet for this ADR) | ||
|
||
# ADR 054x: Composite Run Steps | ||
|
||
**Date**: 2020-06-17 | ||
|
||
**Status**: Proposed | ||
|
||
**Relevant PR**: https://github.com/actions/runner/pull/549 | ||
|
||
## Context | ||
|
||
Customers want to be able to compose actions from actions (ex: https://github.com/actions/runner/issues/438) | ||
|
||
An important step towards meeting this goal is to build in functionality for actions where users can simply execute any number of steps. | ||
|
||
## Guiding Principles | ||
|
||
We don't want the workflow author to need to know how the internal workings of the action work. Users shouldn't know the internal workings of the composite action (for example, `default.shell` and `default.workingDir` should not be inherited from the workflow file to the action file). When deciding how to design certain parts of composite run steps, we want to think one logical step from the consumer. | ||
|
||
A composite action is treated as **one** individual job step. | ||
|
||
|
||
## Decision | ||
|
||
**In this ADR, we only support running multiple run steps in an Action.** In doing so, we build in support for mapping and flowing the inputs, outputs, and env variables (ex: All nested steps should have access to its parents' input variables and nested steps can overwrite the input variables). | ||
|
||
## Steps | ||
|
||
Example `workflow.yml` | ||
|
||
```yaml | ||
jobs: | ||
build: | ||
runs-on: self-hosted | ||
steps: | ||
- id: step1 | ||
uses: actions/setup-python@v1 | ||
- id: step2 | ||
uses: actions/setup-node@v2 | ||
- uses: actions/checkout@v2 | ||
- uses: user/composite@v1 | ||
- name: workflow step 1 | ||
run: echo hello world 3 | ||
- name: workflow step 2 | ||
run: echo hello world 4 | ||
``` | ||
|
||
Example `user/composite/action.yml` | ||
|
||
```yaml | ||
runs: | ||
using: "composite" | ||
steps: | ||
- run: pip install -r requirements.txt | ||
- run: npm install | ||
``` | ||
|
||
Example Output | ||
|
||
```yaml | ||
[npm installation output] | ||
[pip requirements output] | ||
echo hello world 3 | ||
echo hello world 4 | ||
``` | ||
|
||
We add a token called "composite" which allows our Runner code to process composite actions. By invoking "using: composite", our Runner code then processes the "steps" attribute, converts this template code to a list of steps, and finally runs each run step sequentially. If any step fails and there are no `if` conditions defined, the whole composite action job fails. | ||
|
||
## Inputs | ||
|
||
Example `workflow.yml`: | ||
|
||
```yaml | ||
steps: | ||
- id: foo | ||
uses: user/composite@v1 | ||
with: | ||
your_name: "Octocat" | ||
``` | ||
|
||
Example `user/composite/action.yml`: | ||
|
||
```yaml | ||
inputs: | ||
your_name: | ||
description: 'Your name' | ||
default: 'Ethan' | ||
runs: | ||
using: "composite" | ||
steps: | ||
- run: echo hello ${{ inputs.your_name }} | ||
``` | ||
|
||
Example Output: | ||
|
||
``` | ||
hello Octocat | ||
``` | ||
|
||
Each input variable in the composite action is only viewable in its own scope. | ||
|
||
## Outputs | ||
|
||
Example `workflow.yml`: | ||
|
||
```yaml | ||
... | ||
steps: | ||
- id: foo | ||
uses: user/composite@v1 | ||
- run: echo random-number ${{ steps.foo.outputs.random-number }} | ||
``` | ||
|
||
Example `user/composite/action.yml`: | ||
|
||
```yaml | ||
outputs: | ||
random-number: ${{ steps.random-number-generator.outputs.random-id }} | ||
runs: | ||
using: "composite" | ||
steps: | ||
- id: random-number-generator | ||
run: echo "::set-output name=random-id::$(echo $RANDOM)" | ||
``` | ||
|
||
Example Output: | ||
|
||
``` | ||
::set-output name=my-output::43243 | ||
random-number 43243 | ||
``` | ||
|
||
Each of the output variables from the composite action is viewable from the workflow file that uses the composite action. In other words, every child action output(s) is viewable only by its parent using dot notation (ex `steps.foo.outputs.random-number`). | ||
|
||
Moreover, the output ids are only accessible within the scope where it was defined. Note that in the example above, in our `workflow.yml` file, it should not have access to output id (i.e. `random-id`). The reason why we are doing this is because we don't want to require the workflow author to know the internal workings of the composite action. | ||
|
||
## Context | ||
|
||
Similar to the workflow file, the composite action has access to the [same context objects](https://help.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#contexts) (ex: `github`, `env`, `strategy`). | ||
|
||
## Environment | ||
|
||
<del> Example `workflow.yml`: | ||
|
||
```yaml | ||
env: | ||
NAME1: test1 | ||
SERVER: production | ||
steps: | ||
- id: foo | ||
uses: user/test@v1 | ||
- run: echo Server $SERVER | ||
``` | ||
|
||
Example `user/composite/action.yml`: | ||
|
||
```yaml | ||
using: 'composite' | ||
env: | ||
NAME2: test2 | ||
SERVER: development | ||
runs: | ||
using: "composite" | ||
steps: | ||
- id: my-step | ||
run: | | ||
echo NAME2 $NAME2 | ||
echo Server $SERVER | ||
env: | ||
NAME2: test3 | ||
``` | ||
|
||
Example Output: | ||
|
||
``` | ||
NAME2 test3 | ||
Server development | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. According to this logic, workflow authors won't be able to override environment variables. A good compromise here might be to allow the use of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For additional clarity in the example, can we also explicitly say what happens when you specify |
||
Server production | ||
``` | ||
|
||
We plan to use environment variables for Composite Actions similar to the parent/child relationship between nested function calls in programming languages like Python in terms of [lexical scoping](https://inst.eecs.berkeley.edu/~cs61a/fa19/assets/slides/29-Tail_Calls_full.pdf). In Python, let's say you have `functionA` that has local variables called `a` and `b` in this function frame. Let's say we have a `functionB` whose parent frame is `functionA` and has local variable `a` (aka `functionB` is called and defined in `functionA`). `functionB` will have access to its parent input variables that are not overwritten in the local scope (`a`) as well as its own local variable `b`. [Visual Example](http://www.pythontutor.com/visualize.html#code=def%20functionA%28%29%3A%0A%20%20%20%20a%20%3D%201%0A%20%20%20%20b%20%3D%202%0A%20%20%20%20def%20functionB%28%29%3A%0A%20%20%20%20%20%20%20%20b%20%3D%203%0A%20%20%20%20%20%20%20%20print%28%22a%22,%20a%29%0A%20%20%20%20%20%20%20%20print%28%22b%22,%20b%29%0A%20%20%20%20%20%20%20%20return%20b%0A%20%20%20%20return%20functionB%28%29%0A%0A%0A%0AfunctionA%28%29&cumulative=false&curInstr=14&heapPrimitives=nevernest&mode=display&origin=opt-frontend.js&py=3&rawInputLstJSON=%5B%5D&textReferences=false) | ||
|
||
Similar to the above logic, the environment variables will flow from the parent node to its children node. More concretely, whatever workflow/action calls a composite action, that composite action has access to whatever environment variables its caller workflow/action has. Note that the composite action can append its own environment variables or overwrite its parent's environment variables. </del> | ||
|
||
In the Composite Action, you'll only be able to use `::set-env::` to set environment variables just like you could with other actions. | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. what about all the commands we have, ex: There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Update: users can use set-env in composite action that will set env variables for workflow. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Given that actions can set environment variables for the job, it feel like An usecase would be if someone wants to configure AWS credentials as part of their boilerplate for preparing a deploy. The workaround here would be for the action author to output everything using |
||
## Secrets | ||
|
||
We'll pass the secrets from the composite action's parents (ex: the workflow file) to the composite action. Secrets cannot be created in a composite action. | ||
|
||
## If Condition | ||
|
||
Example `workflow.yml`: | ||
|
||
```yaml | ||
steps: | ||
thboop marked this conversation as resolved.
Show resolved
Hide resolved
|
||
- run: exit 1 | ||
- uses: user/composite@v1 # <--- this will run, as it's marked as always runing | ||
if: always() | ||
``` | ||
|
||
Example `user/composite/action.yml`: | ||
|
||
```yaml | ||
runs: | ||
using: "composite" | ||
steps: | ||
- run: echo "just succeeding" | ||
- run: echo "I will run, as my current scope is succeeding" | ||
if: success() | ||
- run: exit 1 | ||
- run: echo "I will not run, as my current scope is now failing" | ||
``` | ||
|
||
**TODO: This if condition implementation is up to discussion. | ||
Discussions: https://github.com/actions/runner/pull/554#discussion_r443661891, ...** | ||
|
||
See the paragraph below for a rudimentary approach (thank you to @cybojenix for the idea, example, and explanation for this approach): | ||
|
||
The `if` statement in the parent (in the example above, this is the `workflow.yml`) shows whether or not we should run the composite action. So, our composite action will run since the `if` condition for running the composite action is `always()`. | ||
|
||
**Note that the if condition on the parent does not propogate to the rest of its children though.** | ||
|
||
In the child action (in this example, this is the `action.yml`), it starts with a clean slate (in other words, no imposing if conditions). Similar to the logic in the paragraph above, `echo "I will run, as my current scope is succeeding"` will run since the `if` condition checks if the previous steps **within this composite action** has not failed. `run: echo "I will not run, as my current scope is now failing"` will not run since the previous step resulted in an error and by default, the if expression is set to `success()` if the if condition is not set for a step. | ||
|
||
|
||
What if a step has `cancelled()`? We do the opposite of our approach above if `cancelled()` is used for any of our composite run steps. We will cancel any step that has this condition if the workflow is cancelled at all. | ||
|
||
#### Exposing Parent's If Condition to Children Via a Variable | ||
It would be nice to have a way to access information from a parent's if condition. We could have a parent variable that is contained in the context similar to other context variables `github`, `strategy`, etc.: | ||
|
||
Example `workflow.yml` | ||
|
||
```yaml | ||
steps: | ||
- run: exit 1 | ||
- uses: user/composite@v1 | ||
if: always() | ||
``` | ||
|
||
Example `user/composite/action.yml` | ||
|
||
```yaml | ||
runs: | ||
using: "composite" | ||
steps: | ||
- run: echo "preparing the slack bot..." # <--- This will run, as nothing has failed within the composite yet | ||
- run: slack.post("All builds passing, ready for a deploy") # <-- this will not run, as the parent fails | ||
if: ${{ parent.success() }} | ||
- run: slack.post("A failure has happened, fix things now", alert=true) # <--- This will run, as the parent fails | ||
if: ${{ parent.failure() }} | ||
``` | ||
|
||
## Timeout-minutes | ||
|
||
Example `workflow.yml`: | ||
|
||
```yaml | ||
steps: | ||
- id: bar | ||
uses: user/test@v1 | ||
timeout-minutes: 50 | ||
``` | ||
|
||
Example `user/composite/action.yml`: | ||
|
||
```yaml | ||
runs: | ||
using: "composite" | ||
steps: | ||
- id: foo1 | ||
run: echo test 1 | ||
timeout-minutes: 10 | ||
- id: foo2 | ||
run: echo test 2 | ||
- id: foo3 | ||
run: echo test 3 | ||
timeout-minutes: 10 | ||
``` | ||
|
||
**TODO: This timeout-minutes condition implementation is up to discussion.** | ||
|
||
A composite action in its entirety is a job. You can set both timeout-minutes for the whole composite action or its steps as long as the the sum of the `timeout-minutes` for each composite action step that has the attribute `timeout-minutes` is less than or equals to `timeout-minutes` for the composite action. There is no default timeout-minutes for each composite action step. | ||
|
||
If the time taken for any of the steps in combination or individually exceed the whole composite action `timeout-minutes` attribute, the whole job will fail. If an individual step exceeds its own `timeout-minutes` attribute but the total time that has been used including this step is below the overall composite action `timeout-minutes`, the individual step will fail but the rest of the steps will run. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
|
||
For reference, in the example above, if the composite step `foo1` takes 11 minutes to run, that step will fail but the rest of the steps, `foo1` and `foo2`, will proceed as long as their total runtime with the previous failed `foo1` action is less than the composite action's `timeout-minutes` (50 minutes). If the composite step `foo2` takes 51 minutes to run, it will cause the whole composite action job to fail. I | ||
|
||
The rationale behind this is that users can configure their steps with the `if` condition to conditionally set how steps rely on each other. Due to the additional capabilities that are offered with combining `timeout-minutes` and/or `if`, we wanted the `timeout-minutes` condition to be as dumb as possible and not effect other steps. | ||
|
||
[Usage limits still apply](https://help.github.com/en/actions/reference/workflow-syntax-for-github-actions?query=if%28%29#usage-limits) | ||
|
||
|
||
## Continue-on-error | ||
|
||
**TODO: This continue-on-error condition implementation is up to discussion.** | ||
|
||
Example `workflow.yml`: | ||
|
||
```yaml | ||
steps: | ||
- run: exit 1 | ||
- id: bar | ||
uses: user/test@v1 | ||
continue-on-error: false | ||
- id: foo | ||
run: echo "Hello World" <------- This step will not run | ||
``` | ||
|
||
Example `user/composite/action.yml`: | ||
|
||
```yaml | ||
runs: | ||
using: "composite" | ||
steps: | ||
- run: exit 1 | ||
continue-on-error: true | ||
- run: echo "Hello World 2" <----- This step will run | ||
``` | ||
|
||
If any of the steps fail in the composite action and the `continue-on-error` is set to `false` for the whole composite action step in the workflow file, then the steps below it will run. On the flip side, if `continue-on-error` is set to `true` for the whole composite action step in the workflow file, the next job step will run. | ||
|
||
For the composite action steps, it follows the same logic as above. In this example, `"Hello World 2"` will be outputted because the previous step has `continue-on-error` set to `true` although that previous step errored. | ||
|
||
## Defaults | ||
|
||
The composite action author will be required to set the `shell` and `workingDir` of the composite action. Moreover, the composite action author will be able to explicitly set the shell for each composite run step. The workflow author will not have the ability to change these attributes. | ||
|
||
## Visualizing Composite Action in the GitHub Actions UI | ||
We want all the composite action's steps to be condensed into the original composite action node. | ||
thboop marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
Here is a visual represenation of the [first example](#Steps) | ||
|
||
```yaml | ||
| composite_action_node | | ||
| echo hello world 1 | | ||
| echo hello world 2 | | ||
| echo hello world 3 | | ||
| echo hello world 4 | | ||
|
||
``` | ||
|
||
|
||
## Conclusion | ||
This ADR lays the framework for eventually supporting nested Composite Actions within Composite Actions. This ADR allows for users to run multiple run steps within a GitHub Composite Action with the support of inputs, outputs, environment, and context for use in any steps as well as the if, timeout-minutes, and the continue-on-error attributes for each Composite Action step. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Will
env
be top level, or nested underruns
, as it is when writing a docker action?Does the behaviour also align with how
env
works for docker actions? (where the workflow can't overrideruns.env
)