Proposal: Plan-apply split: agentic workflow operates read-only and outputs are applied. #85
Description
Motivation
It is very common for an agentic workflow to create an issue comment to publish its result. This is problematic because it requires to give the issue: write, pull_request: write permissions to o the main job. Instead we should not give the write permissions to the main job and move the creation of the issue comment to another job that that has write permissions.
Proposal
The output feature will be built on top of jobs (isolation) , jobs outputs / artifactfs.
- jobs gives us container/permission isolation
- outputs gives us a way to message-pass build artifacts between the jobs.
Principles
- agent job should work with read-only access
- write process with isolated containers treating outputs as adverserial
Syntax
At the user levels, the user gets a high level output field where it can specify how the "output" of the LLM should be ahndled. The goal is to cover common cases in AW
the underlying machinery will be more general and could be surfaced in later markdown syntax.
---
on: ...
output:
# add comment on the current parent (issue, issue_comment, pull_request, pull_request_review_comment)
comment:
# create a new issue
issue:
labels: [bug] # labels to added to issue
# other issue options
# create a new pull request with the git patch
pull-request:
path: # glob to define which files to select
# more options like branch-prefix, assignee, draft state
---
...
- comment, issue, pull_request are exclusive.
Compilation
In the compiler a few things must happen:
- collect and define output in the agent job and publish it as a job output value.
jobs:
agent:
permissions: content: read
steps:
- run: runagent --result-file /tmp/result/file
- run: add /tmp/result/file to GITHUB_OUTPUT
outputs:
text: the agent output
- the compiler should update the codegen to allow the agent to communicate the "text result" of the session and expose it as a gh output.
- collect the current change and generate a git patch and store it in an artifact
- emit a post-main job that processes the agent output
- reply-with-comment.yml - shared workflow that posts a comment back on the conversation
- create-issue.yml - shared workflow that creates a new issue from the conversation
- create-pull-request.yml - share workflow that creates a new pull request from the patch
jobs:
...
create_issue:
needs: [agent]
permissions: issue: write
steps:
sanitized_text = sanitize(needs.agent.outputs.text)
github.create_issue(sanitized_text)
Security
The risk is greatly reduced becaues the agent container has no write writes so the only damage is around create a new issue (secret exfiltration, malicious links generation). To alliviate this, we can do multiple this:
- deterministic cleanup of control characters etc...
- run string through secret scanning (the github MCP has this builtin now so we should ask to render the markdown to the api)
- use github models inference to implement a judge LLM and trap malicious links through the AI Prompt SHield running in the backgroud.
Note that if we would implement multi-step workflows, we could implement this feature as a transformation of the markdown IR, before running into the compiler. This could be a refactoring down the road.