Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add "plugin" support to local backend #2239

Merged
merged 14 commits into from
Aug 22, 2023
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 12 additions & 5 deletions docs/docs/30-administration/22-backends/20-local.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,7 @@ The local backend will execute the pipelines on the local system without any iso

:::note
This backend is still pretty new and can not be treated as stable. Its
implementation and configuration can change at any time. Binary releases of the
agent will be available with the release of the [1.0.0
milestone](https://github.com/woodpecker-ci/woodpecker/milestone/4), so for now
you must compile the agent by yourself, to get the local backend functionality.
<!-- TODO: remove the self-compile note after the release of the agent -->
implementation and configuration can change at any time.
:::

Since the code runs directly in the same context as the agent (same user, same
Expand Down Expand Up @@ -92,6 +88,17 @@ steps:
[...]
```

### Plugins as Executable Binaries

```yaml
steps:
build:
image: /usr/bin/tree
6543 marked this conversation as resolved.
Show resolved Hide resolved
```

If no commands are provided, we treat them as plugins in the usual manner.
qwerty287 marked this conversation as resolved.
Show resolved Hide resolved
In the context of the local backend, plugins are simply executable binaries, which can be located using their name if they are listed in `$PATH`, or through an absolute path.

### Using labels to filter tasks

You can use the [agent configuration
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,7 @@ The local backend will execute the pipelines on the local system without any iso

:::note
This backend is still pretty new and can not be treated as stable. Its
implementation and configuration can change at any time. Binary releases of the
agent will be available with the release of the [1.0.0
milestone](https://github.com/woodpecker-ci/woodpecker/milestone/4), so for now
you must compile the agent by yourself, to get the local backend functionality.
<!-- TODO: remove the self-compile note after the release of the agent -->
implementation and configuration can change at any time.
:::

Since the code runs directly in the same context as the agent (same user, same
Expand Down
14 changes: 8 additions & 6 deletions pipeline/backend/local/clone.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ import (

"github.com/rs/zerolog/log"
"github.com/woodpecker-ci/woodpecker/pipeline/backend/types"
"github.com/woodpecker-ci/woodpecker/shared/constant"
)

// checkGitCloneCap check if we have the git binary on hand
Expand Down Expand Up @@ -64,17 +63,20 @@ func (e *local) setupClone(state *workflowState) error {

// execClone executes a clone-step locally
func (e *local) execClone(ctx context.Context, step *types.Step, state *workflowState, env []string) error {
if err := e.setupClone(state); err != nil {
return fmt.Errorf("setup clone step failed: %w", err)
if scm := step.Environment["CI_REPO_SCM"]; scm != "git" {
return fmt.Errorf("local backend can only clone from git repos")
6543 marked this conversation as resolved.
Show resolved Hide resolved
}

if err := checkGitCloneCap(); err != nil {
return fmt.Errorf("check for git clone capabilities failed: %w", err)
}

if step.Image != constant.DefaultCloneImage {
// TODO: write message into log
log.Warn().Msgf("clone step image '%s' does not match default git clone image. We ignore it assume git.", step.Image)
if err := e.setupClone(state); err != nil {
return fmt.Errorf("setup clone step failed: %w", err)
}

if !strings.Contains(step.Image, "woodpeckerci/plugin-git") {
6543 marked this conversation as resolved.
Show resolved Hide resolved
log.Warn().Msgf("clone step image '%s' does not match default git clone image. We ignore it and use our plugin-git anyway.", step.Image)
}

rmCmd, err := writeNetRC(step, state)
Expand Down
25 changes: 24 additions & 1 deletion pipeline/backend/local/local.go
Original file line number Diff line number Diff line change
Expand Up @@ -120,13 +120,16 @@ func (e *local) StartStep(ctx context.Context, step *types.Step, taskUUID string
return e.execClone(ctx, step, state, env)
case types.StepTypeCommands:
return e.execCommands(ctx, step, state, env)
case types.StepTypePlugin:
return e.execPlugin(ctx, step, state, env)
default:
return ErrUnsupportedStepType
}
}

// execCommands use step.Image as shell and run the commands in it
func (e *local) execCommands(ctx context.Context, step *types.Step, state *workflowState, env []string) error {
// TODO: use commands directly
// TODO: find a way to simulate commands to be exec as stdin user commands instead of generating a script and hope the shell understands
script := ""
for _, cmd := range step.Commands {
script += fmt.Sprintf("echo + %s\n%s\n", strings.TrimSpace(shellescape.Quote(cmd)), cmd)
Expand All @@ -148,6 +151,26 @@ func (e *local) execCommands(ctx context.Context, step *types.Step, state *workf
return cmd.Start()
}

// execPlugin use step.Image as exec binary
func (e *local) execPlugin(ctx context.Context, step *types.Step, state *workflowState, env []string) error {
binary, err := exec.LookPath(step.Image)
if err != nil {
return fmt.Errorf("exec plugin: %w", err)
6543 marked this conversation as resolved.
Show resolved Hide resolved
}

cmd := exec.CommandContext(ctx, binary)
cmd.Env = env
cmd.Dir = state.workspaceDir

// Get output and redirect Stderr to Stdout
e.output, _ = cmd.StdoutPipe()
cmd.Stderr = cmd.Stdout

state.stepCMDs[step.Name] = cmd

return cmd.Start()
}

// WaitStep for the pipeline step to complete and returns
// the completion results.
func (e *local) WaitStep(_ context.Context, step *types.Step, taskUUID string) (*types.State, error) {
Expand Down