CI agent Docker image for Docksal Sandboxes

An agent used to connect CI providers to a Docksal Sandbox Server.

Supported CI providers:

• Bitbucket Pipelines (with build status integration)
• CircleCI (with build status integration)
• GitLab
• Github Actions

This image(s) is part of the Docksal image library.

Docksal Sandboxes

Docksal Sandboxes are continuous integration environments powered by Docksal and deployed to a Docksal Sandbox Server. They can be provisioned for any git branch and are feature-wise identical to project's local Docksal environment.

Use cases:

• automated testing (full stack)
• manual testing
• enhanced pull request review experience
• demos

Build status updates (and sandbox URLs) can be posted to Github and Bitbucket via respective build status APIs.
URLs to sandbox environments can also be published to a Slack channel.

Image variants and versions

Stable

• base - basic (bash, curl, git, etc.), latest version
• php, latest - basic + php stack tools (composer, drush, drupal console, wp-cli, etc.), latest version
• 1.2-base - basic, a specific stable version
• 1.2-php - php, a specific stable version

Development

• edge-base - base, latest development version
• edge-php, php, latest development version

Global Configuration

Required

The following required variables are usually configured at the organization level. This way, all project repos will have access to them. They can as well be configured at the repo level.

DOCKSAL_HOST or DOCKSAL_HOST_IP

The address of the sandbox server. Configure one or the other.
If using DOCKSAL_HOST, make sure the domain is configured as a wildcard DNS entry.
If using DOCKSAL_HOST_IP, the agent will use nip.io for dynamic wildcard domain names for sandboxes.

DOCKSAL_HOST_SSH_KEY

A base64 encoded private SSH key, used to access the sandbox server.

Note: on macOS cat /path/to/<private_key_file> | base64 can be used to create a base64 encoded string from a private SSH key, while on Linux and in WSL on Windows 10 cat /path/to/<private_key_file> | base64 -w 0 should be used to avoid output wrapping of the base64 command).

Optional

BUILD_ENVIRONMENT

Used to set the environment built against. Defaults to local.

CI_SSH_KEY

A base64 encoded private SSH key, used by default for all hosts (set as Host * in ~/.ssh/config). This key will be used to clone/push to git, run commands over SSH on a remote deployment environment, etc.

DOCKSAL_HOST_SSH_PORT

Can be used to override the SSH port of the sandbox server where the agent connects (defaults to 22).

DOCKSAL_DOMAIN

Can be used to set the base URL for sandbox builds (defaults to DOCKSAL_HOST if not set), individually from DOCKSAL_HOST.
This is useful when working with CDNs/ELBs/WAFs/etc (when DOCKSAL_DOMAIN is different from the DOCKSAL_HOST).

DOCKSAL_HOST_USER

The user name used as the build user on the sandbox server. Defaults to build-agent. Must have SSH access to the server.

REMOTE_BUILD_BASE

The default directory location on the sandbox server where the repositories should be cloned down to and built. Defaults to /home/build-agent/builds.

REMOTE_CODEBASE_METHOD

Pick between rsync (default) and git for the codebase initialization method on the sandbox server.

The codebase is initialized on the sandbox server by the sandbox-init (or build-init) command.

git - code is checkout on the sandbox server via git. The server must have access to checkout from the repo. Any build settings and necessary code manipulations must happen on the sandbox server using build-exec commands.

rsync - code is rsync-ed to the sandbox server from the build agent. You can perform necessary code adjustments in the build agent after running build-env and before running sandbox-init (or build-init), which pushes the code to the sandbox server.

REMOTE_BUILD_DIR_CLEANUP

Whether or not the remote build directory is reset during the build. Only supported with REMOTE_CODEBASE_METHOD=git.

Defaults to 1 which wipes the remote build directory and produces a "clean build".
Set to 0 to produce "dirty builds", when file changes in the remote codebase should be preserved.

Note: Switching REMOTE_CODEBASE_METHOD mode will result in a clean build.

SANDBOX_PERMANENT

Set SANDBOX_PERMANENT=true to have a permanent sandbox provisioned.

Permanent sandboxes are exempt from scheduled garbage collection on the sandbox server. They would still hibernate after the configured period of inactivity, but won't be removed from the server after becoming dangling. See https://github.com/docksal/service-vhost-proxy#advanced-proxy-configuration for more information.

This variable is usually set at the branch level in the build settings to designate a specific (one or multiple) branch environments as permanent.

SANDBOX_DOMAIN

Sets a custom domain for a sandbox. Takes precedence over the automatic (branch name based) domain generation.

This can be used for sandbox environments which need a custom (nice) domain name.

GITHUB_TOKEN and BITBUCKET_TOKEN

Used for access to post sandbox URLs via build status API as well as comments on pull requests.

For Github, the token can be generated from the user's account.
Set access to "repo" (http://take.ms/nMqcW).

For Bitbucket, the token can be generated from the user's settings. Instructions on creating an app password.
Set access to "Repositories: Write", "Pull requests: Write" (http://take.ms/98BG5).
When storing the app password it is in the format: USER:PASSWORD.

GIT_USER_EMAIL

The user's email to perform Git operations as. Defaults to ci@docksal.io

GIT_USER_NAME

The user's name to perform Git operations as. Defaults to Docksal CI

DOCKSAL_HOST_TUNNEL

If not empty, localhost:2374 in the agent is mapped to docker.sock on the remote DOCKSAL_HOST via a secure SSH tunnel.
The agent can then run docker commands against the remote DOCKSAL_HOST.

Other features and integrations are usually configured at the repo level. See below.

Project configuration

Bitbucket Pipelines

Here's the most basic configuration for Bitbucket Pipelines. Save it into bitbucket-pipelines.yml in your project repo.

image: docksal/ci-agent:base

pipelines:
  default:
    - step:
        script:
          - source build-env && sandbox-init

For a more advanced example see bitbucket-pipelines.yml.

CircleCI

Here's the most basic configuration for CircleCI. Save it into .circleci/config.yml in your project repo.

version: 2

jobs:
  build:
    working_directory: /home/agent/build
    docker:
      - image: docksal/ci-agent:base
    steps:
      - run:
          name: Configure agent environment
          command: echo 'source build-env' >> $BASH_ENV
      - checkout
      - run:
          name: Build sandbox
          command: sandbox-init

For a more advanced example see config.yml.

GitLab

Here's the most basic configuration for GitLab. Save it into .gitlab-ci.yml in your project repo.

stages:
  - sandbox

sandbox-launch:
  stage: sandbox
  image: docksal/ci-agent:base
  script:
    - export SANDBOX_DOMAIN=$CI_ENVIRONMENT_SLUG--$CI_PROJECT_NAME.$DOCKSAL_HOST
    - source build-env
    - sandbox-init
  environment:
    name: $CI_COMMIT_REF_NAME
    url: https://$CI_ENVIRONMENT_SLUG--$CI_PROJECT_NAME.$DOCKSAL_HOST

For a more advanced example see .gitlab-ci.yml.

Github Actions

Here's the most basic configuration for Github Actions. Save it into .github/workflows/sandbox.yml in your project repo.

You might need to add your ssh key to EC2 /home/build-agent/.ssh and the deploy(public) key to Github Actions for cloning the codebase via git.

name: Docksal sandbox

on:
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    container:
      image: docksal/ci-agent:base
    env:
      CI_SSH_KEY: ${{ secrets.CI_SSH_KEY }}
      DOCKSAL_HOST_IP: ${{ secrets.DOCKSAL_HOST_IP }}
      DOCKSAL_HOST_SSH_KEY: ${{ secrets.DOCKSAL_HOST_SSH_KEY }}
      GITHUB_TOKEN: ${{ github.token }}
      REMOTE_CODEBASE_METHOD: git
    steps:
      - name: Build sandbox
        shell: bash
        run: |
          source build-env
          sandbox-init

For a more advanced example see .sandbox.yml.

Build commands

For a complete list of built-in commands see base/bin.

• build-env - configures build settings on the agent. Usage: source build-env (or DEBUG=1 source build-env)
• build-init- initializes the sandbox codebase and settings on the sandbox server. Usage: build-init
• build-exec - executes a shell command within the build directory on the sandbox server. Usage: build-init pwd
• build-notify - see "Build status notifications" docs below
• sandbox-init - a convenient shortcut to provision a basic sandbox. See sandbox-init

Build environment variables

The following variables are derived from the respective Bitbucket Pipelines, Circle CI, GitLab CI, and Github Actions build variables.

• GIT_REPO_OWNER - git repo machine owner/slug name
• GIT_REPO_NAME - git repo machine name
• GIT_REPO_URL - git repo URL
• GIT_BRANCH_NAME - git branch name
• GIT_COMMIT_HASH - git commit hash
• GIT_PR_NUMBER - git pull request / merge request number
• GIT_REPO_SERVICE - github, bitbucket or gitlab (makes sense mostly for CircleCI)
• BUILD_ID - The unique identifier for a build
• BUILD_DIR - The full path where the repository is cloned and where the job is run in the agent container

REMOTE_BUILD_DIR

The directory location on the sandbox server where current build will happen. Defaults to:

${REMOTE_BUILD_BASE}/${REPO_NAME_SAFE}-${BRANCH_NAME_SAFE}

Feature: Basic HTTP Auth

Protect sandboxes from public access using Basic HTTP authentication.

Configuration

Set the following environment variables at the repo level:

• HTTP_USER
• HTTP_PASS

Feature: Build status notifications

This integration allows the agent to post build status updates and sandbox URL via Github/Bitbucket build status API.
For CircleCI, it is also possible to enable posting the sandbox URL as a comment in pull requests.

Configuration

GITHUB_TOKEN or BITBUCKET_TOKEN must be configured respectively (either globally or at the repo level).

Usage

build-notify <pending|success|failure>

Place the triggers right before and right after fin init call in your build script, e.g.,

build-notify pending 
ssh docker-host "cd $REMOTE_BUILD_DIR && fin init" || ( build-notify failure && exit 1 )
build-notify success

To enable posting sandbox URLs in comments on pull requests, do export PR_COMMENT=1 prior to calling build-notify

Feature: Slack notifications

This integrations allows the agent to post messages to a given Slack channel.
It can be used for notification purposes when a build is started, completed, failed, etc.

Configuration

SLACK_WEBHOOK_URL

The Incoming Webhook integration URL from Slack, e.g., SLACK_WEBHOOK_URL https://hooks.slack.com/services/XXXXXXXXX/XXXXXXXXX/XXxxXXXXxxXXXXxxXXXXxxXX

SLACK_CHANNEL

A public or private channel in Slack, e.g., SLACK_CHANNEL #project-name-bots

SLACK_USER

The username the message should post to Slack as. Defaults to Docksal CI

SLACK_ICON

The icon the message should use to accompany the message: Defaults to :desktop_computer:

Usage

slack 'message' ['#channel'] ['webhook_url'] ['slack_user'] ['slack_icon']

Channel and webhook url can be passed via environment variables. See above.

Limitations

Incoming Webhook integration won't work for private channels, which the owner of the integration does not belong to.

Feature: Build artifact storage

Build artifacts can be stored in an AWS S3 bucket.

Configuration

Set the following environment variables at the organization or repo level:

• AWS_ACCESS_KEY_ID
• AWS_SECRET_ACCESS_KEY
• ARTIFACTS_BUCKET_NAME
• ARTIFACTS_BASE_URL (optional)
• GITHUB_TOKEN (optional)
• BITBUCKET_TOKEN (optional)

Usage

To upload artifacts to the configured S3 bucket use the build-acp command.

build-acp /source/path/

There is no file browsing capability available for private S3 buckets.
An index.html file is used as the directory index, however it has to be created/generated manually.
When uploading a prepared artifacts folder with the index.html file in it, add a trailing slash to the source path to upload the contents of the source folder vs the folder itself.

You can upload additional folders/files by running the command multiple times.

The optional destination argument can be used to define a subdirectory at the destination in the bucket.

build-acp /source/path/ destination/path

Advanced usage

Base URL

The base URL is derived from ARTIFACTS_BUCKET_NAME as follows (assuming AWS S3 us-east-1 region by default):

https://${ARTIFACTS_BUCKET_NAME}.s3.amazonaws.com

It can be overridden via the optional ARTIFACTS_BASE_URL configuration variable at the organization/repo level:

ARTIFACTS_BASE_URL = https://artifacts.example.com

Upload path

The upload path is unique for each commit and is derived as follows:

${REPO_NAME_SAFE}/${BRANCH_NAME_SAFE}-${GIT_COMMIT_HASH}

In certain cases you may want to store build artifacts per branch instead of per commit.
To do this, override the ARTIFACTS_BUCKET_PATH variable before calling the build-acp command:

export ARTIFACTS_BUCKET_PATH="${REPO_NAME_SAFE}/${BRANCH_NAME_SAFE}"
build-acp my-artifacts/

Posting build artifact URLs to Bitbucket

If BITBUCKET_TOKEN is set, the URL to the artifacts will be posted back to Bitbucket via Bitbucket Build Status API.

Security

If a bucket does not exist, it will be created automatically (with no public access). Existing bucket access permissions are not automatically adjusted. It's up to you whether you want to keep them open or not.

When artifacts are uploaded, the destination artifact folder in the bucket is set to be publicly accessible. Anyone with the direct link will be able to access the artifacts, but will not be able to browse the list of all available artifact folders in the bucket (so long as the bucket itself is set to be private).

The URL by default includes a git commit hash, which serves as an authentication token (the URL is impossible to guess). This provides a simple yet efficient level of security for artifacts.

To add an additional level of security follow this guide to set up username/password access to S3 via CloudFront and Lambda@Edge.

Feature: Non-volatile environments

By combining the following configuration options you can get low overhead non-volatile environments.

SANDBOX_DOMAIN=<nice-domain>
SANDBOX_PERMANENT=true
REMOTE_CODEBASE_METHOD=git
REMOTE_BUILD_DIR_CLEANUP=0

Such environments can be used for non-critical production-ish workloads, whenever an on-demand delayed start (5-10s delay) is not a concern.

Feature: Secrets in environment variables

It is best security practice not to store secrets such as API keys in a code repository. Many CI systems already have the ability to set such environment variables during the build process. Any environment variables set at build time whose name starts with SECRET_ will be forwarded as-is to the built environment.