Registry Image Resource

Supports checking, fetching, and pushing of images to Docker registries. This resource can be used in three ways: with tag specified, with tag_regex specified, or with neither tag nor tag_regex specified.

With tag specified, check will detect changes to the digest the tag points to, and out will always push to the specified tag. This is to be used in simpler cases where no real versioning exists.

With tag_regex specified, check will instead detect tags based on the regex provided. If created_at_sort is set to true, the tags will be sorted in descending order by the creation time. This is useful when you want to get the latest tag based on the regex (see Docker registry issue here).

With tag and tag_regex both omitted, check will instead detect tags based on semver versions (e.g. 1.2.3) and return them in semver order. With variant included, check will only detect semver tags that include the variant suffix (e.g. 1.2.3-stretch).

This resource comes with Concourse by default. You can override the version you use within your pipeline if the built-in one is not working for you for some reason

Comparison to docker-image resource

This resource is intended as a replacement for the Docker Image resource. Here are the key differences:

• This resource is implemented in pure Go and does not use the Docker daemon or CLI. This makes it safer (no need for privileged: true), more efficient, and less error-prone (now that we're using Go APIs and not parsing docker CLI output).

• This resource has stronger test coverage.

• This resource does not and will never support building - only registry image pushing/pulling. Building should instead be done with something like the oci-build task (or anything that can produce OCI image tarballs).

• A goal of this resource is to stay as focused and simple as possible. The Docker Image resource grew way too large and complicated. There are simply too many ways to build and publish Docker images. It will be easier to support many smaller resources + tasks rather than one huge interface.

Source Configuration

Field Name	Description
repository (Required)	The URI of the image repository, e.g. alpine or ghcr.io/package/image. Defaults to checking docker.io if no hostname is provided in the URI. Note: If using ecr you only need the repository name, not the full URI e.g. alpine not 012345678910.dkr.ecr.us-east-1.amazonaws.com/alpine. ECR usage is NOT automatically detected. You must set the aws_region to tell the resource to automatically use ECR.
insecure (Optional) Default: false	Allow insecure registry.
tag (Optional) Default: latest	Instead of monitoring semver tags, monitor a single tag for changes (based on digest).
tag_regex (Optional)	Instead of monitoring semver tags, monitor for tags based on a regex provided. The syntax of the regular expressions accepted is the same general syntax used by Perl, Python, and other languages. More precisely, it is the syntax accepted by RE2 and described at https://golang.org/s/re2syntax Note if used, this will override all Semver constraints and features. By default, order of tags is not guaranteed. If you want to sort the tags in descending order, set `created_at_sort` to `true`.
created_at_sort (Optional) Default: false	If set to `true`, the tags will be sorted in descending order using the creation time from the image history. This is useful when you want to get the latest tag based on the tag_regex.
variant (Optional)	Detect only tags matching this variant suffix, and push version tags with this suffix applied. For example, a value of stretch would be used for tags like 1.2.3-stretch. This is typically used without tag - if it is set, this value will only used for pushing, not checking.
semver_constraint (Optional)	Constrain the returned semver tags according to a semver constraint, e.g. "~1.2.x", ">= 1.2 < 3.0.0 || >= 4.2.3". Follows the rules outlined in https://github.com/Masterminds/semver#checking-version-constraints If the value appends with -0 for pre-release versions, pre_releases needs to be true.
pre_releases (Optional)	By default, pre-release versions are ignored. With `pre_releases: true`, they will be included. Note however that variants and pre-releases both use the same syntax: `1.2.3-alpine` is technically also valid syntax for a Semver prerelease. For this reason, the resource will only consider prerelease data starting with `alpha`, `beta`, or `rc` as a proper prerelease, treating anything else as a variant.
username and password (Optional)	A username and password to use when authenticating to the registry. Must be specified for private repos or when using put.
aws_access_key_id (Optional)	The access key ID to use for authenticating with ECR.
aws_secret_access_key (Optional)	The secret access key to use for authenticating with ECR.
aws_session_token (Optional)	The session token to use for authenticating with STS credentials with ECR.
aws_region (Optional)	The region to use for accessing ECR. This is required if you are using ECR. This region will help determine the full repository URL you are accessing (e.g., 012345678910.dkr.ecr.us-east-1.amazonaws.com)
aws_role_arn (Optional)	If set, then this role will be assumed before authenticating to ECR. An error will occur if aws_role_arns is also specified. This is kept for backward compatibility.
aws_role_arns (Optional)	An array of AWS IAM roles. If set, these roles will be assumed in the specified order before authenticating to ECR. An error will occur if aws_role_arn is also specified.
aws_account_id (Optional)	The AWS Account ID that the image is located in. Useful if interacting with images in another account. If omitted then the current AWS account ID will be used. Be sure to wrap the account ID in quotes so it is parsed as a string instead of a number.
platform (Optional) (Experimental)	architecture (Optional): Architecture the image is built for (e.g. `amd64`, `arm64/v8`). If not specified, will default to https://pkg.go.dev/runtime#GOARCH. os (Optional): OS the image is built for (e.g. `linux`, `darwin`, `windows`). If not specified, will default to https://pkg.go.dev/runtime#GOOS.
debug (Optional) Default: false	If set, progress bars will be disabled and debugging output will be printed instead.
registry_mirror (Optional)	host (Required): A hostname pointing to a Docker registry mirror service. Note that this is only used if no registry hostname prefix is specified in the repository key. If the repository contains a registry hostname, such as my-registry.com/foo/bar, the registry_mirror is ignored and the explicitly declared registry in the repository key is used. username and password (Optional): A username and password to use when authenticating to the mirror.
content_trust (Optional)	server (Optional): URL for the notary server. (equal to DOCKER_CONTENT_TRUST_SERVER) repository_key_id (Required): Target key's ID used to sign the trusted collection, could be retrieved by notary key list repository_key (Required): Target key used to sign the trusted collection. repository_passphrase (Required): The passphrase of the signing/target key. (equal to DOCKER_CONTENT_TRUST_REPOSITORY_PASSPHRASE) tls_key (Optional): TLS key for the notary server. tls_cert (Optional): TLS certificate for the notary server. username (Optional): Username for authorize Docker Registry with a Notary server(`content_trust.server`) attached. password (Optional): Password for authorize Docker Registry with a Notary server(`content_trust.server`) attached. scopes (Optional): What access for the resources requested, should be one of ['pull', 'push,pull', 'catalog']
ca_certs(Optional)	An array of PEM-encoded CA certificates. Example: ca_certs: - | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- - | -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- Each entry specifies the x509 CA certificate for the trusted docker registry. This is used to validate the certificate of the docker registry when the registry's certificate is signed by a custom authority (or itself).

Signing with Docker Hub

Configure Docker Content Trust for use with the Docker Hub and Notary service by specifying the above source parameters as follows:

• repository_key should be set to the contents of the DCT key file located in your ~/.docker/trust/private directory.
• repository_key_id should be set to the full key itself, which is also the filename of the key file mentioned above, without the .key extension.

Consider the following resource:

resources:
- name: trusted-image
  type: registry-image
  source:
    repository: docker.io/foo/bar
    username: ((registry_user))
    password: ((registry_pass))
    content_trust:
      repository_key_id: ((registry_key_id))
      repository_key: ((registry_key))
      repository_passphrase: ((registry_passphrase))

Specify the values for these variables as shown in the following static variable file, or preferrably in a configured credential manager:

registry_user: jertel
registry_pass: my_docker_hub_token
registry_passphrase: my_dct_key_passphrase
registry_key_id: 1452a842871e529ffc2be29a012618e1b2a0e6984a89e92e34b5a0fc21a04cd
registry_key: |
  -----BEGIN ENCRYPTED PRIVATE KEY-----
  role: jertel

  MIhsj2sd41fwaa...
  -----END ENCRYPTED PRIVATE KEY-----

NOTE This configuration only applies to the out action. check & in aren't impacted. Hence, it would be possible to check or use in to get unsigned images.

Behavior

check Step (check script) with tag: discover new digests for the tag

Reports the current digest that the registry has for the tag configured in source.

check Step (check script) with tag_regex: discover tags matching regex

Reports the current digest that the registry has for tags matching the regex configured in source. They will be returned in the same order that the source repository lists them unless created_at_sort is set to true.

check Step (check script) without tag or tag_regex: discover semver tags

Detects tags which contain semver version numbers. Version numbers do not need to contain all 3 segments (major/minor/patch).

Each unique digest will be returned only once, with the most specific version tag available. This is to handle "alias" tags like 1, 1.2 pointing to 1.2.3.

Note: the initial check call will return all valid versions, which is unlike most resources which only return the latest version. This is an intentional choice which will become the normal behavior for resources in the future (per concourse/rfcs#38).

Example:

resources:
- name: concourse
  type: registry-image
  source: {repository: concourse/concourse}

The above resource definition would detect the following versions:

[
  {
    "tag": "1.6.0",
    "digest": "sha256:e1ad01d3227569ad869bdb6bd68cf1ea54057566c25bae38b99d92bbe9f28d78"
  },
  {
    "tag": "2.0.0",
    "digest": "sha256:9ab8d1021d97c6602abbb2c40548eab67aa7babca22f6fe33ab80f4cbf8ea92c"
  },
  // ...
]

Variant tags

Docker repositories have a pretty common convention of adding -SUFFIX to tags to denote "variant" images, i.e. the same version but with a different base image or dependency. For example, 1.2.3 vs 1.2.3-alpine.

With a variant value specified, only semver tags with the matching variant will be detected. With variant omitted, tags which include a variant are ignored.

Note: some image tags actually include mutliple variants, e.g. 1.2.3-php7.3-apache. With a variant of only apache configured, these tags will be skipped to avoid accidentally using multiple variants. In order to use these tags, you must specify the full variant combination, e.g. php7.3-apache.

Example:

resources:
- name: concourse
  type: registry-image
  source:
    repository: concourse/concourse
    variant: ubuntu

The above resource definition would detect the following versions:

[
  {
    "tag": "5.2.1-ubuntu",
    "digest": "sha256:91f5d180d84ee4b2cedfae45771adac62c67c3f5f615448d3c34057c09404f27"
  },
  {
    "tag": "5.2.2-ubuntu",
    "digest": "sha256:cb631d788797f0fbbe72a00afd18e5e4bced356e1b988d1862dc9565130a6226"
  },
  // ...
]

get Step (in script): fetch an image

Fetches an image at the exact digest specified by the version.

get Step params

Parameter	Description
format (Optional) Default: rootfs	The format to fetch the image as. Accepted values are: rootfs, oci
skip_download (Optional) Default: false	Skip downloading the image. Useful if you want to trigger a job without using the object or when running after a put step and not needing to download the image you just uploaded.

Files created by the get step

The resource will produce the following files:

• ./repository: A file containing the image's full repository name, e.g. concourse/concourse. For ECR images, this will include the registry the image was pulled from.
• ./tag: A file containing the tag from the version.
• ./digest: A file containing the digest from the version, e.g. sha256:....
• ./labels.json: A file containing a JSON map of image labels, e.g. { "commit": "4e5c4ea" }

The remaining files depend on the configuration value for format:

rootfs Format

The rootfs format will fetch and unpack the image for use by Concourse task and resource type images.

This the default for the sake of brevity in pipelines and task configs.

In this format, the resource will produce the following files:

• ./rootfs/...: the unpacked rootfs produced by the image.
• ./metadata.json: the runtime information to propagate to Concourse.

oci Format

The oci format will fetch the image and write it to disk in OCI format. This is analogous to running docker save.

In this format, the resource will produce the following files:

• ./image.tar: the OCI image tarball, suitable for passing to docker load.

put Step (out script): push and tag an image

Pushes an image to the registry as the specified tags.

The currently encouraged way to build these images is by using the oci-build-task.

Tags may be specified in multiple ways:

• With tag configured in source, the configured tag will always be pushed.
• With version given in params, the image will be pushed using the version number as a tag, optionally with a variant suffix (configured in source).
• With additional_tags given in params, the image will be pushed as each tag listed in the file (whitespace separated). Only those tags are pushed, e.g. the default latest isn't included.

put Steps params

Parameter	Description
image (Required)	The path to the oci image tarball to upload. Expanded with filepath.Glob
version (Optional)	A version number to use as a tag.
bump_aliases (Optional) Default: false	When set to true and version is specified, automatically bump alias tags for the version. For example, when pushing version 1.2.3, push the same image to the following tags: 1.2, if 1.2.3 is the latest version of 1.2.x. 1, if 1.2.3 is the latest version of 1.x. latest, if 1.2.3 is the latest version overall. If variant is configured as foo, push the same image to the following tags: 1.2-foo, if 1.2.3 is the latest version of 1.2.x with foo. 1-foo, if 1.2.3 is the latest version of 1.x with foo. foo, if 1.2.3 is the latest version overall for foo. Determining which tags to bump is done by comparing to the existing tags that exist on the registry.
additional_tags (Optional)	The path to a file with whitespace-separated list of tag values to tag the image with (in addition to the tag configured in source, but not the default `latest` tag if no tag is configured).

Use in tasks

Images used as image resources in tasks are called anonymous resources. Anonymous resources can specify a version, which is the image digest. For example:

image_resource:
  type: docker-image
  source:
    repository: golang
  version:
    digest: 'sha256:5f640aeb8b78e9876546a9d06b928d2ad0c6e51900bcba10ff4e12dc57f6f265'

This is useful when the registry image does not have tags, or when the tags are going to be re-used.

Development

Prerequisites

• golang is required - version 1.11.x or above is required for go mod to work
• docker is required - version 17.06.x is tested; earlier versions may also work.
• go mod is used for dependency management of the golang packages.

Running the tests

The tests have been embedded with the Dockerfile; ensuring that the testing environment is consistent across any docker enabled platform. When the docker image builds, the test are run inside the docker container, on failure they will stop the build.

Run the tests with the following commands:

docker build -t registry-image-resource --target tests --build-arg base_image=paketobuildpacks/run-jammy-base:latest .

Integration tests

The integration requires 3 docker repos: one private dockerhub repo, one public dockerhub repo, and one GCR repo. The docker build step requires setting --build-args so the integration will run.

Run the tests with the following command:

docker build . -t registry-image-resource --target tests \
  --build-arg DOCKER_PRIVATE_USERNAME="some-username" \
  --build-arg DOCKER_PRIVATE_PASSWORD="some-password" \
  --build-arg DOCKER_PRIVATE_REPO="some/repo" \
  --build-arg DOCKER_PUSH_USERNAME="some-username" \
  --build-arg DOCKER_PUSH_PASSWORD="some-password" \
  --build-arg DOCKER_PUSH_REPO="some/repo" \
  --build-arg GCR_PUSH_SERVICE_ACCOUNT_KEY='{"some":"json"}' \
  --build-arg GCR_PUSH_REPO="some/repo"

Note that you may omit any of the repo credentials in order to skip those integration tests.

Contributing

Please make all pull requests to the master branch and ensure tests pass locally.