Add a filter parameter - to select for the images to cache

[jamie-wearsafe]

Is your feature request related to a problem? Please describe.
Not all images need caching, and can take up valuable room.
Some images are loaded before the cache is loaded such as the images in services.
So excluding those images would help reduce the size and speed up the save/load process.

Describe the solution you'd like
When doing docker save $(docker images -q) you can provide a list of the image ID's to save.
If we could provide a filter at that point, it would mean we were only saving the images that we needed and so instead of this list:

REPOSITORY                                    TAG                       IMAGE ID       CREATED         SIZE
codacy/codacy-prospector                      2.3.9                     2ed875f0c09d   2 days ago      267MB
codacy/codacy-analysis-cli                    latest                    9d412c689197   2 days ago      421MB
snyk/snyk                                     maven-3-jdk-11            882d05531c51   3 days ago      734MB
codacy/codacy-rubocop                         5.1.6                     88f504735018   7 days ago      166MB
codacy/codacy-tsqllint                        1.2.4                     92b7ddf6589f   7 days ago      91.8MB
codacy/codacy-pylint-python3                  2.2.16                    504237065f5b   7 days ago      161MB
api                                           11                        413eb2357edb   9 days ago      512MB
codacy/codacy-bandit                          2.3.1                     5c95af379274   12 days ago     264MB
codacy/codacy-pmd                             3.11.0                    bb50d6e2a005   2 weeks ago     162MB
tomcat                                        8.5-jdk11-corretto        ae7b0918b39c   4 weeks ago     473MB
codacy/codacy-remark-lint                     2.5.59                    ef1454679208   6 weeks ago     155MB
codacy/codacy-bundler-audit                   4.2.12                    8d9828916ae8   2 months ago    86MB
redis                                         5-alpine                  caa6655434a5   2 months ago    29.7MB
codacy/codacy-spotbugs                        1.10.5                    999b0c7e7945   2 months ago    128MB
amazonlinux                                   xray                      b88042e00023   3 months ago    395MB
codacy/codacy-checkstyle                      2.3.0                     5b7b8322d896   4 months ago    118MB
codacy/codacy-hadolint                        1.5.2                     04e2b64ed099   4 months ago    9.31MB
codacy/codacy-metrics-pmd                     0.2.2                     8b5299fc6999   4 months ago    111MB
codacy/codacy-shellcheck                      2.4.1                     509fffd67432   4 months ago    33.5MB
codacy/codacy-sqlint                          2.3.2                     f21a8ce8c730   4 months ago    148MB
codacy/codacy-jackson-linter                  5.2.2                     d06c7b65bd22   4 months ago    104MB
codacy/codacy-brakeman                        1.3.2                     f95377ddb330   4 months ago    146MB
codacy/codacy-pylint                          3.2.1                     582388b48c67   4 months ago    242MB
codacy/codacy-codenarc                        0.4.2                     88739f96d424   4 months ago    111MB
mysql                                         5.6                       c580203d8753   4 months ago    302MB
codacy/codacy-metrics-cloc                    0.2.4                     0f47e48065ab   7 months ago    197MB
amazon/amazon-ecs-local-container-endpoints   latest                    b005329f50b1   7 months ago    180MB
codacy/codacy-duplication-pmdcpd              2.0.145                   e121e0516f79   11 months ago   132MB
codacy/codacy-metrics-rubocop                 0.2.2                     ee82a4e0ba6a   11 months ago   269MB
codacy/codacy-metrics-radon                   0.1.66                    19f7a80d4001   11 months ago   154MB
codacy/codacy-duplication-flay                2.0.154                   ba95f1e54cfa   11 months ago   166MB
codacy/codacy-pmdjava                         2.0.0-pmdlegacy.57fdbf2   dee6d8e1c0e5   20 months ago   231MB
boxfuse/flyway                                5.2.4-alpine              df49c5fb49bb   2 years ago     115MB
memcached                                     1.5.10-alpine             fb0df03da449   2 years ago     8.97MB

A filter would be added at this point:

docker save $(docker images -q --filter=reference='codacy/*')

and we would get this list instead:

REPOSITORY                         TAG                       IMAGE ID       CREATED         SIZE
codacy/codacy-prospector           2.3.9                     2ed875f0c09d   2 days ago      267MB
codacy/codacy-analysis-cli         latest                    9d412c689197   2 days ago      421MB
codacy/codacy-rubocop              5.1.6                     88f504735018   7 days ago      166MB
codacy/codacy-tsqllint             1.2.4                     92b7ddf6589f   7 days ago      91.8MB
codacy/codacy-pylint-python3       2.2.16                    504237065f5b   7 days ago      161MB
codacy/codacy-bandit               2.3.1                     5c95af379274   12 days ago     264MB
codacy/codacy-pmd                  3.11.0                    bb50d6e2a005   2 weeks ago     162MB
codacy/codacy-remark-lint          2.5.59                    ef1454679208   6 weeks ago     155MB
codacy/codacy-bundler-audit        4.2.12                    8d9828916ae8   2 months ago    86MB
codacy/codacy-spotbugs             1.10.5                    999b0c7e7945   2 months ago    128MB
codacy/codacy-checkstyle           2.3.0                     5b7b8322d896   4 months ago    118MB
codacy/codacy-hadolint             1.5.2                     04e2b64ed099   4 months ago    9.31MB
codacy/codacy-metrics-pmd          0.2.2                     8b5299fc6999   4 months ago    111MB
codacy/codacy-shellcheck           2.4.1                     509fffd67432   4 months ago    33.5MB
codacy/codacy-sqlint               2.3.2                     f21a8ce8c730   4 months ago    148MB
codacy/codacy-jackson-linter       5.2.2                     d06c7b65bd22   4 months ago    104MB
codacy/codacy-brakeman             1.3.2                     f95377ddb330   4 months ago    146MB
codacy/codacy-pylint               3.2.1                     582388b48c67   4 months ago    242MB
codacy/codacy-codenarc             0.4.2                     88739f96d424   4 months ago    111MB
codacy/codacy-metrics-cloc         0.2.4                     0f47e48065ab   7 months ago    197MB
codacy/codacy-duplication-pmdcpd   2.0.145                   e121e0516f79   11 months ago   132MB
codacy/codacy-metrics-rubocop      0.2.2                     ee82a4e0ba6a   11 months ago   269MB
codacy/codacy-metrics-radon        0.1.66                    19f7a80d4001   11 months ago   154MB
codacy/codacy-duplication-flay     2.0.154                   ba95f1e54cfa   11 months ago   166MB
codacy/codacy-pmdjava              2.0.0-pmdlegacy.57fdbf2   dee6d8e1c0e5   20 months ago   231MB

[jamie-wearsafe]

as in at this part of the code change it to this:

import * as exec from 'actions-exec-listener'
import * as core from '@actions/core'

export class ImageDetector {
  async getExistingImages(): Promise<string[]> {
    const _filter = core.getInput(`filter`)
    const filter = _filter ? `--filter=${_filter}` : ''
    const existingSet = new Set<string>([])
    const ids = (await exec.exec(`docker image ls -q ${filter}`, [], { silent: true, listeners: { stderr: console.warn }})).stdoutStr.split(`\n`).filter(id => id !== ``)
    const repotags = (await exec.exec(`docker`, `image ls --format {{.Repository}}:{{.Tag}} ${filter} --filter=dangling=false`.split(' '), { silent: true, listeners: { stderr: console.warn }})).stdoutStr.split(`\n`).filter(id => id !== ``);
    core.debug(JSON.stringify({ log: "getExistingImages", ids, repotags }));
    ([...ids, ...repotags]).forEach(image => existingSet.add(image))
    core.debug(JSON.stringify({ existingSet }))
    return Array.from(existingSet)
  }

  async getImagesShouldSave(alreadRegisteredImages: string[]): Promise<string[]> {
    const resultSet = new Set(await this.getExistingImages())
    alreadRegisteredImages.forEach(image => resultSet.delete(image))
    return Array.from(resultSet)
  }
}

[leigh-hobbii]

this would be cool, at the moment i'm caching many images that really dont need to be (like mysql etc), but OTOH I really want to cache the images that are built locally.. (I'm running this as docker-compose)

[MostefaKamalLala]

Imagine in windows containers, this would be amazing! Have you tested it @jamie-wearsafe ? would be nice to have a pull request if you've done it 🙏

[jamie-wearsafe]

Imagine in windows containers, this would be amazing! Have you tested it @jamie-wearsafe ? would be nice to have a pull request if you've done it 🙏

I am using a fork where it's working in production.

https://github.com/Broadshield/action-docker-layer-caching

[cb109]

@jamie-wearsafe Thanks! Using your fork for now, hopefully the filter input can be merged into this project.

For anyone curious on how to use this here is an example:

- name: Cache docker images
  uses: Broadshield/action-docker-layer-caching@main
  continue-on-error: true
  with:
    filter: reference=my_custom_images_prefix*

[jamie-wearsafe]

@jamie-wearsafe Thanks! Using your fork for now, hopefully the filter input can be merged into this project.

For anyone curious on how to use this here is an example:

- name: Cache docker images

  uses: Broadshield/action-docker-layer-caching@main

  continue-on-error: true

  with:

    filter: reference=my_custom_images_prefix*

Ahh sorry I didn't provide an example!

[madhavajay]

@jamie-wearsafe Great work, however im seeing this:

Post job cleanup.
/usr/bin/docker image ls --format={{.ID}} {{.Repository}}:{{.Tag}} --filter=dangling=false --filter=reference='openmined/*'
There is no image to save.

However when using tunshell.com to debug the CI process I can run that exact command and it outputs several hashes...
https://github.com/OpenMined/PySyft/runs/3511338358?check_suite_focus=true

[jsirianni]

@jamie-wearsafe your fork worked great for me. I needed to cache a single image to be restored by a job later in my workflow. Without the filter, the cache would load all images from previous runs (because they are uniquely tagged).

      - name: Set Image Tag
        run: echo "GITHUB_SHA_SHORT=$(git rev-parse --short HEAD)" >> $GITHUB_ENV
      - name: Cache Container Image
        uses: Broadshield/action-docker-layer-caching@main
        with:
          filter: reference=account/imagename:${{ env.GITHUB_SHA_SHORT }}

[jamie-wearsafe]

@jsirianni Nice to see :)

[jamie-wearsafe]

@madhavajay i know its been a long time but do you need that issue addressed?

[madhavajay]

Hi @jamie-wearsafe yeah that would be great, I am currently just using cache all and my guess is the cache is getting regularly evicted:

      - name: docker cache
        uses: actions/cache@v3
        if: steps.changes.outputs.stack == 'true'
        continue-on-error: true
        with:
          path: .docker-cache
          key: ${{ runner.os }}-docker
          restore-keys: |
            ${{ runner.os }}-docker

[jamie-wearsafe]

@madhavajay so, can you try adjusting your filter for me?

      - name: Cache OpenmineD Container Image
        uses: Broadshield/action-docker-layer-caching@main
        with:
          filter: reference="*/openmined*:*"
          key: docker-layer-caching-${{ github.workflow }}-${{ github.head_ref || github.ref }}-${{ github.event_name }}-{hash}
          restore-keys: |
            docker-layer-caching-${{ github.workflow }}-${{ github.head_ref || github.ref }}-${{ github.event_name }}-{hash}
            docker-layer-caching-${{ github.workflow }}-${{ github.head_ref || github.ref }}-${{ github.event_name }}
            docker-layer-caching-${{ github.workflow }}-${{ github.head_ref || github.ref }}
            docker-layer-caching-${{ github.workflow }}
            docker-layer-caching-