Skip to content

Commit

Permalink
Docker multistage (#48)
Browse files Browse the repository at this point in the history
* testing docker from github

* minor fix to travis-ci

* changed branch to $TRAVIS_BRANCH

* adding env variables

* adding env variables

* double checking that travis variables are set up

* double checking that travis variables are set up

* double checking that travis variables are set up

* changing from deploy tag to after_success tag

* lowercase slug

* trying to fix the build tag name

* trying to fix the build tag name

* chewbbaca with python3

* chewbbaca with python3

* debugging docker tag

* debugging docker tag

* debugging docker tag

* debugging docker tag

* debugging docker tag

* debugging docker tag

* fixing docker tag

* fixing docker login

* explicitly giving lskatz for username

* secret variable space

* trying to get secret env variables for docker login

* trying out docker caching

* Update .travis.yml

* fix "docker save"

* trying the other kind of caching

* fixed single quote

* view dockerImages.stdout

* added cache folder

* test for size of dockerImages.stdout

* from xargs to bash loop

* fixed a quote

* a quick rm cache

* Update .travis.yml

* after removing all cache

* rm cache

* checking if file exists

* rm cache

* back to normal

* Create travis-docker.yml

* Update travis-docker.yml

* Update travis-docker.yml

* docker login

* passwordless login

* docker build

* updating run commands for variable printing

* updating run commands for variable printing

* updating run commands for variable printing

* more debugging; switching build directory to HOME

* more debugging

* more debugging

* more debugging

* actions/checkout@v2

* actions/checkout on this branch

* actions/checkout on this branch

* actions/checkout on this branch

* actions/checkout on this branch

* fixed space typo

* fixed working directory during docker build

* fixed working directory during docker build

* fixing docker build

* fixing docker build

* fixing docker build

* fixing docker build

* fixing docker build

* trying docker push

* using a more appropriate env variable

* fixed case on docker tag

* moving env out of job and to global

* separating out to multiple jobs that depend on each other, in preparation for any concurrency

* some concurrency

* m

* m

* added some debugging

* m

* added init and debugging jobs

* init needs debugging

* exporting REMOTE_TAG

* figuring out variable continuity

* figuring out variable continuity

* passing outputs between jobs

* passing outputs between jobs

* passing outputs between jobs

* debugging more

* debugging more

* debugging more with JSON

* debugging more with JSON

* debugging more with JSON2

* moved git checkout to init job

* maybe I needed to do fromJSON. And moved git checkout back.

* JSON3

* JSON3

* JSON4

* JSON5

* JSON6

* test slug

* removed all underscores and dashes in this output chain

* export slug

* export slug2

* just trying to make env work at all

* env2

* env3

* making the env work properly with format()

* env2

* env3

* env4

* env5

* env6

* trying things out

* trying things out

* trying things out

* trying things out

* trying things out

* just making the variables work

* docker cache action

* docker cache action

* Trying to follow satackey more closely

* Update travis-docker.yml

* added docker login

* login before tagging

* Rename travis-docker.yml to travis-docker.yml.bak

* Create github-docker.yml

* removed needs:init

* corrected the docker repo

* pushing to both docker and github

https://docs.github.com/en/actions/guides/publishing-docker-images

* renamed the github container name from my-image to sneakernet

* tags: latest

* adding extra LC_ALL to the end

* newer blast in Dockerfile

* blast library COPY statements

* reverting blast libs

* getting blast from mlst container

* manual installation of staramr

* manual installation of staramr2

* tag with hashsum values too

* trying to invoke the sha sum

* adding apt-get packages for staramr

* trying out docker push v2

* trying docker action v2

* fixed newline error

* removed ipython-notebook

* Update github-docker.yml

* Update Dockerfile

* numpy installation through pip now

* Removing multistage for pip-based images

* fixed apt-get command syntax

* changed tags to a value instead a list

* updated the tag

docker/build-push-action#160

* list of tags

* pushing to the github registry (?)

* one space before pipe

* Not sending to github packages; yes sending multiple tags to dockerhub

* removed list again

* Rename github-docker.yml to github-docker.yml.bak

* Create build-and-push-container.yml

* added this branch for github actions

* added libpython3.5-dev

```
#130 [linux/386 stage-11 34/47] RUN pip3 install poetry &&  git clone https:/...
#130 11.73   building '_openssl' extension
#130 11.73   creating build/temp.linux-x86_64-3.6/build
#130 11.73   creating build/temp.linux-x86_64-3.6/build/temp.linux-x86_64-3.6
#130 11.73   i686-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c build/temp.linux-x86_64-3.6/_openssl.c -o build/temp.linux-x86_64-3.6/build/temp.linux-x86_64-3.6/_openssl.o -Wconversion -Wno-error=sign-conversion
#130 11.73   build/temp.linux-x86_64-3.6/_openssl.c:50:14: fatal error: pyconfig.h: No such file or directory
#130 11.73    #    include <pyconfig.h>
#130 11.73                 ^~~~~~~~~~~~
#130 11.73   compilation terminated.
#130 11.73   error: command 'i686-linux-gnu-gcc' failed with exit status 1
#130 11.73   ----------------------------------------
#130 11.73   ERROR: Failed building wheel for cryptography
#130 11.73 Failed to build cryptography
#130 11.74 ERROR: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly
#130 ERROR: executor failed running [/bin/sh -c pip3 install poetry &&  git clone https://github.com/hcdenbakker/SalmID.git --branch 0.1.23 --single-branch &&  cd SalmID &&  poetry build -vvv &&  pip3 install dist/salmid*.whl]: buildkit-runc did not terminate successfully
```

* fixed libpython-dev

* libpython3-dev

* newline.  Does it make things work

* separate some pip3 installations for staramr; removed numpy pip3

* removing staramr and salmid for right now

* removed all platforms except linux/amd64

* uncomment pip commands

* combined some pip commands for fewer layers

* Combined some chewbbaca steps into single layers

* RUN echo "LC_ALL=C" >> /etc/environment

* switched to UTF-8 per instructions

* apt-get install locales

Co-authored-by: Lee Katz - Aspen <gzu2@cdc.gov>
  • Loading branch information
lskatz and lskatz authored Oct 13, 2020
1 parent 7f15f01 commit bb9c56e
Show file tree
Hide file tree
Showing 6 changed files with 344 additions and 118 deletions.
96 changes: 96 additions & 0 deletions .github/workflows/build-and-push-container.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
# https://docs.github.com/en/actions/reference/encrypted-secrets#using-encrypted-secrets-in-a-workflow
# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions
# https://docs.github.com/en/actions/guides/publishing-docker-images
name: github-docker
on:
push:
branches:
- docker-multistage



env:
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
DOCKER_USERNAME: lskatz
CI_PROJECT_NAME: sneakernet
github_repository: lskatz/sneakernet
tag: fake-tag

jobs:
push_to_registry:
name: Push Docker image to multiple registries v2
runs-on: ubuntu-18.04
steps:
- name: Check out the repo
uses: actions/checkout@v2
- name: Set up QEMU
uses: docker/setup-qemu-action@v1
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
with:
driver-opts: image=moby/buildkit:master
- name: Prepare env
id: prep
run: |
DOCKER_IMAGE=${{ env.github_repository }}
VERSION=${GITHUB_SHA}
if [ "${{ github.event_name }}" = "schedule" ]; then
VERSION=nightly
elif [[ $GITHUB_REF == refs/tags/* ]]; then
VERSION=${GITHUB_REF#refs/tags/}
elif [[ $GITHUB_REF == refs/heads/* ]]; then
VERSION=$(echo ${GITHUB_REF#refs/heads/} | sed -r 's#/+#-#g')
if [ "${{ github.event.repository.default_branch }}" = "$VERSION" ]; then
VERSION=edge
fi
elif [[ $GITHUB_REF == refs/pull/* ]]; then
VERSION=pr-${{ github.event.number }}
fi
TAGS="${DOCKER_IMAGE}:${VERSION}"
TAGS="$TAGS,ghcr.io/${DOCKER_IMAGE}:${VERSION}"
if [[ $VERSION =~ ^v[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}$ ]]; then
MINOR=${VERSION%.*}
MAJOR=${MINOR%.*}
TAGS="$TAGS,${DOCKER_IMAGE}:${MINOR},${DOCKER_IMAGE}:${MAJOR},${DOCKER_IMAGE}:latest"
TAGS="$TAGS,ghcr.io/${DOCKER_IMAGE}:${MINOR},ghcr.io/${DOCKER_IMAGE}:${MAJOR},ghcr.io/${DOCKER_IMAGE}:latest"
elif [ "${{ github.event_name }}" = "push" ]; then
TAGS="$TAGS,${DOCKER_IMAGE}:sha-${GITHUB_SHA::8}"
TAGS="$TAGS,ghcr.io/${DOCKER_IMAGE}:sha-${GITHUB_SHA::8}"
fi
echo ::set-output name=version::${VERSION}
echo ::set-output name=tags::${TAGS}
echo ::set-output name=created::$(date -u +'%Y-%m-%dT%H:%M:%SZ')
- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: lskatz
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Login to GitHub Container Registry
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.CR_PAT }}

- name: Build and push
id: docker_build
uses: docker/build-push-action@v2
with:
context: .
file: ./Dockerfile
platforms: linux/amd64
push: ${{ github.event_name != 'pull_request' }}
tags: ${{ steps.prep.outputs.tags }}
labels: |
org.opencontainers.image.title=${{ github.event.repository.name }}
org.opencontainers.image.description=${{ github.event.repository.description }}
org.opencontainers.image.url=${{ github.event.repository.html_url }}
org.opencontainers.image.source=${{ github.event.repository.clone_url }}
org.opencontainers.image.version=${{ steps.prep.outputs.version }}
org.opencontainers.image.created=${{ steps.prep.outputs.created }}
org.opencontainers.image.revision=${{ github.sha }}
org.opencontainers.image.licenses=${{ github.event.repository.license.spdx_id }}
74 changes: 74 additions & 0 deletions .github/workflows/github-docker.yml.bak
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
# https://docs.github.com/en/actions/reference/encrypted-secrets#using-encrypted-secrets-in-a-workflow
# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions
# https://docs.github.com/en/actions/guides/publishing-docker-images
name: github-docker
on:
push:
branches:
- docker-multistage

env:
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
DOCKER_USERNAME: lskatz
#DOCKER_USERNAME_SECRET: ${{ secrets.DOCKER_USERNAME }}
CI_PROJECT_NAME: sneakernet
github_repository: lskatz/sneakernet
#SLUG: ${{ format('lskatz/sneakernet:{0}', GITHUB_SHA ) }}
#LOCAL_TAG_LATEST: ${{ format('sneakernet:{0}', ${{ env.GITHUB_SHA }} ) }}

jobs:
push_to_registry_v1:
name: Push Docker image to multiple registries v1
runs-on: ubuntu-18.04
steps:
- name: Check out the repo
uses: actions/checkout@v2
- name: Push to Docker Hub
uses: docker/build-push-action@v1
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
repository: lskatz/sneakernet
tags: |
latest
${{ github.GITHUB_SHA }}
#tag_with_ref: true
- name: Push to GitHub Packages
uses: docker/build-push-action@v1
with:
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: docker.pkg.github.com
repository: lskatz/sneakernet/sneakernet
tags: |
latest
${{ github.GITHUB_SHA }}
#tag_with_ref: true
push_to_registry_v2:
name: Push Docker image to multiple registries v2
runs-on: ubuntu-18.04
steps:
- name: Check out the repo
uses: actions/checkout@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
- name: Login to DockerHub
uses: docker/login-action@v1
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_PASSWORD }}
- name: Build and push
uses: docker/build-push-action@v2
with:
context: .
file: ./Dockerfile
pull: true
push: true
#build-args:
cache-from: type=registry,ref=lskatz/sneakernet
cache-to: type=registry,ref=lskatz/sneakernet
tags: lskatz/sneakernet:latest
# lskatz/sneakernet:${{ github.GITHUB_SHA }}
# docker.pkg.github.com/lskatz/sneakernet:latest
# docker.pkg.github.com/lskatz/sneakernet:${{ github.GITHUB_SHA }}
#
62 changes: 62 additions & 0 deletions .github/workflows/travis-docker.yml.bak
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
# https://docs.github.com/en/actions/reference/encrypted-secrets#using-encrypted-secrets-in-a-workflow
# https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions
name: travis-docker
on:
push:
branches:
- docker-multistage

env:
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
DOCKER_USERNAME: lskatz
#DOCKER_USERNAME_SECRET: ${{ secrets.DOCKER_USERNAME }}
CI_PROJECT_NAME: sneakernet
github_repository: lskatz/sneakernet
#SLUG: ${{ format('lskatz/sneakernet:{0}', GITHUB_SHA ) }}
#LOCAL_TAG_LATEST: ${{ format('sneakernet:{0}', ${{ env.GITHUB_SHA }} ) }}

jobs:
init:
runs-on: ubuntu-18.04
steps:
- id: todo
run: echo "todo"

debugging:
runs-on: ubuntu-18.04
needs: init
steps:
- id: print-globals
run: |
echo "Docker username starts with ${DOCKER_USERNAME:0:3}..."
echo "CI_PROJECT_NAME:GITHUB_SHA $CI_PROJECT_NAME:$GITHUB_SHA"
echo "HOME:$HOME"
echo "PWD:$PWD"
echo "GITHUB_WORKSPACE:$GITHUB_WORKSPACE"
echo "GITHUB_ACTOR/CI_PROJECT_NAME:GITHUB_SHA $github_repository:$GITHUB_SHA"

docker_build:
runs-on: ubuntu-18.04
needs: init
steps:
- uses: actions/checkout@v2
with:
ref: docker-multistage
fetch-depth: 1
- uses: satackey/action-docker-layer-caching@v0.0.8
continue-on-error: true
- id: ls-all-the-things
run: |
ls -R $GITHUB_WORKSPACE
ls -lha $GITHUB_WORKSPACE
- id: docker-build
#run: docker build -t $CI_PROJECT_NAME:$GITHUB_SHA $GITHUB_WORKSPACE
run: docker build . --file $GITHUB_WORKSPACE/Dockerfile --tag $CI_PROJECT_NAME:$GITHUB_SHA
- id: docker-login
run: echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin
- id: docker-tag-latest
run: |
docker tag $CI_PROJECT_NAME:$GITHUB_SHA $github_repository:$GITHUB_SHA
- id: docker-push-latest
run: docker push $CI_PROJECT_NAME:$GITHUB_SHA

139 changes: 40 additions & 99 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,105 +2,46 @@ language: perl
os: linux
perl:
- "5.24-shrplib"
python:
- "3.7"
dist: xenial
#services:
# - docker
addons:
apt:
#sources:
packages:
- build-essential
- libberkeleydb-perl # helps with perl module DB_FILE
- bioperl
- bioperl-run
- ncbi-blast+
- prodigal
- tree
- sendmail
- python3
- python3-pip
#- kraken # travis kraken testing for another day
before_install:
- sudo update-alternatives --install /usr/bin/python python /usr/bin/python3 1
- sudo update-alternatives --set python /usr/bin/python3
- python --version
- pip3 --version
# Configure cpanm properly
- cpanm --local-lib=~/perl5 local::lib && eval $(perl -I ~/perl5/lib/perl5/ -Mlocal::lib)
# Docker
#- docker pull staphb/skesa:2.3.0
#- docker pull staphb/mlst:2.16.2
#- docker pull staphb/prokka:1.14.0
#- docker pull staphb/kraken:1.1.1
#- export PATH=$PATH:$(pwd)/scripts/dockerAliases
# Perl modules
- export PERL5LIB=$PERL5LIB:/usr/share/perl5
- export PERL5LIB=$PERL5LIB:$HOME/lib/perl5 # user libs
- export PERL5LIB=$PERL5LIB:$HOME/lib/perl5/x86_64-linux-gnu-thread-multi # user libs
- export PERL5LIB=$PERL5LIB:$HOME/lib/perl5/x86_64-linux-gnu-thread-multi/auto # user libs
#- find / -type f -name SeqIO.pm 2>/dev/null || echo "SeqIO.pm NOT FOUND"
- echo $PERL5LIB | tr ':' '\n' | cat -n
- cpanm -l ~ Bio::FeatureIO --notest --force
- cpanm -l ~ Statistics::Descriptive
- cpanm -l ~ File::Slurp
- cpanm -l ~ Config::Simple
- tree -d $HOME -L 5
- find $HOME -type f -name Slurp.pm
- find $HOME -type f -name Descriptive.pm
- find $HOME -type f -name Simple.pm
- perl -MData::Dumper -MFile::Slurp -MStatistics::Descriptive -Mthreads -e 'print Dumper \%INC'
- perl -MBio::SeqIO -e 1; # Check for Bio::SeqIO from bioperl
#- cpanm -l ~ Bio::FeatureIO --force --notest --verbose # b/c it wasn't getting installed properly somehow with cpanm --installdeps
# perl debugging statement
# CGP
- wget --progress=dot:mega --no-check-certificate https://github.com/lskatz/CG-Pipeline/archive/v0.5.tar.gz
- tar zxvf v0.5.tar.gz
- export PATH=$PATH:$(pwd -P)/CG-Pipeline-0.5/scripts
- wget --progress=dot:mega --no-check-certificate https://github.com/tseemann/shovill/archive/v1.0.0.tar.gz
- tar zxvf v1.0.0.tar.gz
- export PATH=$PATH:$(pwd -P)/shovill-1.0.0/bin
- tree $(pwd -P)/shovill*
# Python things
- export PATH=$PATH:$HOME/.local/bin
# staramr
#- pip3 install staramr --user
# chewbbaca
#- pip install chewbbaca --user
#- find $HOME -maxdepth 3 -name chewBBACA.py
script: true
install:
# Kraken testing
#- wget --progress=dot:mega --no-check-certificate https://ccb.jhu.edu/software/kraken/dl/minikraken_20171019_4GB.tgz
#- tar zxvf minikraken_20171019_4GB.tgz
# Krona
- wget --progress=dot:mega --no-check-certificate https://github.com/marbl/Krona/archive/xl2.5.tar.gz
- tar zxvf xl2.5.tar.gz
- perl Krona-xl2.5/KronaTools/install.pl -prefix $HOME/krona
# Configure SneakerNet
- cp -rv config.bak config
- sed -i '/KRAKEN_DEFAULT_DB/d' config/settings.conf
- sed -i '/KRAKENDIR/d' config/settings.conf
- echo -e "KRAKEN_DEFAULT_DB\t$(pwd -P)/minikraken_20171013_4GB" >> config/settings.conf
- echo -e "KRAKENDIR\t$HOME/.linuxbrew/bin" >> config/settings.conf
- sed -i '/KRONADIR/d' config/settings.conf
- echo -e "KRONADIR\t$HOME/krona/bin" >> config/settings.conf
- head -n 50 config/*.conf # display configuration file
- export PATH=$PATH:$(pwd -P)/scripts:$(pwd -P)/SneakerNet.plugins
script:
- cpanm --installdeps --notest .
- perl Makefile.PL
- make test
- echo "Just testing docker on this branch"

after_success:
- CI_PROJECT_NAME=sneakernet
- docker build -t $CI_PROJECT_NAME:$TRAVIS_COMMIT .
- SLUG=$(echo "$TRAVIS_REPO_SLUG" | tr '[:upper:]' '[:lower:]')
- echo "$CI_PROJECT_NAME"
- echo $SLUG
- echo $REPOSITORY_URL
- echo $TRAVIS_COMMIT
- DOCKER_PASSWORD=${{secrets.DOCKER_PASSWORD}}
- DOCKER_USERNAME=${{secrets.DOCKER_USERNAME}}
- echo $DOCKER_USERNAME
- echo docker tag $CI_PROJECT_NAME:$TRAVIS_COMMIT $SLUG:$TRAVIS_COMMIT
- docker tag $CI_PROJECT_NAME:$TRAVIS_COMMIT $SLUG:$TRAVIS_COMMIT
- docker tag $CI_PROJECT_NAME:$TRAVIS_COMMIT $SLUG:latest
- echo "$DOCKER_PASSWORD" | docker login --username "$DOCKER_USERNAME" --password-stdin
- docker push $CI_PROJECT_NAME:$TRAVIS_COMMIT
- docker push $CI_PROJECT_NAME:latest

#https://stackoverflow.com/a/58505735
before_install:
# - rm -rvf $HOME/dockerImages.stdout $HOME/docker/*
- for file in $HOME/docker/*.tar.gz; do if [ ! -e "$file" ]; then continue; fi; echo "loading $file"; zcat "$file" | docker load; done
before_cache:
- mkdir -p $HOME/docker
- docker images -a --filter='dangling=false' --format '{{.Repository}}:{{.Tag}} {{.ID}}' > $HOME/dockerImages.stdout
- cat $HOME/dockerImages.stdout
- if [[ -s $HOME/dockerImages.stdout ]]; then xargs -n 2 -t sh -c 'test -e $HOME/docker/$1.tar.gz || docker save $0 | gzip -2 > $HOME/docker/$1.tar.gz' < $HOME/dockerImages.stdout; fi
cache:
bundler: true
directories:
- $HOME/docker

# deploying to dockerhub
deploy:
on:
branch: master
provider: script
script:
- docker build -t $CI_PROJECT_NAME:$TRAVIS_REPO_SLUG .
- docker tag ${CI_PROJECT_NAME}:${TRAVIS_REPO_SLUG} $REPOSITORY_URL/$CI_PROJECT_NAME:$TRAVIS_PULL_REQUEST_SHA
- docker tag ${CI_PROJECT_NAME}:${TRAVIS_REPO_SLUG} $REPOSITORY_URL/$CI_PROJECT_NAME:latest
- echo "$DOCKER_PASSWORD" | docker login -u "$DOCKER_USERNAME" --password-stdin
- docker push $REPOSITORY_URL/$CI_PROJECT_NAME:$TRAVIS_PULL_REQUEST_SHA
- docker push $REPOSITORY_URL/$CI_PROJECT_NAME:latest
#deploy:
# provider: script
# script: bash .travis-ci/docker_push
# on:
# branch: $TRAVIS_BRANCH

Loading

0 comments on commit bb9c56e

Please sign in to comment.