docker image building for all of the docker images our integration tests require. #622

julialongtin · 2019-02-13T18:10:51Z

No description provided.

…it checkouts to specified commits, add minio, bump the tini version, and fix fakesqs and localstack.

…lso, amd64 support.

jschaul

(I'll review the Makefile tomorrow)

deploy/docker-ephemeral/build/Makefile

deploy/docker-ephemeral/build/README.md

…to JEL-DOCKER-1

jschaul

The parallel execution seems not to work (it stops halfway), due to the interleaved output it's not easy to see what is going wrong, however.

I've seen this in the logs:

no such manifest: docker.io/wireserver/dynamodb_local:0.0.7-386
Makefile:105: recipe for target 'manifest-create-dynamodb_local' failed
make: *** [manifest-create-dynamodb_local] Error 1
make: *** Waiting for unfinished jobs....

Currently trying without parallelism...

doc/Dependencies.md

deploy/docker-ephemeral/build/README.md

deploy/docker-ephemeral/build/Makefile

jschaul · 2019-02-14T16:51:39Z

Running DIST=ALPINE make while, in another terminal, make is running, leads to this failure:

(...)
Step 3/22 : RUN apk add --no-cache openjdk8-jre su-exec
 ---> Running in 73cb9f624495
standard_init_linux.go:190: exec user process caused "exec format error"
The command '/bin/sh -c apk add --no-cache openjdk8-jre su-exec' returned a non-zero code: 1
docker push wireserver/elasticsearch:0.0.7-arm32v6 | cat
The push refers to repository [docker.io/wireserver/elasticsearch]
tag does not exist: wireserver/elasticsearch:0.0.7-arm32v6
docker manifest create wireserver/elasticsearch:0.0.7 wireserver/elasticsearch:0.0.7-amd64 wireserver/elasticsearch:0.0.7-386 wireserver/elasticsearch:0.0.7-arm32v6 --amend
no such manifest: docker.io/wireserver/elasticsearch:0.0.7-amd64
Makefile:105: recipe for target 'manifest-create-elasticsearch' failed
make: *** [manifest-create-elasticsearch] Error 1

Full logs of DIST=ALPINE make:

DIST=ALPINE make
git clone https://github.com/blacktop/docker-elasticsearch-alpine.git elasticsearch-all
Cloning into 'elasticsearch-all'...
remote: Enumerating objects: 54, done.
remote: Counting objects: 100% (54/54), done.
remote: Compressing objects: 100% (22/22), done.
remote: Total 1386 (delta 39), reused 44 (delta 31), pack-reused 1332
Receiving objects: 100% (1386/1386), 357.06 KiB | 1.41 MiB/s, done.
Resolving deltas: 100% (974/974), done.
cd elasticsearch-all && git reset --hard 06779bd8db7ab81d6706c8ede9981d815e143ea3
HEAD is now at 06779bd Update README.md
cp -R elasticsearch-all/5.6/ elasticsearch
cd elasticsearch && cat Dockerfile | sed "s/^\(MAINTAINER\).*/\1 Wire \"backend@wire.com\"/" | sed "s=^\(FROM \)\(.*\)$=\1\2=" > Dockerfile-amd64
cd elasticsearch && docker build -t wireserver/elasticsearch:0.0.7-amd64 -f Dockerfile-amd64 . | cat
Sending build context to Docker daemon  20.99kB
Step 1/22 : FROM alpine:3.7
3.7: Pulling from library/alpine
48ecbb6b270e: Pulling fs layer
48ecbb6b270e: Download complete
48ecbb6b270e: Pull complete
Digest: sha256:4013ae48be82862082484fc3cc68120d42b752c156abad5fd3877543116994ce
Status: Downloaded newer image for alpine:3.7
 ---> bc8fb6e6e49d
Step 2/22 : LABEL maintainer "https://github.com/blacktop"
 ---> Running in 6e1f46321c46
Removing intermediate container 6e1f46321c46
 ---> 2f2110665b37
Step 3/22 : RUN apk add --no-cache openjdk8-jre su-exec
 ---> Running in 9eb915130579
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
(1/40) Installing libffi (3.2.1-r4)
(2/40) Installing libtasn1 (4.12-r3)
(3/40) Installing p11-kit (0.23.2-r2)
(4/40) Installing p11-kit-trust (0.23.2-r2)
(5/40) Installing ca-certificates (20171114-r0)
(6/40) Installing java-cacerts (1.0-r0)
(7/40) Installing libxau (1.0.8-r2)
(8/40) Installing libbsd (0.8.6-r1)
(9/40) Installing libxdmcp (1.1.2-r4)
(10/40) Installing libxcb (1.12-r1)
(11/40) Installing libx11 (1.6.6-r0)
(12/40) Installing libxcomposite (0.4.4-r1)
(13/40) Installing libxext (1.3.3-r2)
(14/40) Installing libxi (1.7.9-r1)
(15/40) Installing libxrender (0.9.10-r2)
(16/40) Installing libxtst (1.2.3-r1)
(17/40) Installing alsa-lib (1.1.4.1-r2)
(18/40) Installing libbz2 (1.0.6-r6)
(19/40) Installing libpng (1.6.34-r1)
(20/40) Installing freetype (2.8.1-r4)
(21/40) Installing libgcc (6.4.0-r5)
(22/40) Installing giflib (5.1.4-r1)
(23/40) Installing libjpeg-turbo (1.5.3-r2)
(24/40) Installing libstdc++ (6.4.0-r5)
(25/40) Installing openjdk8-jre-lib (8.191.12-r0)
(26/40) Installing java-common (0.1-r0)
(27/40) Installing krb5-conf (1.0-r1)
(28/40) Installing libcom_err (1.43.7-r0)
(29/40) Installing keyutils-libs (1.5.10-r0)
(30/40) Installing libverto (0.3.0-r0)
(31/40) Installing krb5-libs (1.15.4-r0)
(32/40) Installing lcms2 (2.8-r1)
(33/40) Installing nspr (4.17-r0)
(34/40) Installing sqlite-libs (3.25.3-r0)
(35/40) Installing nss (3.34.1-r0)
(36/40) Installing pcsc-lite-libs (1.8.22-r0)
(37/40) Installing lksctp-tools (1.0.17-r0)
(38/40) Installing openjdk8-jre-base (8.191.12-r0)
(39/40) Installing openjdk8-jre (8.191.12-r0)
(40/40) Installing su-exec (0.2-r0)
Executing busybox-1.27.2-r11.trigger
Executing ca-certificates-20171114-r0.trigger
Executing java-common-0.1-r0.trigger
OK: 81 MiB in 53 packages
Removing intermediate container 9eb915130579
 ---> 3e44b4927897
Step 4/22 : ENV VERSION 5.6.14
 ---> Running in dc4d5c931f8c
Removing intermediate container dc4d5c931f8c
 ---> a005dde2a7ee
Step 5/22 : ENV DOWNLOAD_URL "https://artifacts.elastic.co/downloads/elasticsearch"
 ---> Running in 9ed302342e27
Removing intermediate container 9ed302342e27
 ---> 58532297a3f5
Step 6/22 : ENV ES_TARBAL "${DOWNLOAD_URL}/elasticsearch-${VERSION}.tar.gz"
 ---> Running in 96d3da006ddc
Removing intermediate container 96d3da006ddc
 ---> d18bfbe34b6b
Step 7/22 : ENV ES_TARBALL_ASC "${DOWNLOAD_URL}/elasticsearch-${VERSION}.tar.gz.asc"
 ---> Running in 9feefa14a926
Removing intermediate container 9feefa14a926
 ---> 7c79f27064e7
Step 8/22 : ENV EXPECTED_SHA_URL "${DOWNLOAD_URL}/elasticsearch-${VERSION}.tar.gz.sha512"
 ---> Running in 24e6bf858e06
Removing intermediate container 24e6bf858e06
 ---> ed96ea779e33
Step 9/22 : ENV ES_TARBALL_SHA "d384797c652d5ceb2e081c3a05d62a8af3c543e818fa501bc0caff0826e6a56447f0dfc03d1e7391f7eb043e500c8a9942083bb141d18a02d464c5e5d8011bb2"
 ---> Running in fb252e66643c
Removing intermediate container fb252e66643c
 ---> 150d574eb60f
Step 10/22 : ENV GPG_KEY "46095ACC8548582C1A2699A9D27D666CD88E42B4"
 ---> Running in 0b734cf48d06
Removing intermediate container 0b734cf48d06
 ---> e814dae3c497
Step 11/22 : RUN apk add --no-cache bash
 ---> Running in 19791d9fa286
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
(1/6) Installing pkgconf (1.3.10-r0)
(2/6) Installing ncurses-terminfo-base (6.0_p20171125-r1)
(3/6) Installing ncurses-terminfo (6.0_p20171125-r1)
(4/6) Installing ncurses-libs (6.0_p20171125-r1)
(5/6) Installing readline (7.0.003-r0)
(6/6) Installing bash (4.4.19-r1)
Executing bash-4.4.19-r1.post-install
Executing busybox-1.27.2-r11.trigger
OK: 90 MiB in 59 packages
Removing intermediate container 19791d9fa286
 ---> 932b1de61d65
Step 12/22 : RUN apk add --no-cache -t .build-deps wget ca-certificates gnupg openssl   && set -ex   && cd /tmp   && echo "===> Install Elasticsearch..."   && wget --progress=bar:force -O elasticsearch.tar.gz "$ES_TARBAL";   if [ "$ES_TARBALL_SHA" ]; then   echo "$ES_TARBALL_SHA *elasticsearch.tar.gz" | sha512sum -c -;   fi;   if [ "$ES_TARBALL_ASC" ]; then   wget --progress=bar:force -O elasticsearch.tar.gz.asc "$ES_TARBALL_ASC";   export GNUPGHOME="$(mktemp -d)";   ( gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$GPG_KEY"   || gpg --keyserver pgp.mit.edu --recv-keys "$GPG_KEY"   || gpg --keyserver keyserver.pgp.com --recv-keys "$GPG_KEY" );   gpg --batch --verify elasticsearch.tar.gz.asc elasticsearch.tar.gz;   rm -rf "$GNUPGHOME" elasticsearch.tar.gz.asc || true;   fi;   tar -xf elasticsearch.tar.gz   && ls -lah   && mv elasticsearch-$VERSION /usr/share/elasticsearch   && adduser -D -h /usr/share/elasticsearch elasticsearch   && echo "===> Creating Elasticsearch Paths..."   && for path in   /usr/share/elasticsearch/data   /usr/share/elasticsearch/logs   /usr/share/elasticsearch/config   /usr/share/elasticsearch/config/scripts   /usr/share/elasticsearch/plugins   ; do   mkdir -p "$path";   chown -R elasticsearch:elasticsearch "$path";   done   && rm -rf /tmp/*   && apk del --purge .build-deps
 ---> Running in 717b3c507923
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
(1/20) Installing wget (1.20.1-r0)
(2/20) Installing libgpg-error (1.27-r1)
(3/20) Installing libassuan (2.4.4-r0)
(4/20) Installing libcap (2.25-r1)
(5/20) Installing pinentry (1.0.0-r0)
Executing pinentry-1.0.0-r0.post-install
(6/20) Installing libgcrypt (1.8.3-r0)
(7/20) Installing gmp (6.1.2-r1)
(8/20) Installing nettle (3.3-r0)
(9/20) Installing libunistring (0.9.7-r0)
(10/20) Installing gnutls (3.6.1-r0)
(11/20) Installing libksba (1.3.5-r0)
(12/20) Installing db (5.3.28-r0)
(13/20) Installing libsasl (2.1.26-r11)
(14/20) Installing libldap (2.4.45-r3)
(15/20) Installing npth (1.5-r1)
(16/20) Installing gnupg (2.2.3-r1)
(17/20) Installing libcrypto1.0 (1.0.2q-r0)
(18/20) Installing libssl1.0 (1.0.2q-r0)
(19/20) Installing openssl (1.0.2q-r0)
(20/20) Installing .build-deps (0)
Executing busybox-1.27.2-r11.trigger
Executing ca-certificates-20171114-r0.trigger
OK: 106 MiB in 79 packages
===> Install Elasticsearch...
+ cd /tmp
+ echo ===> Install Elasticsearch...
+ wget --progress=bar:force -O elasticsearch.tar.gz https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.14.tar.gz
--2019-02-14 16:43:34--  https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.14.tar.gz
Resolving artifacts.elastic.co... 151.101.2.222, 151.101.66.222, 151.101.130.222, ...
Connecting to artifacts.elastic.co|151.101.2.222|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 33898195 (32M) [application/x-gzip]
Saving to: 'elasticsearch.tar.gz'

elasticsearch.tar.g 100%[===================>]  32.33M  76.6MB/s    in 0.4s    

2019-02-14 16:43:35 (76.6 MB/s) - 'elasticsearch.tar.gz' saved [33898195/33898195]

+ [ d384797c652d5ceb2e081c3a05d62a8af3c543e818fa501bc0caff0826e6a56447f0dfc03d1e7391f7eb043e500c8a9942083bb141d18a02d464c5e5d8011bb2 ]
+ echo d384797c652d5ceb2e081c3a05d62a8af3c543e818fa501bc0caff0826e6a56447f0dfc03d1e7391f7eb043e500c8a9942083bb141d18a02d464c5e5d8011bb2 *elasticsearch.tar.gz
+ sha512sum -c -
elasticsearch.tar.gz: OK
+ [ https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.14.tar.gz.asc ]
+ wget --progress=bar:force -O elasticsearch.tar.gz.asc https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.14.tar.gz.asc
--2019-02-14 16:43:35--  https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.14.tar.gz.asc
Resolving artifacts.elastic.co... 151.101.2.222, 151.101.66.222, 151.101.130.222, ...
Connecting to artifacts.elastic.co|151.101.2.222|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 473 [text/plain]
Saving to: 'elasticsearch.tar.gz.asc'

elasticsearch.tar.g 100%[===================>]     473  --.-KB/s    in 0s      

2019-02-14 16:43:36 (108 MB/s) - 'elasticsearch.tar.gz.asc' saved [473/473]

+ mktemp -d
+ export GNUPGHOME=/tmp/tmp.bLPbfp
+ gpg --keyserver ha.pool.sks-keyservers.net --recv-keys 46095ACC8548582C1A2699A9D27D666CD88E42B4
gpg: keybox '/tmp/tmp.bLPbfp/pubring.kbx' created
gpg: keyserver receive failed: Address not available
+ gpg --keyserver pgp.mit.edu --recv-keys 46095ACC8548582C1A2699A9D27D666CD88E42B4
gpg: keyserver receive failed: No data
+ gpg --keyserver keyserver.pgp.com --recv-keys 46095ACC8548582C1A2699A9D27D666CD88E42B4
gpg: keyserver receive failed: Operation timed out
The command '/bin/sh -c apk add --no-cache -t .build-deps wget ca-certificates gnupg openssl   && set -ex   && cd /tmp   && echo "===> Install Elasticsearch..."   && wget --progress=bar:force -O elasticsearch.tar.gz "$ES_TARBAL";   if [ "$ES_TARBALL_SHA" ]; then   echo "$ES_TARBALL_SHA *elasticsearch.tar.gz" | sha512sum -c -;   fi;   if [ "$ES_TARBALL_ASC" ]; then   wget --progress=bar:force -O elasticsearch.tar.gz.asc "$ES_TARBALL_ASC";   export GNUPGHOME="$(mktemp -d)";   ( gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$GPG_KEY"   || gpg --keyserver pgp.mit.edu --recv-keys "$GPG_KEY"   || gpg --keyserver keyserver.pgp.com --recv-keys "$GPG_KEY" );   gpg --batch --verify elasticsearch.tar.gz.asc elasticsearch.tar.gz;   rm -rf "$GNUPGHOME" elasticsearch.tar.gz.asc || true;   fi;   tar -xf elasticsearch.tar.gz   && ls -lah   && mv elasticsearch-$VERSION /usr/share/elasticsearch   && adduser -D -h /usr/share/elasticsearch elasticsearch   && echo "===> Creating Elasticsearch Paths..."   && for path in   /usr/share/elasticsearch/data   /usr/share/elasticsearch/logs   /usr/share/elasticsearch/config   /usr/share/elasticsearch/config/scripts   /usr/share/elasticsearch/plugins   ; do   mkdir -p "$path";   chown -R elasticsearch:elasticsearch "$path";   done   && rm -rf /tmp/*   && apk del --purge .build-deps' returned a non-zero code: 2
docker push wireserver/elasticsearch:0.0.7-amd64 | cat
The push refers to repository [docker.io/wireserver/elasticsearch]
An image does not exist locally with the tag: wireserver/elasticsearch
cd elasticsearch && cat Dockerfile | sed "s/^\(MAINTAINER\).*/\1 Wire \"backend@wire.com\"/" | sed "s=^\(FROM \)\(.*\)$=\1i386/\2=" > Dockerfile-386
cd elasticsearch && docker build -t wireserver/elasticsearch:0.0.7-386 -f Dockerfile-386 . | cat
Sending build context to Docker daemon  24.06kB
Step 1/22 : FROM i386/alpine:3.7
3.7: Pulling from i386/alpine
0a615535caf9: Pulling fs layer
bb8fd280f9a4: Pulling fs layer
bb8fd280f9a4: Verifying Checksum
bb8fd280f9a4: Download complete
0a615535caf9: Download complete
0a615535caf9: Pull complete
bb8fd280f9a4: Pull complete
Digest: sha256:bf3ebda797f3ca009b0e391bac66246a202af2a479f9ea0df3ac9a90fb459d5f
Status: Downloaded newer image for i386/alpine:3.7
 ---> 5e175f432c4f
Step 2/22 : LABEL maintainer "https://github.com/blacktop"
 ---> Running in 7053ea1fbb18
Removing intermediate container 7053ea1fbb18
 ---> eaff780357a6
Step 3/22 : RUN apk add --no-cache openjdk8-jre su-exec
 ---> Running in 52320655ba3e
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86/APKINDEX.tar.gz
(1/40) Installing libffi (3.2.1-r4)
(2/40) Installing libtasn1 (4.12-r3)
(3/40) Installing p11-kit (0.23.2-r2)
(4/40) Installing p11-kit-trust (0.23.2-r2)
(5/40) Installing ca-certificates (20171114-r0)
(6/40) Installing java-cacerts (1.0-r0)
(7/40) Installing libxau (1.0.8-r2)
(8/40) Installing libbsd (0.8.6-r1)
(9/40) Installing libxdmcp (1.1.2-r4)
(10/40) Installing libxcb (1.12-r1)
(11/40) Installing libx11 (1.6.6-r0)
(12/40) Installing libxcomposite (0.4.4-r1)
(13/40) Installing libxext (1.3.3-r2)
(14/40) Installing libxi (1.7.9-r1)
(15/40) Installing libxrender (0.9.10-r2)
(16/40) Installing libxtst (1.2.3-r1)
(17/40) Installing alsa-lib (1.1.4.1-r2)
(18/40) Installing libbz2 (1.0.6-r6)
(19/40) Installing libpng (1.6.34-r1)
(20/40) Installing freetype (2.8.1-r4)
(21/40) Installing libgcc (6.4.0-r5)
(22/40) Installing giflib (5.1.4-r1)
(23/40) Installing libjpeg-turbo (1.5.3-r2)
(24/40) Installing libstdc++ (6.4.0-r5)
(25/40) Installing openjdk8-jre-lib (8.191.12-r0)
(26/40) Installing java-common (0.1-r0)
(27/40) Installing krb5-conf (1.0-r1)
(28/40) Installing libcom_err (1.43.7-r0)
(29/40) Installing keyutils-libs (1.5.10-r0)
(30/40) Installing libverto (0.3.0-r0)
(31/40) Installing krb5-libs (1.15.4-r0)
(32/40) Installing lcms2 (2.8-r1)
(33/40) Installing nspr (4.17-r0)
(34/40) Installing sqlite-libs (3.25.3-r0)
(35/40) Installing nss (3.34.1-r0)
(36/40) Installing pcsc-lite-libs (1.8.22-r0)
(37/40) Installing lksctp-tools (1.0.17-r0)
(38/40) Installing openjdk8-jre-base (8.191.12-r0)
(39/40) Installing openjdk8-jre (8.191.12-r0)
(40/40) Installing su-exec (0.2-r0)
Executing busybox-1.27.2-r11.trigger
Executing ca-certificates-20171114-r0.trigger
Executing java-common-0.1-r0.trigger
OK: 81 MiB in 53 packages
Removing intermediate container 52320655ba3e
 ---> f0f9a4e908eb
Step 4/22 : ENV VERSION 5.6.14
 ---> Running in bef489914bb9
Removing intermediate container bef489914bb9
 ---> b0f719b681b4
Step 5/22 : ENV DOWNLOAD_URL "https://artifacts.elastic.co/downloads/elasticsearch"
 ---> Running in bf46d5de909c
Removing intermediate container bf46d5de909c
 ---> b2467ff73368
Step 6/22 : ENV ES_TARBAL "${DOWNLOAD_URL}/elasticsearch-${VERSION}.tar.gz"
 ---> Running in 8079c6f3eb8a
Removing intermediate container 8079c6f3eb8a
 ---> 902cfeb7b97b
Step 7/22 : ENV ES_TARBALL_ASC "${DOWNLOAD_URL}/elasticsearch-${VERSION}.tar.gz.asc"
 ---> Running in c63a1b372b80
Removing intermediate container c63a1b372b80
 ---> f51a60e6be8d
Step 8/22 : ENV EXPECTED_SHA_URL "${DOWNLOAD_URL}/elasticsearch-${VERSION}.tar.gz.sha512"
 ---> Running in 9194956f5b1c
Removing intermediate container 9194956f5b1c
 ---> 7302f759af51
Step 9/22 : ENV ES_TARBALL_SHA "d384797c652d5ceb2e081c3a05d62a8af3c543e818fa501bc0caff0826e6a56447f0dfc03d1e7391f7eb043e500c8a9942083bb141d18a02d464c5e5d8011bb2"
 ---> Running in 4f684b12c8d7
Removing intermediate container 4f684b12c8d7
 ---> a525abc71ed8
Step 10/22 : ENV GPG_KEY "46095ACC8548582C1A2699A9D27D666CD88E42B4"
 ---> Running in 6c3a11382b54
Removing intermediate container 6c3a11382b54
 ---> 5de0ef377ef6
Step 11/22 : RUN apk add --no-cache bash
 ---> Running in c4097eef3ee6
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86/APKINDEX.tar.gz
(1/6) Installing pkgconf (1.3.10-r0)
(2/6) Installing ncurses-terminfo-base (6.0_p20171125-r1)
(3/6) Installing ncurses-terminfo (6.0_p20171125-r1)
(4/6) Installing ncurses-libs (6.0_p20171125-r1)
(5/6) Installing readline (7.0.003-r0)
(6/6) Installing bash (4.4.19-r1)
Executing bash-4.4.19-r1.post-install
Executing busybox-1.27.2-r11.trigger
OK: 90 MiB in 59 packages
Removing intermediate container c4097eef3ee6
 ---> 32992c3866d1
Step 12/22 : RUN apk add --no-cache -t .build-deps wget ca-certificates gnupg openssl   && set -ex   && cd /tmp   && echo "===> Install Elasticsearch..."   && wget --progress=bar:force -O elasticsearch.tar.gz "$ES_TARBAL";   if [ "$ES_TARBALL_SHA" ]; then   echo "$ES_TARBALL_SHA *elasticsearch.tar.gz" | sha512sum -c -;   fi;   if [ "$ES_TARBALL_ASC" ]; then   wget --progress=bar:force -O elasticsearch.tar.gz.asc "$ES_TARBALL_ASC";   export GNUPGHOME="$(mktemp -d)";   ( gpg --keyserver ha.pool.sks-keyservers.net --recv-keys "$GPG_KEY"   || gpg --keyserver pgp.mit.edu --recv-keys "$GPG_KEY"   || gpg --keyserver keyserver.pgp.com --recv-keys "$GPG_KEY" );   gpg --batch --verify elasticsearch.tar.gz.asc elasticsearch.tar.gz;   rm -rf "$GNUPGHOME" elasticsearch.tar.gz.asc || true;   fi;   tar -xf elasticsearch.tar.gz   && ls -lah   && mv elasticsearch-$VERSION /usr/share/elasticsearch   && adduser -D -h /usr/share/elasticsearch elasticsearch   && echo "===> Creating Elasticsearch Paths..."   && for path in   /usr/share/elasticsearch/data   /usr/share/elasticsearch/logs   /usr/share/elasticsearch/config   /usr/share/elasticsearch/config/scripts   /usr/share/elasticsearch/plugins   ; do   mkdir -p "$path";   chown -R elasticsearch:elasticsearch "$path";   done   && rm -rf /tmp/*   && apk del --purge .build-deps
 ---> Running in f44bd5803563
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86/APKINDEX.tar.gz
(1/20) Installing wget (1.20.1-r0)
(2/20) Installing libgpg-error (1.27-r1)
(3/20) Installing libassuan (2.4.4-r0)
(4/20) Installing libcap (2.25-r1)
(5/20) Installing pinentry (1.0.0-r0)
Executing pinentry-1.0.0-r0.post-install
(6/20) Installing libgcrypt (1.8.3-r0)
(7/20) Installing gmp (6.1.2-r1)
(8/20) Installing nettle (3.3-r0)
(9/20) Installing libunistring (0.9.7-r0)
(10/20) Installing gnutls (3.6.1-r0)
(11/20) Installing libksba (1.3.5-r0)
(12/20) Installing db (5.3.28-r0)
(13/20) Installing libsasl (2.1.26-r11)
(14/20) Installing libldap (2.4.45-r3)
(15/20) Installing npth (1.5-r1)
(16/20) Installing gnupg (2.2.3-r1)
(17/20) Installing libcrypto1.0 (1.0.2q-r0)
(18/20) Installing libssl1.0 (1.0.2q-r0)
(19/20) Installing openssl (1.0.2q-r0)
(20/20) Installing .build-deps (0)
Executing busybox-1.27.2-r11.trigger
Executing ca-certificates-20171114-r0.trigger
OK: 107 MiB in 79 packages
===> Install Elasticsearch...
+ cd /tmp
+ echo ===> Install Elasticsearch...
+ wget --progress=bar:force -O elasticsearch.tar.gz https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.14.tar.gz
--2019-02-14 16:44:52--  https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.14.tar.gz
Resolving artifacts.elastic.co... 151.101.2.222, 151.101.66.222, 151.101.130.222, ...
Connecting to artifacts.elastic.co|151.101.2.222|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 33898195 (32M) [application/x-gzip]
Saving to: 'elasticsearch.tar.gz'

elasticsearch.tar.g 100%[===================>]  32.33M  37.1MB/s    in 0.9s    

2019-02-14 16:44:56 (37.1 MB/s) - 'elasticsearch.tar.gz' saved [33898195/33898195]

+ [ d384797c652d5ceb2e081c3a05d62a8af3c543e818fa501bc0caff0826e6a56447f0dfc03d1e7391f7eb043e500c8a9942083bb141d18a02d464c5e5d8011bb2 ]
+ echo d384797c652d5ceb2e081c3a05d62a8af3c543e818fa501bc0caff0826e6a56447f0dfc03d1e7391f7eb043e500c8a9942083bb141d18a02d464c5e5d8011bb2 *elasticsearch.tar.gz
+ sha512sum -c -
elasticsearch.tar.gz: OK
+ [ https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.14.tar.gz.asc ]
+ wget --progress=bar:force -O elasticsearch.tar.gz.asc https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.14.tar.gz.asc
--2019-02-14 16:44:56--  https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.14.tar.gz.asc
Resolving artifacts.elastic.co... 151.101.2.222, 151.101.66.222, 151.101.130.222, ...
Connecting to artifacts.elastic.co|151.101.2.222|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 473 [text/plain]
Saving to: 'elasticsearch.tar.gz.asc'

elasticsearch.tar.g 100%[===================>]     473  --.-KB/s    in 0s      

2019-02-14 16:44:56 (86.2 MB/s) - 'elasticsearch.tar.gz.asc' saved [473/473]

+ mktemp -d
+ export GNUPGHOME=/tmp/tmp.FEcAFJ
+ gpg --keyserver ha.pool.sks-keyservers.net --recv-keys 46095ACC8548582C1A2699A9D27D666CD88E42B4
gpg: keybox '/tmp/tmp.FEcAFJ/pubring.kbx' created
gpg: /tmp/tmp.FEcAFJ/trustdb.gpg: trustdb created
gpg: key D27D666CD88E42B4: public key "Elasticsearch (Elasticsearch Signing Key) <dev_ops@elasticsearch.org>" imported
gpg: Total number processed: 1
gpg:               imported: 1
+ gpg --batch --verify elasticsearch.tar.gz.asc elasticsearch.tar.gz
gpg: Signature made Wed Dec  5 22:36:41 2018 UTC
gpg:                using RSA key D27D666CD88E42B4
gpg: Good signature from "Elasticsearch (Elasticsearch Signing Key) <dev_ops@elasticsearch.org>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 4609 5ACC 8548 582C 1A26  99A9 D27D 666C D88E 42B4
+ rm -rf /tmp/tmp.FEcAFJ elasticsearch.tar.gz.asc
+ tar -xf elasticsearch.tar.gz
+ ls -lah
total 33116
drwxrwxrwt    1 root     root        4.0K Feb 14 16:44 .
drwxr-xr-x    1 root     root        4.0K Feb 14 16:44 ..
drwxr-xr-x    7 root     root        4.0K Feb 14 16:44 elasticsearch-5.6.14
-rw-r--r--    1 root     root       32.3M Dec 11 14:59 elasticsearch.tar.gz
+ mv elasticsearch-5.6.14 /usr/share/elasticsearch
+ adduser -D -h /usr/share/elasticsearch elasticsearch
+ echo ===> Creating Elasticsearch Paths...
+ mkdir -p /usr/share/elasticsearch/data
===> Creating Elasticsearch Paths...
+ chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data
+ mkdir -p /usr/share/elasticsearch/logs
+ chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/logs
+ mkdir -p /usr/share/elasticsearch/config
+ chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/config
+ mkdir -p /usr/share/elasticsearch/config/scripts
+ chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/config/scripts
+ mkdir -p /usr/share/elasticsearch/plugins
+ chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/plugins
+ rm -rf /tmp/elasticsearch.tar.gz
+ apk del --purge .build-deps
WARNING: Ignoring APKINDEX.70c88391.tar.gz: No such file or directory
WARNING: Ignoring APKINDEX.5022a8a2.tar.gz: No such file or directory
(1/20) Purging .build-deps (0)
(2/20) Purging wget (1.20.1-r0)
(3/20) Purging gnupg (2.2.3-r1)
(4/20) Purging pinentry (1.0.0-r0)
Executing pinentry-1.0.0-r0.post-deinstall
(5/20) Purging openssl (1.0.2q-r0)
(6/20) Purging libksba (1.3.5-r0)
(7/20) Purging libgcrypt (1.8.3-r0)
(8/20) Purging libassuan (2.4.4-r0)
(9/20) Purging libgpg-error (1.27-r1)
(10/20) Purging libcap (2.25-r1)
(11/20) Purging gnutls (3.6.1-r0)
(12/20) Purging nettle (3.3-r0)
(13/20) Purging gmp (6.1.2-r1)
(14/20) Purging libunistring (0.9.7-r0)
(15/20) Purging libldap (2.4.45-r3)
(16/20) Purging libsasl (2.1.26-r11)
(17/20) Purging db (5.3.28-r0)
(18/20) Purging npth (1.5-r1)
(19/20) Purging libssl1.0 (1.0.2q-r0)
(20/20) Purging libcrypto1.0 (1.0.2q-r0)
Executing busybox-1.27.2-r11.trigger
Executing ca-certificates-20171114-r0.trigger
OK: 90 MiB in 59 packages
Removing intermediate container f44bd5803563
 ---> 7f69f53823b1
Step 13/22 : COPY config/elastic /usr/share/elasticsearch/config
 ---> 1e92e83442fd
Step 14/22 : COPY config/logrotate /etc/logrotate.d/elasticsearch
 ---> ab0faed10317
Step 15/22 : COPY elastic-entrypoint.sh /
 ---> fd488e50648a
Step 16/22 : COPY docker-healthcheck /usr/local/bin/
 ---> b78cb06395c0
Step 17/22 : WORKDIR /usr/share/elasticsearch
 ---> Running in 9b08f969886e
Removing intermediate container 9b08f969886e
 ---> 69bea77af0a1
Step 18/22 : ENV PATH /usr/share/elasticsearch/bin:$PATH
 ---> Running in 698e78e023dc
Removing intermediate container 698e78e023dc
 ---> e42250a03f1f
Step 19/22 : VOLUME ["/usr/share/elasticsearch/data"]
 ---> Running in 284d095a803b
Removing intermediate container 284d095a803b
 ---> ae31760bea3a
Step 20/22 : EXPOSE 9200 9300
 ---> Running in 9eb3bfbad147
Removing intermediate container 9eb3bfbad147
 ---> 2d2c2fd5e330
Step 21/22 : ENTRYPOINT ["/elastic-entrypoint.sh"]
 ---> Running in 792f3a760b78
Removing intermediate container 792f3a760b78
 ---> 20ffe577717b
Step 22/22 : CMD ["elasticsearch"]
 ---> Running in e5b3d661bba6
Removing intermediate container e5b3d661bba6
 ---> 48a0ab5ab661
Successfully built 48a0ab5ab661
Successfully tagged wireserver/elasticsearch:0.0.7-386
docker push wireserver/elasticsearch:0.0.7-386 | cat
The push refers to repository [docker.io/wireserver/elasticsearch]
d0c2bb027f7f: Preparing
482ea20ce284: Preparing
25b00bfca0e5: Preparing
be0698e19985: Preparing
2d4abbd20262: Preparing
51f29bf31635: Preparing
61f85c9b388b: Preparing
cfc1645ce3d1: Preparing
ff5b27077a01: Preparing
51f29bf31635: Waiting
ff5b27077a01: Waiting
61f85c9b388b: Waiting
cfc1645ce3d1: Waiting
25b00bfca0e5: Pushed
d0c2bb027f7f: Pushed
be0698e19985: Pushed
cfc1645ce3d1: Mounted from i386/alpine
482ea20ce284: Pushed
ff5b27077a01: Mounted from i386/alpine
51f29bf31635: Pushed
2d4abbd20262: Pushed
61f85c9b388b: Pushed
0.0.7-386: digest: sha256:332173b148aa1f7c461fcd5e32f523d3c4005148f8f85659f8036898be11973f size: 2198
cd elasticsearch && cat Dockerfile | sed "s/^\(MAINTAINER\).*/\1 Wire \"backend@wire.com\"/" | sed "s=^\(FROM \)\(.*\)$=\1arm32v6/\2=" > Dockerfile-arm32v6
cd elasticsearch && docker build -t wireserver/elasticsearch:0.0.7-arm32v6 -f Dockerfile-arm32v6 . | cat
Sending build context to Docker daemon  27.14kB
Step 1/22 : FROM arm32v6/alpine:3.7
3.7: Pulling from arm32v6/alpine
786f62ef66ae: Pulling fs layer
4271bf13258b: Pulling fs layer
4271bf13258b: Download complete
786f62ef66ae: Verifying Checksum
786f62ef66ae: Download complete
786f62ef66ae: Pull complete
4271bf13258b: Pull complete
Digest: sha256:fc9150d7932fda71f248a9233da285d08df1ce1d9a327cabca01a953525f0283
Status: Downloaded newer image for arm32v6/alpine:3.7
 ---> 1499e8e0088e
Step 2/22 : LABEL maintainer "https://github.com/blacktop"
 ---> Running in 7bf7dffd057c
Removing intermediate container 7bf7dffd057c
 ---> 6f583cc0f48a
Step 3/22 : RUN apk add --no-cache openjdk8-jre su-exec
 ---> Running in 73cb9f624495
standard_init_linux.go:190: exec user process caused "exec format error"
The command '/bin/sh -c apk add --no-cache openjdk8-jre su-exec' returned a non-zero code: 1
docker push wireserver/elasticsearch:0.0.7-arm32v6 | cat
The push refers to repository [docker.io/wireserver/elasticsearch]
tag does not exist: wireserver/elasticsearch:0.0.7-arm32v6
docker manifest create wireserver/elasticsearch:0.0.7 wireserver/elasticsearch:0.0.7-amd64 wireserver/elasticsearch:0.0.7-386 wireserver/elasticsearch:0.0.7-arm32v6 --amend
no such manifest: docker.io/wireserver/elasticsearch:0.0.7-amd64
Makefile:105: recipe for target 'manifest-create-elasticsearch' failed
make: *** [manifest-create-elasticsearch] Error 1

jschaul · 2019-02-14T17:15:02Z

Without parallelism:

docker push wireserver/airdock_fakesqs:0.0.7-amd64 | cat
The push refers to repository [docker.io/wireserver/airdock_fakesqs]
90cae6f23657: Preparing
703848784bfd: Preparing
4cafd325540e: Preparing
3b5803f7366f: Preparing
acd812c5a274: Preparing
0b17f64af87b: Preparing
acd812c5a274: Mounted from wireserver/airdock_rvm
3b5803f7366f: Mounted from wireserver/airdock_rvm
4cafd325540e: Mounted from wireserver/airdock_rvm
0b17f64af87b: Mounted from wireserver/airdock_rvm
703848784bfd: Pushed
90cae6f23657: Pushed
0.0.7-amd64: digest: sha256:583778f3c9a956a7a7ba9d33751c2a3d9c7006e506dc2e8157832afc6a6f6099 size: 1580
docker manifest create wireserver/airdock_fakesqs:0.0.7 wireserver/airdock_fakesqs:0.0.7-arm32v5 wireserver/airdock_fakesqs:0.0.7-arm32v7 wireserver/airdock_fakesqs:0.0.7-386 wireserver/airdock_fakesqs:0.0.7-amd64 --amend
no such manifest: docker.io/wireserver/airdock_fakesqs:0.0.7-arm32v5
Makefile:105: recipe for target 'manifest-create-airdock_fakesqs' failed
make: *** [manifest-create-airdock_fakesqs] Error 1

There seems to still be an ordering issue here somewhere.

julialongtin · 2019-02-15T14:49:05Z

I've managed to reproduce the failure you saw. not a critical one, but i'm looking into it.

ChrisPenner · 2019-02-18T09:42:03Z

@julialongtin Do you have a good Makefile guide you can suggest? I understand the basics, but I'd love a resource where I can skim through advanced features :) A lot of these symbols and forms are tough to google.

…es, along with race condition fixes.

Co-Authored-By: julialongtin <julia.longtin@gmail.com>

…to JEL-DOCKER-1

…ram when running integration tests.

deploy/docker-ephemeral/build/Makefile

ChrisPenner · 2019-02-28T09:22:27Z

This is some pretty complex stuff; props to you for getting it working, but I don't think I understand enough to give it a real review; maybe this is a good candidate for a show & tell?

ChrisPenner

The docs look great! I'll give the actual Makefile another look through on Monday 😄

…a, instead of the architecture specific image name.

fisx

I left some TODOs in the docs you made.
Could you follow the naming conventions for the mardown files (all small letters, dashes, not underscores)?
I didn't finish reading the docs, but I skimmed through everything and established that it's more than good enough to make the Makefile maintainable whenever we need to get back to that.
I have run the integration tests with your new docker-ephemeral deps, and failed because elastic search is not coming up.
I am running make build-all -j now. Let's see if that helps.

deploy/docker-ephemeral/build/Makefile

fisx · 2019-03-18T11:27:53Z

building does not work either:

~/src/wire-server/deploy/docker-ephemeral/build$ make build-all -j
[...]
Step 1/13 : FROM wireserver/airdock_rvm:0.0.9-386
manifest for wireserver/airdock_rvm:0.0.9-386 not found

ChrisPenner · 2019-03-19T13:29:06Z

@julialongtin In the future I'd prefer if you could address outstanding comments and feedback before merging; there are a few comments on this PR which haven't been resolved or discussed yet. 😸

fisx · 2019-03-19T19:20:51Z

oops, that may have been my fault. i said "merge", and neither of us checked if everything was done. @julialongtin?

julialongtin · 2019-03-20T07:15:42Z

I'll take a look when i hit the office. ;)

…

On Tue, Mar 19, 2019 at 8:20 PM fisx ***@***.***> wrote: oops, that may have been my fault. i said "merge", and neither of us checked if everything was done. @julialongtin <https://github.com/julialongtin>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#622 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AQDAApfqCk9m9rl-0mQnZGDov1pX7yOcks5vYTiWgaJpZM4a5-da> .

julialongtin added 5 commits February 7, 2019 15:00

build our docker dependencies for multiple archetectures.

66fbe1e

blacklist builds that do not work on some emulators, fix all of the g…

8f45eff

…it checkouts to specified commits, add minio, bump the tini version, and fix fakesqs and localstack.

break documentation out of the Makefile.

712ba49

add another note.

941b256

run through the process on another laptop, fleshing out the README. a…

827d559

…lso, amd64 support.

jschaul reviewed Feb 13, 2019

View reviewed changes

deploy/docker-ephemeral/build/Makefile Outdated Show resolved Hide resolved

deploy/docker-ephemeral/build/README.md Outdated Show resolved Hide resolved

deploy/docker-ephemeral/build/README.md Outdated Show resolved Hide resolved

julialongtin and others added 6 commits February 14, 2019 09:45

missing dollar sign.

f557320

take feedback into account, and add some sanity checks.

0fa1a17

fix missing comma, that was making variants not be used for arm.

e6fe98f

wire name

70cbed4

take into account creating the dockerhub repositories.

a864665

Merge branch 'JEL-DOCKER-1' of github.com:julialongtin/wire-server in…

6c3af68

…to JEL-DOCKER-1

jschaul reviewed Feb 14, 2019

View reviewed changes

julialongtin and others added 8 commits February 27, 2019 16:40

add cassandra, and provide ways to decrease memory usage by java imag…

8b8c495

…es, along with race condition fixes.

typo fix.

0f8ed2b

Update deploy/docker-ephemeral/build/Makefile

d8a11db

Co-Authored-By: julialongtin <julia.longtin@gmail.com>

Merge branch 'JEL-DOCKER-1' of github.com:julialongtin/wire-server in…

8012796

…to JEL-DOCKER-1

seperate out two ways of using this.

194b1a6

Merge remote-tracking branch 'origin/master' into JEL-DOCKER-1

072b370

add a note about docker image upload being optional.

b4e1cc1

use new cassandra and elasticsearch images, to save multiple gigs of …

c303409

…ram when running integration tests.

ChrisPenner reviewed Feb 28, 2019

View reviewed changes

deploy/docker-ephemeral/build/Makefile Outdated Show resolved Hide resolved

switch to https access instead of ssh access.

27a8480

ChrisPenner reviewed Feb 28, 2019

View reviewed changes

deploy/docker-ephemeral/build/Makefile Outdated Show resolved Hide resolved

make the list of arches overridable.

8c1bb6d

ChrisPenner reviewed Mar 7, 2019

View reviewed changes

julialongtin and others added 12 commits March 7, 2019 17:08

even more boxes.

e2575c8

add an extra empty line at the end of file.

9deba5d

remove extra line.

4ca25b5

simplify make rules, and spacing changes.

dbf06f5

almost done...

9206132

only two rules left...

de2287f

Done.

76bb78a

merge changes from fisx

acc1963

make instructions a bit more clear, and use the manifest for cassandr…

6b74596

…a, instead of the architecture specific image name.

switch to hand built images for most images.

c2c57b6

rm trailing whitespace, normalize tabs.

05fd751

review

8fd97a1

fisx requested changes Mar 18, 2019

View reviewed changes

deploy/docker-ephemeral/build/Makefile Show resolved Hide resolved

julialongtin added 4 commits March 19, 2019 10:12

take review comments into account.

0ce3d53

take review comments into account.

bb04d5d

rename document to adhere to naming conventions for markdown files.

240fe2b

try an environment variable to leave elasticsearch in development mode.

c9b5ba6

fisx approved these changes Mar 19, 2019

View reviewed changes

julialongtin merged commit 4223427 into wireapp:develop Mar 19, 2019

julialongtin added a commit that referenced this pull request Mar 20, 2019

take some missed feedback into account from PR #622

7fc7093

julialongtin mentioned this pull request Mar 20, 2019

Jel docker 1 #668

Merged

julialongtin added a commit that referenced this pull request Mar 20, 2019

take some missed feedback into account from PR #622

788736c

julialongtin added a commit that referenced this pull request Mar 20, 2019

take some missed feedback into account from PR #622

cf927be

julialongtin added a commit that referenced this pull request Mar 20, 2019

take some missed feedback into account from PR #622 (#668)

5df850d

fisx mentioned this pull request Mar 25, 2019

Release/2019-03-25 #674

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docker image building for all of the docker images our integration tests require. #622

docker image building for all of the docker images our integration tests require. #622

julialongtin commented Feb 13, 2019

jschaul left a comment

jschaul left a comment

jschaul commented Feb 14, 2019

jschaul commented Feb 14, 2019

julialongtin commented Feb 15, 2019

ChrisPenner commented Feb 18, 2019

ChrisPenner commented Feb 28, 2019

ChrisPenner left a comment

fisx left a comment

fisx commented Mar 18, 2019

ChrisPenner commented Mar 19, 2019

fisx commented Mar 19, 2019

julialongtin commented Mar 20, 2019 via email

docker image building for all of the docker images our integration tests require. #622

docker image building for all of the docker images our integration tests require. #622

Conversation

julialongtin commented Feb 13, 2019

jschaul left a comment

Choose a reason for hiding this comment

jschaul left a comment

Choose a reason for hiding this comment

jschaul commented Feb 14, 2019

jschaul commented Feb 14, 2019

julialongtin commented Feb 15, 2019

ChrisPenner commented Feb 18, 2019

ChrisPenner commented Feb 28, 2019

ChrisPenner left a comment

Choose a reason for hiding this comment

fisx left a comment

Choose a reason for hiding this comment

fisx commented Mar 18, 2019

ChrisPenner commented Mar 19, 2019

fisx commented Mar 19, 2019

julialongtin commented Mar 20, 2019 via email