race builds don't work in our docker images

[Dieterbe]

(note: you can build race builds using make bin-race)

the race detector requires CGO, which means dynamic linking, which causes breakage when we try to run the binary in our docker image based on alpine.
the container will log:

2018/11/26 17:20:54 cassandra:9042 is up. maintained connection for 3 seconds!
/usr/bin/wait_for_endpoint.sh: exec: line 67: /usr/bin/metrictank: not found

from golang/go#12122:

The race detector is not written in pure Go, it requires some libc support.

Below are some attempts to resolve this

trying to enforce the go net resolver rather than cgo

diff --git a/scripts/build.sh b/scripts/build.sh
index 9c136edb..3b5f49fb 100755
--- a/scripts/build.sh
+++ b/scripts/build.sh
@@ -36,7 +36,7 @@ for bin in *; do
   then
     set -x
     # -race requires CGO
-    CGO_ENABLED=1 go build -race -ldflags "-X main.gitHash=$GITVERSION" -o $BUILDDIR/$bin || fail
+    CGO_ENABLED=1 go build -ldflags "-X main.gitHash=$GITVERSION" -race -tags netgo -o $BUILDDIR/$bin || fail
   else
     set -x
     go build -ldflags "-X main.gitHash=$GITVERSION" -o $BUILDDIR/$bin || fail
diff --git a/docker/docker-cluster/docker-compose.yml b/docker/docker-cluster/docker-compose.yml
index adc9ae64..53c7d438 100644
--- a/docker/docker-cluster/docker-compose.yml
+++ b/docker/docker-cluster/docker-compose.yml
@@ -23,6 +23,7 @@ services:
       MT_KAFKA_CLUSTER_PARTITIONS: 0,1,2,3
       MT_KAFKA_MDM_IN_PARTITIONS: 0,1,2,3
       MT_LOG_LEVEL: info
+      GODEBUG: netdns=go
     links:
      - cassandra

results in

/usr/bin/wait_for_endpoint.sh: exec: line 67: /usr/bin/metrictank: not found

note that this still links to libc, so that's why the startup fails.

$ ldd build/metrictank                                                                                                                                                ⏎
	linux-vdso.so.1 (0x00007ffff79b7000)
	libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f406fff2000)
	libc.so.6 => /usr/lib/libc.so.6 (0x00007f406fe2e000)
	/lib64/ld-linux-x86-64.so.2 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007f4070055000)

same, but with installing glibc in alpine

diff --git a/scripts/Dockerfile b/scripts/Dockerfile
index 96ac537d..162a901b 100644
--- a/scripts/Dockerfile
+++ b/scripts/Dockerfile
@@ -1,7 +1,9 @@
 FROM alpine
 MAINTAINER Dieter Plaetinck dieter@grafana.com
 
-RUN apk add -U tzdata
+RUN apk add -U tzdata libc6-compat
 
 RUN mkdir -p /etc/metrictank /usr/share/metrictank/examples
 COPY config/metrictank-docker.ini /etc/metrictank/metrictank.ini

2018/11/26 17:15:59 cassandra:9042 is up. maintained connection for 3 seconds!
Error relocating /usr/bin/metrictank: __libc_free: symbol not found
Error relocating /usr/bin/metrictank: __libc_malloc: symbol not found
Error relocating /usr/bin/metrictank: __libc_realloc: symbol not found
Error relocating /usr/bin/metrictank: __libc_stack_end: symbol not found

=> not compatible with the libc that go needs for the race detector.
this is a known issue. golang/go#14481

trying to switch to debian

diff --git a/scripts/Dockerfile b/scripts/Dockerfile
index 96ac537d..de9336bf 100644
--- a/scripts/Dockerfile
+++ b/scripts/Dockerfile
@@ -1,6 +1,8 @@
-FROM alpine
+FROM debian:8

 MAINTAINER Dieter Plaetinck dieter@grafana.com

-RUN apk add -U tzdata
+RUN apt-get update && apt-get install -y netcat && apt-get clean
 
 RUN mkdir -p /etc/metrictank /usr/share/metrictank/examples

diff --git a/scripts/k8s/Dockerfile b/scripts/k8s/Dockerfile
index 276b46e9..06aa8e9b 100644
--- a/scripts/k8s/Dockerfile
+++ b/scripts/k8s/Dockerfile
@@ -1,5 +1,6 @@
 FROM grafana/metrictank
-RUN apk add --no-cache curl jq ca-certificates python py-pip
+RUN echo N | tee /sys/module/overlay/parameters/metacopy
+RUN apt-get update && apt-get install -y curl jq ca-certificates python python-pip
 RUN pip install kazoo
 COPY entrypoint.sh /entrypoint.sh
 COPY getOffset.py /getOffset.py

this breaks make docker
the apt-get commands trigger:

dpkg: error: error creating new backup file '/var/lib/dpkg/status-old': Invalid cross-device link
E: Sub-process /usr/bin/dpkg returned an error code (2)

people recommend addressing this with a command that also fails.

Step 2/7 : RUN echo N | tee /sys/module/overlay/parameters/metacopy
 ---> Running in cc0bfc07d977
tee: /sys/module/overlay/parameters/metacopy: Read-only file system
N
The command '/bin/sh -c echo N | tee /sys/module/overlay/parameters/metacopy' returned a non-zero code: 1

perhaps it is supposed to be run on the host, but i'm not sure if i want to try that, seems this is getting a bit out of hand.

[Dieterbe]

switch to debian + echo N | sudo tee /sys/module/overlay/parameters/metacopy on host

this seems to work, for now. but not a clean solution at all.

diff --git a/scripts/Dockerfile b/scripts/Dockerfile
index 96ac537d..0a469d3c 100644
--- a/scripts/Dockerfile
+++ b/scripts/Dockerfile
@@ -1,7 +1,7 @@
-FROM alpine
+FROM debian:8
 MAINTAINER Dieter Plaetinck dieter@grafana.com
 
-RUN apk add -U tzdata
+RUN apt-get update && apt-get install -y netcat && apt-get clean && rm -rf /var/cache/apt/* /var/lib/apt/lists/* /tmp/* /var/tmp/*

 
 RUN mkdir -p /etc/metrictank /usr/share/metrictank/examples
 COPY config/metrictank-docker.ini /etc/metrictank/metrictank.ini
diff --git a/scripts/k8s/Dockerfile b/scripts/k8s/Dockerfile
index 276b46e9..c65e2a6e 100644
--- a/scripts/k8s/Dockerfile
+++ b/scripts/k8s/Dockerfile
@@ -1,5 +1,5 @@
 FROM grafana/metrictank
-RUN apk add --no-cache curl jq ca-certificates python py-pip
+RUN apt-get update && apt-get install -y curl jq ca-certificates python python-pip && apt-get clean && rm -rf /var/cache/apt/* /var/lib/apt/lists/* /tmp/* /var/tmp/*

 RUN pip install kazoo
 COPY entrypoint.sh /entrypoint.sh
 COPY getOffset.py /getOffset.py

[Dieterbe]

note, size comparison:

# docker image size:
# master
grafana/metrictank                         0.10.1-227-g6aa4eaa        3f45b4a44948        About a minute ago   228MB
us.gcr.io/metrictank-gcr/metrictank        0.10.1-227-g6aa4eaa        833d339bdd4f        54 seconds ago       280MB
# chunk-4h-sparse-bugfix-longterm + debian
grafana/metrictank                         0.10.1-255-g960810a        51ac2cce0517        About a minute ago   411MB
us.gcr.io/metrictank-gcr/metrictank        0.10.1-255-g960810a        f55c186a8c4d        20 seconds ago       642MB

we might be able to make the alpine images lighter via rm -rf /var/cache/apk/* (untested)

the building is also much slower.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.