doh_server fails to build

[willis936]

• DoTH-DNS version: 7.0.0
• Bash version: 5.0.3(1)-release (arm-unknown-linux-gnueabihf)
• Operating System: DietPi 7.0.2, Raspbian GNU/Linux 10 (buster), kernel 5.10.17-v7l+, bootloader: 1599135103, VL805: 00138a1

Description

Running dothdns fails to build doh_server.

I have also tried installing from source (version 7.0.2.dev1+ga5ce6eb) and observed the same behavior.

# dothdns run
SUCCESS:   Set environment variables in `.env` file.
WARNING:   No ['cert.crt', 'key.key'] files found. Dashboards, DoH and DoT need both a 'certificate' and a corresponding 'key'. If you have not set those files up on another way encryption will not work properly.
SUCCESS:   Set environment variables in `.env` file.
INFO:      Image for 'doh_server' not found.
INFO:      Compiling image for 'doh_server' for version 2.2.4. This may last a bit.
ERROR:     The build of 'doh_server' image raised an error: The command '/bin/sh -c set -x     && delgroup ping     && addgroup -g 8053 doh     && adduser -D -G doh -u 8053 doh     && apk add -q --no-cache --virtual .build-deps gcc git go musl-dev     && apk add -q --no-cache bash ca-certificates shadow su-exec tzdata     && go get github.com/m13253/dns-over-https/doh-server     && cp -r /go/bin/* /usr/local/bin     && apk del -q --purge .build-deps     && rm -rf /go /root/.cache/* /tmp/* /var/cache/apk/*' returned a non-zero code: 5
Aborted!

What I Did

Install DoTH-DNS via pip then run dothdns run.

[Cielquan]

Unfortunately the error msg does not give much info about the root cause, except that is has something to do with the RUN part in the Dockerfile for the DoH server.

Could you locate your DoTH-DNS installation and manipulate the Dockerfile, which should reside in <venv>/lib/pyhtonX.Y/site-packages/dothdns/container_configs/doh-docker/Dockerfile.

Please change the RUN entry

RUN \
    set -x \
    && delgroup ping \
    && addgroup -g 8053 doh \
    && adduser -D -G doh -u 8053 doh \
    && apk add -q --no-cache --virtual .build-deps gcc git go musl-dev \
    && apk add -q --no-cache bash ca-certificates shadow su-exec tzdata \
    && go get github.com/m13253/dns-over-https/doh-server \
    && cp -r /go/bin/* /usr/local/bin \
    && apk del -q --purge .build-deps \
    && rm -rf /go /root/.cache/* /tmp/* /var/cache/apk/*

with this:

RUN \
    set -x \
    && (delgroup ping || exit exit 11) \
    && (addgroup -g 8053 doh || exit exit 22) \
    && (adduser -D -G doh -u 8053 doh || exit exit 33) \
    && (apk add -q --no-cache --virtual .build-deps gcc git go musl-dev || exit exit 44) \
    && (apk add -q --no-cache bash ca-certificates shadow su-exec tzdata || exit exit 55) \
    && (go get github.com/m13253/dns-over-https/doh-server || exit exit 66) \
    && (cp -r /go/bin/* /usr/local/bin || exit 77) \
    && (apk del -q --purge .build-deps || exit 88) \
    && (rm -rf /go /root/.cache/* /tmp/* /var/cache/apk/* || exit 99)

Afterwards try running dothdns again. I added exit codes to each subcommand .. hopefully with the exit code you get then we can localize the erroring subcommand and see further to find a fix.

[willis936]

I’m new to containers, so pardon mistakes.

I couldn’t find a directory that had a site-packages directory in lib/python3.7, but I did find a Dockerfile in DoTH-DNS/doh-docker after running dothdns config.

I made the changes to the run command in that Dockerfile and ran dothdns run again.

The error code is now 2 instead of 5, but this doesn’t appear to be at any of your new exit codes.

# dothdns run
SUCCESS:   Set environment variables in `.env` file.
SUCCESS:   Set environment variables in `.env` file.
INFO:      Image for 'doh_server' not found.
INFO:      Compiling image for 'doh_server' for version 2.2.4. This may last a bit.
ERROR:     The build of 'doh_server' image raised an error: The command '/bin/sh -c set -x     && (delgroup ping || exit exit 11)     && (addgroup -g 8053 doh || exit exit 22)     && (adduser -D -G doh -u 8053 doh || exit exit 33)     && (apk add -q --no-cache --virtual .build-deps gcc git go musl-dev || exit exit 44)     && (apk add -q --no-cache bash ca-certificates shadow su-exec tzdata || exit exit 55)     && (go get github.com/m13253/dns-over-https/doh-server || exit exit 66)     && (cp -r /go/bin/* /usr/local/bin || exit 77)     && (apk del -q --purge .build-deps || exit 88)     && (rm -rf /go /root/.cache/* /tmp/* /var/cache/apk/* || exit 99)' returned a non-zero code: 2
Aborted!

Update: I also tried changing set -x to (set -x || exit exit 12) but still got exit code 2.

EDIT: If anyone reads this in the future and doesn't know how to find their site-packages directory: run this command
pip list -v

Mine ended up being /usr/local/lib/python3.7/dist-packages.

[Cielquan]

I also only worked a bit with docker, so I only know some basics. But for this container I never had any issues regardless of the system I used.

Could you again change the RUN part in the Dockerfile to this:

RUN set -x
RUN delgroup ping
RUN addgroup -g 8053 doh
RUN adduser -D -G doh -u 8053 doh
RUN apk add -q --no-cache --virtual .build-deps gcc git go musl-dev
RUN apk add -q --no-cache bash ca-certificates shadow su-exec tzdata
RUN go get github.com/m13253/dns-over-https/doh-server
RUN cp -r /go/bin/* /usr/local/bin
RUN apk del -q --purge .build-deps
RUN rm -rf /go /root/.cache/* /tmp/* /var/cache/apk/*

This split should hopefully let us identify the culprit command.

set -x should not fail. It is just a setting for bash to print the commands used afterwards. You can use this e.g. for debugging in bash scripts.

[willis936]

Progress!

It appears to fail on the apk add line. I'm not sure if apk needs to be installed in the host system, but running apk in the root shell returns -bash: apk: command not found. I'm on a debian-based distro that uses apt as the package manager.

# dothdns run
SUCCESS:   Set environment variables in `.env` file.
SUCCESS:   Set environment variables in `.env` file.
INFO:      Image for 'doh_server' not found.
INFO:      Compiling image for 'doh_server' for version 2.2.4. This may last a bit.
ERROR:     The build of 'doh_server' image raised an error: The command '/bin/sh -c apk add -q --no-cache --virtual .build-deps gcc git go musl-dev' returned a non-zero code: 5
Aborted!

[Cielquan]

Nice!

Containers only need the host's kernel (to my knowledge). So software is not transferred from the host into the container. Therefore the apk command is there to install the dependencies into the container image.

By convention exit code 127 is "command not found" so apk should be available. But cannot find something on exit code 5 for apk.

I'll try some things when I'm back home.

[Cielquan]

So .. I tried dothdns run on my linux mint 19.3 VM and it build the doh_server image without problem.

As it seems to be a problem with docker and not doth-dns lets skip doth-dns for now and work directly with docker. Could you cd into the directory where the Dockerfile is and run:
docker build .
Without doth-dns and the underlying docker lib for python the output will be much more verbose and hopefully give as more clues.

[willis936]

I think I read somewhere that someone who had a similar issue fixed it by changing their DNS resolver. My pi's upstream DNS server right now is another pi-hole. I haven't run into issues with filter lists filtering repos before, but perhaps that is happening here.

# docker --version
Docker version 20.10.5, build 55c4c88

# docker build .
Sending build context to Docker daemon  6.144kB
Step 1/18 : FROM alpine
 ---> 9c74a18b2325
Step 2/18 : ARG BUILD_DATE
 ---> Using cache
 ---> eead8807458d
Step 3/18 : ARG VERSION
 ---> Using cache
 ---> 2822bb03d93f
Step 4/18 : LABEL     org.label-schema.vendor="Cielquan - cielquan@protonmail.com"     org.label-schema.url="https://github.com/Cielquan/DoTH-DNS/"     org.label-schema.name="DoH Server"     org.label-schema.version=$VERSION     org.label-schema.build-date=$BUILD_DATE     org.label-schema.license="GPL-3.0"     org.label-schema.schema-version="1.0"
 ---> Running in 0c8cea81b888
Removing intermediate container 0c8cea81b888
 ---> 0e3dca947028
Step 5/18 : ENV     GOPATH="/go"     GOCACHE="/tmp/gocache"
 ---> Running in 61d9200d2c24
Removing intermediate container 61d9200d2c24
 ---> a8c0d5f926a3
Step 6/18 : WORKDIR /opt/dns-over-https
 ---> Running in ed655c8a475f
Removing intermediate container ed655c8a475f
 ---> 2df7a501e79b
Step 7/18 : COPY configs/ conf/
 ---> 4219715e0758
Step 8/18 : RUN set -x
 ---> Running in 25487a8381a0
Removing intermediate container 25487a8381a0
 ---> d190ce0042a4
Step 9/18 : RUN delgroup ping
 ---> Running in b856e5307aba
Removing intermediate container b856e5307aba
 ---> ba099c1ae032
Step 10/18 : RUN addgroup -g 8053 doh
 ---> Running in ede7b3d27915
Removing intermediate container ede7b3d27915
 ---> d3a436b7e5ad
Step 11/18 : RUN adduser -D -G doh -u 8053 doh
 ---> Running in 4a60898246a9
Removing intermediate container 4a60898246a9
 ---> 33278237b9dc
Step 12/18 : RUN apk add -q --no-cache --virtual .build-deps gcc git go musl-dev
 ---> Running in ca92a92f4b37
ERROR: unable to select packages:
  gcc (no such package):
    required by: .build-deps-19700101.000000[gcc]
  git (no such package):
    required by: .build-deps-19700101.000000[git]
  go (no such package):
    required by: .build-deps-19700101.000000[go]
  musl-dev (no such package):
    required by: .build-deps-19700101.000000[musl-dev]
The command '/bin/sh -c apk add -q --no-cache --virtual .build-deps gcc git go musl-dev' returned a non-zero code: 5

[Cielquan]

Yeah something like connectivity or filtering would have been my next guess according to the log above.

I'm curios if the pihole is the issue here.

[willis936]

I disabled DNS filtering the upstream pi-hole and observed the same results. I added RUN apk update just above the first apk add in case the repo list needed to be initialized and got this error:

Step 12/19 : RUN apk update
 ---> Running in 0a815477021a
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/main/armv7/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.13/main: temporary error (try again later)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/main: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/v3.13/community/armv7/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.13/community: temporary error (try again later)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.13/community: No such file or directory
2 errors; 14 distinct packages available
The command '/bin/sh -c apk update' returned a non-zero code: 2

I might try a fresh distro install tonight.

Update: I've also tried uncommenting DOCKER_OPTS="--dns 8.8.8.8 --dns 8.8.4.4" in \etc\default\docker, rebooting, and trying to build again, with the same results (both with and without the RUN apk update line).

[Cielquan]

I tried the URLs mentioned in the output and all work when I copy them into my browser. So for me the No such file or directory error` hints towards some connectivity problem of some sorts.

Could you confirm that the URLs work for you too in a browser or with curl?

[willis936]

I am able to curl from the machine that is failing to builder doh server, even with DNS filtering enabled on the upstream pi-hole. I will try a fresh docker install and if that fails then I will try a fresh distro.

# curl https://dl-cdn.alpinelinux.org/alpine/v3.13/main/armv7/APKINDEX.tar.gz --output APKINDEX.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  609k  100  609k    0     0  1507k      0 --:--:-- --:--:-- --:--:-- 1507k

Update: same issue with a full removal of docker and docker-compose (via dietpi-software), then reinstall of docker.

[Cielquan]

I found this issue: gliderlabs/docker-alpine#334

Maybe docker build . --network=host works. If it does doth-dns should find the pre-built image and proceed with it.

[willis936]

I've been playing around with build options (including setting the network to host during disabled dns filtering) but haven't observed any change in behavior. I'm not sure how to get docker network debug output yet.

Update: I did a fresh install of dietpi and am still observing the same behavior. I feel like this might be an issue for an upstream repo but I'm not sure which.

I'm going to try a good old-fashioned kernel compile then call it a day. I'm not sure what to try next.

Update: the kernel compile (commit: raspberrypi/linux@13c4388) didn't change the behavior. I have noticed that some of my scripts do not work the way I expect them to in a debian OS. I'm wondering if there is something up with dietpi's root shell. I will try a raspbian lite install tomorrow.

Update: I have tried removing docker via the dietpi-software interface and instead installing it via get-docker.sh. I also tried running as a non-root user that was added to the docker user group. Same behavior in both cases.

[willis936]

I ran dothdns run on another raspberry pi 4 I have with an existing full raspberry pi os image. I installed docker via the curl link and dothdns via the pip compile command. I did notice that I had to change the command from pip to pip3 to get the script to use python 3.7.3 instead of the also installed python 2.7.16.

When I ran dothdns run (or just docker build . in the doh server directory) I get the same error. I don’t think this is a dietpi specific issue.

[Cielquan]

I tested it on my RasPi4 also:

I used the official Raspberry Pi OS Lite (Release date: January 11th 2021 Kernel version: 5.4) on my RasPi 4B 4GB.
After boot I did via ssh:

# Upgrade system
$ sudo apt update && sudo apt upgrade -y
# Install docker and check version
$ sudo apt install docker.io -y
$ docker -v
Docker version 18.09.1, build 4c52b90
# Add user to docker group to run docker command without sudo
$ sudo usermod -aG docker pi
# Reboot to apply group change
$ sudo reboot
# Check if docker daemon runs
$ docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
# I needed to start docker manually as the daemon was not running after the reboot
$ sudo systemctl start docker
# Check python3 version
$ python3 --version
Python 3.7.3
# Install python3-venv (which not always comes with python by default)
$ sudo apt install python3-venv -y
# Create python virtual environment
$ python3 -m venv .venv
# Activate the venv
$ source .venv/bin/activate
# Install DoTH-DNS (I don't need pip3 here explicitly as I use a venv and the pip version
# there will be used which will be pip3 because I  used python3 for the venv creation above)
$ pip install doth-dns
# Run DoTH-DNS and get the same error
$ dothdns run
SUCCESS:   Set environment variables in `.env` file.
WARNING:   No ['cert.crt', 'key.key'] files found. Dashboards, DoH and DoT need both a 'certificate' and a corresponding 'key'. If you have not set those files up on another way encryption will not work properly.
SUCCESS:   Set environment variables in `.env` file.
INFO:      Image for 'doh_server' not found.
INFO:      Compiling image for 'doh_server' for version 2.2.4. This may last a bit.
ERROR:     The build of 'doh_server' image raised an error: The command '/bin/sh -c set -x     && delgroup ping     && addgroup -g 8053 doh     && adduser -D -G doh -u 8053 doh     && apk add -q --no-cache --virtual .build-deps gcc git go musl-dev     && apk add -q --no-cache bash ca-certificates shadow su-exec tzdata     && go get github.com/m13253/dns-over-https/doh-server     && cp -r /go/bin/* /usr/local/bin     && apk del -q --purge .build-deps     && rm -rf /go /root/.cache/* /tmp/* /var/cache/apk/*' returned a non-zero code: 5
Aborted!
# Change the Dockerfile
$ nano DoTH-DNS/doh-docker/Dockerfile
# Run again and see apk failing
$ dothdns run
SUCCESS:   Set environment variables in `.env` file.
WARNING:   No ['cert.crt', 'key.key'] files found. Dashboards, DoH and DoT need both a 'certificate' and a corresponding 'key'. If you have not set those files up on another way encryption will not work properly.
SUCCESS:   Set environment variables in `.env` file.
INFO:      Image for 'doh_server' not found.
INFO:      Compiling image for 'doh_server' for version 2.2.4. This may last a bit.
ERROR:     The build of 'doh_server' image raised an error: The command '/bin/sh -c apk add -q --no-cache --virtual .build-deps gcc git go musl-dev' returned a non-zero code: 5
Aborted!
# Run docker build for more output
$ docker build DoTH-DNS/doh-docker/
Sending build context to Docker daemon  6.656kB
Step 1/18 : FROM alpine
 ---> 9c74a18b2325
Step 2/18 : ARG BUILD_DATE
 ---> Using cache
 ---> f6771d7e8956
Step 3/18 : ARG VERSION
 ---> Using cache
 ---> 5b76f8dc59e1
Step 4/18 : LABEL     org.label-schema.vendor="Cielquan - cielquan@protonmail.com"     org.label-schema.url="https://github.com/Cielquan/DoTH-DNS/"     org.label-schema.name="DoH Server"     org.label-schema.version=$VERSION     org.label-schema.build-date=$BUILD_DATE     org.label-schema.license="GPL-3.0"     org.label-schema.schema-version="1.0"
 ---> Running in 77389d46ede4
Removing intermediate container 77389d46ede4
 ---> 83eafa454e96
Step 5/18 : ENV     GOPATH="/go"     GOCACHE="/tmp/gocache"
 ---> Running in 678b0ee25adf
Removing intermediate container 678b0ee25adf
 ---> 5eda845c0e92
Step 6/18 : WORKDIR /opt/dns-over-https
 ---> Running in 97917072f224
Removing intermediate container 97917072f224
 ---> aca188e759e2
Step 7/18 : COPY configs/ conf/
 ---> c522e52da0b3
Step 8/18 : RUN set -x
 ---> Running in 180ab5f6666a
Removing intermediate container 180ab5f6666a
 ---> 43b889674008
Step 9/18 : RUN delgroup ping
 ---> Running in ab4f471e2603
Removing intermediate container ab4f471e2603
 ---> 26bdfc26137a
Step 10/18 : RUN addgroup -g 8053 doh
 ---> Running in d555ba3a7bc3
Removing intermediate container d555ba3a7bc3
 ---> 47a0e16b5e70
Step 11/18 : RUN adduser -D -G doh -u 8053 doh
 ---> Running in db722f50ab09
Removing intermediate container db722f50ab09
 ---> a727712f4f75
Step 12/18 : RUN apk add -q --no-cache --virtual .build-deps gcc git go musl-dev
 ---> Running in a5a64e9c23cf
ERROR: unable to select packages:  gcc (no such package):

    required by: .build-deps-19700101.000000[gcc]
  git (no such package):
    required by: .build-deps-19700101.000000[git]
  go (no such package):
    required by: .build-deps-19700101.000000[go]
  musl-dev (no such package):
    required by: .build-deps-19700101.000000[musl-dev]
The command '/bin/sh -c apk add -q --no-cache --virtual .build-deps gcc git go musl-dev' returned a non-zero code: 5

I then tested on my Linux Mint 20.1 VM (last time was 19.3) and the Image built just fine.

The difference are the hardware and OS.

I thought about moving the Dockerfile to ubuntu, but the transition is not that trivial as it seems because of the dependencies. I originally copied the Dockfile from https://github.com/goofball222/dns-over-https/blob/master/stable/Dockerfile and modified it to DoTH-DNS needs. If I remember correctly it was mostly simplification by removing unnecessary parts and changing the entrypoint.

In a last attempt I ran the same procedure above again on the again fresh RasPi 4 but changed the base image from FROM alpine to FROM alpine:3.12 in the Dockerfile prior running dothdns run the first time and it worked. Without a tag it defaults to FROM alpine:latest which is currently FROM alpine:3.13. (https://hub.docker.com/_/alpine/)

So there must be a problem with alpine 3.13 + arm hardware (?) + apk I guess?!

I guess changing the alpine version a feature release back could be a temporary solution.

What you could also test to see if maybe a patch version and not the 3.13 feature version is the culprit:

• FROM alpine:3.13.0
• FROM alpine:3.13.1
• FROM alpine:3.13.2 (which is latest)

[willis936]

I ran docker build . in the doh server directory on my raspberry pi 4 dietpi.

FROM alpine version:
3.12: succeeded (!!)
3.13.0: failed
3.13.1: failed
3.13.2: failed

It looks like this issue:
dotnet/runtime#47423 (comment)

Thanks for the investigation.

[Cielquan]

I would like to move these additional issue to new ones, as the OP issue is solved.

EDIT: I marked the comments off-topic. @willis936 please see the last hidden comment for my answer.