Unable to build docker image from source due to multiple reasons

[sushruta]

I am trying to build the docker image of deepvariant but running into problems. I'm on r1.7 branch and my branch is up to date. I am also facing the same issues on r1.6.1 branch as well. The command I am using is the following -

docker build --build-arg=FROM_IMAGE=nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 \
             --build-arg=DV_GPU_BUILD=1 \
             -t deepvariant_gpu .

I am building this docker image on my laptop (M3 macbook).

PackagesNotFoundError error

The first error I get is -

1.247 Platform: linux-aarch64
1.247 Collecting package metadata (repodata.json): ...working... done
6.190 Solving environment: ...working... failed
6.260
6.260 PackagesNotFoundError: The following packages are not available from current channels:
6.260
6.260   - bioconda::samtools==1.15
6.260   - bioconda::bcftools==1.15
6.260

I resolved this error by removing the version numbers. i.e., removed the ==1.15 from both the lines

Error in the build-prerunreq.sh script

Once, I cross the previous error, I get this error -

 > [builder 6/6] RUN ./build-prereq.sh   && PATH="${HOME}/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ./build_release_binaries.sh  # PATH for bazel:
0.101 ========== This script is only maintained for Ubuntu 22.04.
0.101 ========== Load config settings.
0.103 ========== [Thu Oct 31 21:28:00 UTC 2024] Stage 'Install the runtime packages' starting
0.104 ========== This script is only maintained for Ubuntu 22.04.
0.104 ========== Load config settings.
0.105 ========== [Thu Oct 31 21:28:00 UTC 2024] Stage 'Misc setup' starting
1.955 W: GPG error: https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa  InRelease: At least one invalid signature was encountered.
1.955 E: The repository 'https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/sbsa  InRelease' is not signed.
1.955 W: GPG error: http://ports.ubuntu.com/ubuntu-ports jammy InRelease: At least one invalid signature was encountered.
1.955 E: The repository 'http://ports.ubuntu.com/ubuntu-ports jammy InRelease' is not signed.
1.955 W: GPG error: http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease: At least one invalid signature was encountered.
1.955 E: The repository 'http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease' is not signed.
1.955 W: GPG error: http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease: At least one invalid signature was encountered.
1.955 E: The repository 'http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease' is not signed.
1.955 W: GPG error: http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease: At least one invalid signature was encountered.
1.955 E: The repository 'http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease' is not signed.
------
Dockerfile:50
--------------------
  49 |
  50 | >>> RUN ./build-prereq.sh \
  51 | >>>   && PATH="${HOME}/bin:${PATH}" ./build_release_binaries.sh  # PATH for bazel
  52 |
--------------------
ERROR: failed to solve: process "/bin/sh -c ./build-prereq.sh   && PATH=\"${HOME}/bin:${PATH}\" ./build_release_binaries.sh  # PATH for bazel" did not complete successfully: exit code: 100

Looks like the repositories are either old or the sign has expired.

How to fix this error?

[kishwarshafin]

@sushruta the error mainly is here:

ERROR: failed to solve: process "/bin/sh -c ./build-prereq.sh   && PATH=\"${HOME}/bin:${PATH}\" ./build_release_binaries.sh  # PATH for bazel" did not complete successfully: exit code: 100

Did you clone the entire repository correctly? Also, what OS you are trying to build this on?

[sushruta]

As noted in my previous command, I did a docker build --

docker build --build-arg=FROM_IMAGE=nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04 \
             --build-arg=DV_GPU_BUILD=1 \
             -t deepvariant_gpu .

From the docker image it is clear that it was ubuntu 22.04. This is exactly the image that is listed in your Dockerfile here -> https://github.com/google/deepvariant/blob/r1.7/Dockerfile#L11

---

Separately, I have also tried ubuntu 20.04 for r1.6 and that didn't work either

[kishwarshafin]

@sushruta , your log shows Platform: linux-aarch64, we build our docker on n2-standard-64 with x64 configuration. My question was what platform you are operating on?

[pichuan]

@sushruta mentioned that "I am building this docker image on my laptop (M3 macbook).", which is ARM architecture.

@sushruta , DeepVariant wasn't tested (or built) on Mac.

@kishwarshafin already pointed this out. I'll provide one more pointer:
We have example environment described here https://github.com/google/deepvariant/blob/r1.6.1/docs/deepvariant-details.md#commands-for-requesting-machines-used-in-case-studies
You should be able to build the Docker image on this type of machine too.

[sushruta]

I went to a linux machine and tried the above command. And while building the dockerfile with this base image -> nvidia/cuda:11.8.0-cudnn8-devel-ubuntu22.04, it errors out in the build_release_binaries.sh in line 48 of the dockerfile. This is the error -

(19:24:55) ERROR: /root/deepvariant/third_party/nucleus/io/python/BUILD:332:11: CLIF wrapping third_party/nucleus/io/python/hts_verbose.clif failed: (Exit 3): pyclif failed: error executing command
  (cd /root/.cache/bazel/_bazel_root/8422bf851bfac3671a35809acde131a7/execroot/com_google_deepvariant && \
  exec env - \
  bazel-out/k8-opt-exec-50AE0418/bin/external/clif/pyclif --modname third_party.nucleus.io.python.hts_verbose -c bazel-out/k8-opt/bin/third_party/nucleus/io/python/hts_verbose.cc -g bazel-out/k8-opt/bin/third_party/nucleus/io/python/hts_verbose.h -i bazel-out/k8-opt/bin/third_party/nucleus/io/python/hts_verbose_init.cc --prepend clif/python/types.h -I. -Ibazel-out/k8-opt/bin -Iexternal/htslib -Ibazel-out/k8-opt/bin/external/htslib -Iexternal/clif -Ibazel-out/k8-opt/bin/external/clif -Iexternal/com_google_glog -Ibazel-out/k8-opt/bin/external/com_google_glog -Iexternal/com_github_gflags_gflags -Ibazel-out/k8-opt/bin/external/com_github_gflags_gflags -Iexternal/com_google_protobuf -Ibazel-out/k8-opt/bin/external/com_google_protobuf -Iexternal/zlib -Ibazel-out/k8-opt/bin/external/zlib -Iexternal/local_config_python -Ibazel-out/k8-opt/bin/external/local_config_python -Iexternal/htslib/htslib/htslib_1_18 -Ibazel-out/k8-opt/bin/external/htslib/htslib/htslib_1_18 -Iexternal/com_google_protobuf/src -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src -Iexternal/local_config_python/python_include -Ibazel-out/k8-opt/bin/external/local_config_python/python_include -Iexternal/local_config_cc '-f-I. -Ibazel-out/k8-opt/bin -Iexternal/htslib -Ibazel-out/k8-opt/bin/external/htslib -Iexternal/clif -Ibazel-out/k8-opt/bin/external/clif -Iexternal/com_google_glog -Ibazel-out/k8-opt/bin/external/com_google_glog -Iexternal/com_github_gflags_gflags -Ibazel-out/k8-opt/bin/external/com_github_gflags_gflags -Iexternal/com_google_protobuf -Ibazel-out/k8-opt/bin/external/com_google_protobuf -Iexternal/zlib -Ibazel-out/k8-opt/bin/external/zlib -Iexternal/local_config_python -Ibazel-out/k8-opt/bin/external/local_config_python -Iexternal/htslib/htslib/htslib_1_18 -Ibazel-out/k8-opt/bin/external/htslib/htslib/htslib_1_18 -Iexternal/com_google_protobuf/src -Ibazel-out/k8-opt/bin/external/com_google_protobuf/src -Iexternal/local_config_python/python_include -Ibazel-out/k8-opt/bin/external/local_config_python/python_include -Iexternal/local_config_cc -std=c++17' third_party/nucleus/io/python/hts_verbose.clif)
# Configuration: e1a7a4d137d50a384683c717ca1c1ad9a118c21394cdd2b8374680c50a911b8a
# Execution platform: @local_execution_config_platform//:platform

Line .123456789.123456789.123456789.123456789
   1:# Copyright 2018 Google LLC.\n
   2:#\n
   3:# Redistribution and use in source and binary forms, with or without\n
   4:# modification, are permitted provided that the following conditions\n
   5:# are met:\n
   6:#\n
   7:# 1. Redistributions of source code must retain the above copyright notice,\n
   8:#    this list of conditions and the following disclaimer.\n
   9:#\n
  10:# 2. Redistributions in binary form must reproduce the above copyright\n
  11:#    notice, this list of conditions and the following disclaimer in the\n
  12:#    documentation and/or other materials provided with the distribution.\n
  13:#\n
  14:# 3. Neither the name of the copyright holder nor the names of its\n
  15:#    contributors may be used to endorse or promote products derived from this\n
  16:#    software without specific prior written permission.\n
  17:#\n
  18:# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"\n
  19:# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n
  20:# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\n
  21:# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\n
  22:# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\n
  23:# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\n
  24:# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\n
  25:# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\n
  26:# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\n
  27:# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\n
  28:# POSSIBILITY OF SUCH DAMAGE.\n
  29:\n
  30:from "third_party/nucleus/io/hts_verbose.h":\n
  31:  enum htsLogLevel\n
  32:  namespace `nucleus`:\n
  33:    def `HtsGetLogLevel` as get() -> htsLogLevel\n
  34:    def `HtsSetLogLevel` as set(level: htsLogLevel)\n
_ParseError: include "clif/python/types.h" not found

[akolesnikov]

Clif is installed as part of docker build. Could you check if there are other errors prior to this one related to the Clif installation?