circleci build failed

[kpepper]

I tried running "circleci build" in bioconda-recipes but I get a "cannot find git" error and the build fails. So I updated the bioconda-utils-build-env docker container before running:

docker pull bioconda/bioconda-utils-build-env

but that has made no difference. (have also deleted the container and re-pulled).

Same thing yesterday and today. Any ideas what the issue could be?
Is git included in the latest docker container?

Thanks.

Build log:

====>> Spin up Environment
Build-agent version 0.0.4666-05dad47 (2018-01-25T18:14:33+0000)
Starting container bioconda/bioconda-utils-build-env
using image bioconda/bioconda-utils-build-env@sha256:bb7f4dd3dfd337e097b9aa498604940901d7c58e740071cbbf415b59e312d97a

Using build environment variables:
BASH_ENV=/tmp/.bash_env-localbuild-1517572626
CI=true
CIRCLECI=true
CIRCLE_BRANCH=artemis
CIRCLE_BUILD_NUM=
CIRCLE_JOB=build
CIRCLE_NODE_INDEX=0
CIRCLE_NODE_TOTAL=1
CIRCLE_REPOSITORY_URL=https://github.com/kpepper/bioconda-recipes.git
CIRCLE_SHA1=4c64726b799453dd6f857b41cc54e47a2ff5c65d
CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1517572626
CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code
#!/bin/sh
mkdir -p /root/project && cp -r /tmp/_circleci_local_build_repo/. /root/project
====>> Setup ssh
#!/bin/bash -eo pipefail
mkdir -p ~/.ssh
ssh-keyscan -H github.com >> ~/.ssh/known_hosts

github.com SSH-2.0-libssh_0.7.0

====>> Download common definitions
#!/bin/bash -eo pipefail
curl -s https://raw.githubusercontent.com/bioconda/bioconda-common/master/common.sh > .circleci/common.sh
====>> Setup bioconda-utils
#!/bin/bash -eo pipefail
.circleci/setup.sh
.circleci/setup.sh: line 14: type: bioconda-utils: not found
Setting up bioconda-utils...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 33.4M 100 33.4M 0 0 27.6M 0 0:00:01 0:00:01 --:--:-- 30.8M
PREFIX=/tmp/workspace/miniconda
installing: python-3.6.1-2 ...
installing: asn1crypto-0.22.0-py36_0 ...
installing: cffi-1.10.0-py36_0 ...
installing: conda-env-2.6.0-0 ...
installing: cryptography-1.8.1-py36_0 ...
installing: idna-2.5-py36_0 ...
installing: libffi-3.2.1-1 ...
installing: openssl-1.0.2l-0 ...
installing: packaging-16.8-py36_0 ...
installing: pycosat-0.6.2-py36_0 ...
installing: pycparser-2.17-py36_0 ...
installing: pyopenssl-17.0.0-py36_0 ...
installing: pyparsing-2.1.4-py36_0 ...
installing: readline-6.2-2 ...
installing: requests-2.14.2-py36_0 ...
installing: ruamel_yaml-0.11.14-py36_1 ...
installing: setuptools-27.2.0-py36_0 ...
installing: six-1.10.0-py36_0 ...
installing: sqlite-3.13.0-0 ...
installing: tk-8.5.18-0 ...
installing: xz-5.2.2-1 ...
installing: yaml-0.1.6-0 ...
installing: zlib-1.2.8-3 ...
installing: conda-4.3.21-py36_0 ...
installing: pip-9.0.1-py36_1 ...
installing: wheel-0.29.0-py36_0 ...
Python 3.6.1 :: Continuum Analytics, Inc.
creating default environment...
installation finished.
Warning: 'defaults' already in 'channels' list, moving to the top
Warning: 'conda-forge' already in 'channels' list, moving to the top
Warning: 'bioconda' already in 'channels' list, moving to the top
Fetching package metadata .............
Solving package specifications: .

Package plan for installation in environment /tmp/workspace/miniconda:

The following NEW packages will be INSTALLED:

alabaster:                0.7.10-py36_1                 conda-forge
anaconda-client:          1.6.5-py_0                    conda-forge
argh:                     0.26.2-py36_1                 conda-forge
babel:                    2.5.1-py36_0                  conda-forge
beautifulsoup4:           4.6.0-py36_0                  conda-forge
blas:                     1.1-openblas                  conda-forge
certifi:                  2017.11.5-py36_0              conda-forge
chardet:                  3.0.4-py36_0                  conda-forge
clyent:                   1.2.2-py36_0                  conda-forge
colorlog:                 3.1.0-py36_0                  conda-forge
conda-build:              2.1.18-py36_0                 conda-forge
conda-verify:             2.0.0-py36_0                  conda-forge
decorator:                4.1.2-py36_0                  conda-forge
docutils:                 0.14-py36_0                   conda-forge
filelock:                 2.0.6-py36_0                  conda-forge
galaxy-lib:               17.9.10-py36_0                bioconda   
imagesize:                0.7.1-py36_0                  conda-forge
involucro:                1.1.2-0                       bioconda   
ipython_genutils:         0.2.0-py36_0                  conda-forge
jinja2:                   2.10-py36_0                   conda-forge
jsonschema:               2.6.0-py36_0                  conda-forge
jupyter_core:             4.4.0-py_0                    conda-forge
libgfortran:              3.0.0-1                       defaults   
markupsafe:               1.0-py36_0                    conda-forge
nbformat:                 4.4.0-py36_0                  conda-forge
networkx:                 1.11-py36_0                   conda-forge
numpy:                    1.14.0-py36_blas_openblas_200 conda-forge [blas_openblas]
openblas:                 0.2.20-7                      conda-forge
pandas:                   0.22.0-py36_0                 conda-forge
patchelf:                 0.9-2                         conda-forge
pkginfo:                  1.4.1-py36_0                  conda-forge
pyaml:                    17.12.1-py_0                  conda-forge
pycrypto:                 2.6.1-py36_1                  conda-forge
pydotplus:                2.0.2-py36_0                  conda-forge
pygithub:                 1.34-py36_0                   conda-forge
pygments:                 2.2.0-py36_0                  conda-forge
pyjwt:                    1.5.3-py_0                    conda-forge
pysocks:                  1.6.8-py36_1                  conda-forge
python-dateutil:          2.6.1-py36_0                  conda-forge
pytz:                     2017.3-py_2                   conda-forge
pyyaml:                   3.12-py36_1                   conda-forge
snowballstemmer:          1.2.1-py36_0                  conda-forge
sphinx:                   1.6.6-py36_0                  conda-forge
sphinx_rtd_theme:         0.2.4-py36_0                  conda-forge
sphinxcontrib-websupport: 1.0.1-py36_0                  conda-forge
traitlets:                4.3.2-py36_0                  conda-forge
typing:                   3.5.2.2-py36_0                bioconda   
urllib3:                  1.22-py36_0                   conda-forge

The following packages will be UPDATED:

conda:                    4.3.21-py36_0                 defaults    --> 4.3.33-py36_0 conda-forge
requests:                 2.14.2-py36_0                 defaults    --> 2.18.4-py36_1 conda-forge
six:                      1.10.0-py36_0                 defaults    --> 1.11.0-py36_1 conda-forge

involucro-1.1. 100% || Time: 0:00:01 2.06 MB/ss
libgfortran-3. 100% || Time: 0:00:00 13.11 MB/ss
patchelf-0.9-2 100% || Time: 0:00:00 3.90 MB/ss
openblas-0.2.2 100% || Time: 0:00:02 6.32 MB/ss
alabaster-0.7. 100% || Time: 0:00:00 89.49 kB/ss
argh-0.26.2-py 100% || Time: 0:00:00 208.97 kB/ss
beautifulsoup4 100% || Time: 0:00:00 543.42 kB/ss
blas-1.1-openb 100% || Time: 0:00:00 760.10 kB/ss
certifi-2017.1 100% || Time: 0:00:00 1.19 MB/ss
chardet-3.0.4- 100% || Time: 0:00:00 1.11 MB/ss
colorlog-3.1.0 100% || Time: 0:00:00 121.11 kB/ss
decorator-4.1. 100% || Time: 0:00:00 192.75 kB/ss
docutils-0.14- 100% || Time: 0:00:00 943.85 kB/ss
filelock-2.0.6 100% || Time: 0:00:00 3.55 MB/ss
imagesize-0.7. 100% || Time: 0:00:00 2.40 MB/ss
ipython_genuti 100% || Time: 0:00:00 12.91 MB/ss
jsonschema-2.6 100% || Time: 0:00:00 22.37 MB/ss
markupsafe-1.0 100% || Time: 0:00:00 16.01 MB/ss
pkginfo-1.4.1- 100% || Time: 0:00:00 13.53 MB/ss
pycrypto-2.6.1 100% || Time: 0:00:00 1.15 MB/ss
pysocks-1.6.8- 100% || Time: 0:00:00 7.36 MB/ss
pytz-2017.3-py 100% || Time: 0:00:00 1.33 MB/ss
pyyaml-3.12-py 100% || Time: 0:00:00 1.20 MB/ss
six-1.11.0-py3 100% || Time: 0:00:00 10.18 MB/ss
snowballstemme 100% || Time: 0:00:00 1.01 MB/ss
sphinx_rtd_the 100% || Time: 0:00:01 991.79 kB/ss
sphinxcontrib- 100% || Time: 0:00:00 18.79 MB/ss
typing-3.5.2.2 100% || Time: 0:00:00 15.09 MB/ss
babel-2.5.1-py 100% || Time: 0:00:04 1.02 MB/ss
clyent-1.2.2-p 100% || Time: 0:00:00 11.71 MB/ss
conda-verify-2 100% || Time: 0:00:00 7.53 MB/ss
galaxy-lib-17. 100% || Time: 0:00:00 1.72 MB/ss
jinja2-2.10-py 100% || Time: 0:00:00 1.14 MB/ss
networkx-1.11- 100% || Time: 0:00:00 1.83 MB/ss
numpy-1.14.0-p 100% || Time: 0:00:07 1.33 MB/ss
pyaml-17.12.1- 100% || Time: 0:00:00 107.80 kB/ss
pydotplus-2.0. 100% || Time: 0:00:00 249.50 kB/ss
pygments-2.2.0 100% || Time: 0:00:01 1.19 MB/ss
python-dateuti 100% || Time: 0:00:00 415.93 kB/ss
traitlets-4.3. 100% || Time: 0:00:00 824.08 kB/ss
jupyter_core-4 100% || Time: 0:00:00 8.48 MB/ss
pandas-0.22.0- 100% || Time: 0:00:14 1.84 MB/ss
nbformat-4.4.0 100% || Time: 0:00:00 868.02 kB/ss
pyjwt-1.5.3-py 100% || Time: 0:00:00 92.95 kB/ss
pygithub-1.34- 100% || Time: 0:00:01 1.17 MB/ss
urllib3-1.22-p 100% || Time: 0:00:00 384.65 kB/ss
requests-2.18. 100% || Time: 0:00:00 382.28 kB/ss
anaconda-clien 100% || Time: 0:00:00 735.46 kB/ss
conda-4.3.33-p 100% || Time: 0:00:00 1.06 MB/ss
sphinx-1.6.6-p 100% || Time: 0:00:01 1.17 MB/ss
conda-build-2. 100% || Time: 0:00:00 685.95 kB/ss
Collecting git+https://github.com/bioconda/bioconda-utils.git@v0.12.0
Cloning https://github.com/bioconda/bioconda-utils.git (to v0.12.0) to /tmp/pip-mtd470q7-build
Error [Errno 2] No such file or directory: 'git' while executing command git clone -q https://github.com/bioconda/bioconda-utils.git /tmp/pip-mtd470q7-build
Cannot find command 'git'
Error: Exited with code 1
Step failed
Task failed

[johanneskoester]

Should be fixed with PR #7509. Feel free to reopen if it still does not work.

[karl616]

I still have this problem. I could solve it by inserting a:
conda install git
in .circleci/setup.sh

Could the fact that I have to run circleci with sudo have an impact on this? I only have user-specific installation of conda... no installation for root... and I can see that the setup-script is referring to the /root folder. I'm still not versed enough with containers to pick out when it is /root on the host or in the container.

[johanneskoester]

Since 2 days, bioconda-utils depends on git, such that it should now be installed automatically. Can you do a docker pull bioconda/bioconda-utils-build-env before circleci build? Does that help, without the manual change to install git?

[karl616]

I tried to do the pull before commenting. Then it didn't work without the manual fix. Today it does. I cannot explain why, but I'm happy with the result. Thanks!

[mbhall88]

I am seeing this issue also.
I ran docker pull bioconda/bioconda-utils-build-env today.
Then circleci build and receive the following.

Docker image digest: sha256:a6decb2448f4371c64efde9dad90cc4efe0a72216ba7dc288fc89341d0b0c9d2
====>> Spin up Environment
Build-agent version 0.1.1809-71691ee9 (2019-02-08T15:06:57+0000)
Starting container bioconda/bioconda-utils-build-env
  using image bioconda/bioconda-utils-build-env@sha256:deafc3d76b2239cd1802d4012cbc6673850ed7e3900da5185d12aa51f5831734

Using build environment variables:
  BASH_ENV=/tmp/.bash_env-localbuild-1549639793
  CI=true
  CIRCLECI=true
  CIRCLE_BRANCH=mykrobe
  CIRCLE_BUILD_NUM=
  CIRCLE_JOB=build
  CIRCLE_NODE_INDEX=0
  CIRCLE_NODE_TOTAL=1
  CIRCLE_REPOSITORY_URL=https://github.com/mbhall88/bioconda-recipes.git
  CIRCLE_SHA1=4ed98aea65fa1375c71fef84134b22e0ca5156e1
  CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1549639793
  CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code
  #!/bin/bash -eo pipefail
mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files -z | xargs -0 tar -c | tar -x -C /root/project
/bin/bash: git: command not found
tar: Cowardly refusing to create an empty archive
Try `tar --help' or `tar --usage' for more information.
tar: This does not look like a tar archive
tar: Exiting with failure status due to previous errors
Error: Exited with code 2
Step failed
Error: runner failed
{"Runner":true,"level":"error","msg":"runner failed","task-id":"localbuild-1549639793","time":"2019-02-08T15:29:56Z"}
Task failed
Error: task failed

I am wondering whether this has something to do with commit bioconda/bioconda-utils@d446d42#diff-3254677a7917c6c01f55212f86c57fbf changing the base image from conda-forge?

[holtjma]

I'm having the exact same issue, @mbhall88. I'm new, so is there some work around and should we reopen this issue?

[Takadonet]

Also having same issue with the latest docker of bioconda/bioconda-utils-build-env where git is not found.

[edgano]

+1 with the same issue.
I'm using this container:using image bioconda/bioconda-utils-build-env@sha256:027b5c49311e9f08ac52ff3e688ea17b63a8b9bb3495a4b947a65e135a5520b2

  #!/bin/bash -eo pipefail
mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files -z | xargs -0 tar -c | tar -x -C /root/project
/bin/bash: git: command not found

[seb-mueller]

I've got the same error:

$ docker pull bioconda/bioconda-utils-build-env                                                                                                                                                       
Using default tag: latest
latest: Pulling from bioconda/bioconda-utils-build-env
Digest: sha256:907795f9f5a5635987b1ba0b93351d6ecba5e4eab70bddf4fe5b75fb99ff61da
Status: Image is up to date for bioconda/bioconda-utils-build-env:latest

Followed by:

circleci build              
...                                                                                                                                                                       
/bin/bash: git: command not found

Note sure how to best proceed from here? Or maybe this still not fixed?

[cpwardell]

I'm seeing the same error:

`$ circleci build
Docker image digest: sha256:60a98b5fc00a4302c46be2217d6e4e4eb8491d466fa62fed14362a85100bcd68
====>> Spin up Environment
Build-agent version 1.0.15283-35354d0e (2019-09-09T12:09:35+0000)
Docker Engine Version: 18.09.3
Kernel Version: Linux 6730d5672d20 3.10.0-514.21.1.el7.x86_64 #1 SMP Thu May 25 17:04:51 UTC 2017 x86_64 Linux
Starting container bioconda/bioconda-utils-build-env
using image bioconda/bioconda-utils-build-env@sha256:524360f41b2e90f9127f8313ff189b0c7fed7c5717f2e15429353a89f337f985

Using build environment variables
BASH_ENV=/tmp/.bash_env-localbuild-1568386778
CI=true
CIRCLECI=true
CIRCLE_BRANCH=fings
CIRCLE_BUILD_NUM=
CIRCLE_JOB=build
CIRCLE_NODE_INDEX=0
CIRCLE_NODE_TOTAL=1
CIRCLE_REPOSITORY_URL=https://github.com/cpwardell/bioconda-recipes.git
CIRCLE_SHA1=9a73ad6de226f2e19c9561883e3aab223903df34
CIRCLE_SHELL_ENV=/tmp/.bash_env-localbuild-1568386778
CIRCLE_WORKING_DIRECTORY=~/project

====>> Checkout code
#!/bin/bash -eo pipefail
mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files | tar -T - -c | tar -x -C /root/project && cp -a /tmp/_circleci_local_build_repo/.git /root/project
{"Runner":true,"level":"error","msg":"Error copying logs: read /dev/ptmx: input/output error","task-id":"localbuild-1568386778","time":"2019-09-13T14:59:41Z"}
/bin/bash: git: command not found
Error: Exited with code 127
Step failed
Error: runner failed (exited with 101)
Task failed
Error: task failed
`

[bgruening]

You need to have git installed.

[seb-mueller]

@bgruening , what do you mean by "you"? The local host system or the docker container?
As for my part git is installed (/usr/bin/git)
To have it installed in the container, it would be helpful to be a bit more specific on how to do this or how to point to the local install from within the container. As far as I know this should have been included in the docker image.

[nathanweeks]

I started to look into this within the last couple weeks, as I see the same problem on macOS; IIRC git is installed in the container image, but for some reason it is not in the PATH during execution.

[nathanweeks]

git is in /opt/conda/bin/ ; the following patch, which modifyies .circleci/setup.yml so that the conda environment is activated before invoking git allows it to be available in the PATH during circleci build, appears to fix circleci build on macOS:

diff --git a/.circleci/config.yml b/.circleci/config.yml
index 229ab2dc0..895592b41 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -78,14 +78,14 @@ jobs:
     docker:
       - image: bioconda/bioconda-utils-build-env
     steps:
-      - checkout
+      - run: echo ". /opt/conda/etc/profile.d/conda.sh" >> $BASH_ENV
+      - run: echo "conda activate" >> $BASH_ENV
+      - run: mkdir -p /root/project && cd /tmp/_circleci_local_build_repo && git ls-files | tar -T - -c | tar -x -C /root/project && cp -a /tmp/_circleci_local_build_repo/.git /root/project
       - run:
           name: Setup ssh
           command: |
             mkdir -p ~/.ssh
             ssh-keyscan -H github.com >> ~/.ssh/known_hosts
-      - run: echo ". /opt/conda/etc/profile.d/conda.sh" >> $BASH_ENV
-      - run: echo "conda activate" >> $BASH_ENV
       - *common
       - *setup
       - run:

The above patch also replaces the circleci checkout command to work around this issue:
https://discuss.circleci.com/t/local-checkout-fails-using-cli-tar-terminates-with-signal-13/28632
(an alternative workaround is presented here here: CircleCI-Public/circleci-cli#281). I'm not familiar-enough with CircleCI to know that workaround is future-proof, though.