unable to do fresh install, conda problem

[parlar]

Version info

• bcbio version (bcbio_nextgen.py --version): latest stable (1.2.3)
• OS name and version: Ubuntu 18.04

To Reproduce

• Exact bcbio command you have used:
  python bcbio_nextgen_install.py /home/parlar/bcbio --tooldir=/home/user/bcbio/tools --nodata

Observed behavior

Traceback (most recent call last):
File "/home/parlar/bcbio/anaconda/bin/conda", line 12, in
from conda.cli import main
ModuleNotFoundError: No module named 'conda'
Traceback (most recent call last):
File "bcbio_nextgen_install.py", line 290, in
main(parser.parse_args(), sys.argv[1:])
File "bcbio_nextgen_install.py", line 47, in main
bcbio = install_conda_pkgs(anaconda, args)
File "bcbio_nextgen_install.py", line 108, in install_conda_pkgs
"--file", os.path.basename(REMOTES["requirements"]), TARGETPY], env=env)
File "/usr/lib/python2.7/subprocess.py", line 190, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/home/parlar/bcbio/anaconda/bin/conda', 'install', '--yes', '--file', 'requirements-conda.txt', 'python=3.6']' returned non-zero exit status 1

Expected behavior
Normal, clean installation

Log files
NA

Additional context
Removed potentially conflicting conda installations, tried 'development' instead of 'stable' with the same result.

[parlar]

In the installer, the following line in the install_conda_pkgs function seems to break downstream use of conda and mamba:

    subprocess.check_call([mamba_bin, "install", "--yes", "--only-deps", "bcbio-nextgen",
                           TARGETPY], env=env)

[naumenko-sa]

Hi Pär @parlar!

Thank for reporting - sorry about the issue!
I confirm I see it as well in my AWS Ubuntu install.

I tried to revert changes we recently did last month while cleaning the installation scripts,
but the error is still there, and we also had successful installations when testing the changes and last week.

So it is likely that the issue is not with the installation script, maybe it is rather related to conda/conda#9672 and similar conda issues?

Can you please elaborate more, why do you think https://github.com/bcbio/bcbio-nextgen/blob/master/scripts/bcbio_nextgen_install.py#L105
breaks mamba and conda downstream?

@hackdna also recently reviewed the installation process in detail.

Sergey

[naumenko-sa]

upd:
current installations works as follows:

1. python set to 3.7 in the TARGETPY: https://github.com/bcbio/bcbio-nextgen/blob/master/scripts/bcbio_nextgen_install.py#L35
   otherwise conda installation breaks with the error reported above
2. when conda itself is installed python gets downgraded to python3.6:
   https://github.com/chapmanb/cloudbiolinux/blob/030234475142c9517652774779d17e289929ce88/cloudbio/package/conda.py#L16
   because some bioconda tools are not compatible with python3.7 (as of 2019/04)

in the future we may test python3.7 against tools and use python3.7. However, it needs testing - we don't want tools to break randomly in the pipelines.

Pär @parlar,
please download the updated install script, try to install again, and confirm whether it works for you.

S

[parlar]

Thanks for working on this. Will try!

[gabeng]

This issue still exists and affects for example the current bcbio docker image 1.2.4-9722272. There seems to be a back and forth from python 3.7 to python 3.6 during the installation.
The previous docker image is not affected.

bcbio_nextgen.py --version
Traceback (most recent call last):
  File "/usr/local/bin/bcbio_nextgen.py", line 34, in <module>
    from bcbio.setpath import prepend_bcbiopath
ModuleNotFoundError: No module named 'bcbio' 

Is there any way to fix an installation?
It would be good to have some kind of a smoke test for the docker image maybe even during the build job to ensure that bcbio is actually installed correctly.

--
Ben