Conda env create fails with FileNotFoundError

[mwalker174]

README.md instructs developers to create the conda environment with the command:

conda env create -n gatk -f scripts/gatkcondaenv.yml

This currently fails with the following message (at least on MacOS):

Requirement 'build/gatkPythonPackageArchive.zip' looks like a filename, but the file does not exist
Processing ./build/gatkPythonPackageArchive.zip
Exception:
Traceback (most recent call last):
  File "/Users/markw/anaconda/envs/gatk/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main
    status = self.run(options, args)
  File "/Users/markw/anaconda/envs/gatk/lib/python3.6/site-packages/pip/commands/install.py", line 335, in run
    wb.build(autobuilding=True)
  File "/Users/markw/anaconda/envs/gatk/lib/python3.6/site-packages/pip/wheel.py", line 749, in build
    self.requirement_set.prepare_files(self.finder)
  File "/Users/markw/anaconda/envs/gatk/lib/python3.6/site-packages/pip/req/req_set.py", line 380, in prepare_files
    ignore_dependencies=self.ignore_dependencies))
  File "/Users/markw/anaconda/envs/gatk/lib/python3.6/site-packages/pip/req/req_set.py", line 620, in _prepare_file
    session=self.session, hashes=hashes)
  File "/Users/markw/anaconda/envs/gatk/lib/python3.6/site-packages/pip/download.py", line 809, in unpack_url
    unpack_file_url(link, location, download_dir, hashes=hashes)
  File "/Users/markw/anaconda/envs/gatk/lib/python3.6/site-packages/pip/download.py", line 715, in unpack_file_url
    unpack_file(from_path, location, content_type, link)
  File "/Users/markw/anaconda/envs/gatk/lib/python3.6/site-packages/pip/utils/__init__.py", line 599, in unpack_file
    flatten=not filename.endswith('.whl')
  File "/Users/markw/anaconda/envs/gatk/lib/python3.6/site-packages/pip/utils/__init__.py", line 482, in unzip_file
    zipfp = open(filename, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/Users/markw/IdeaProjects/gatk/scripts/build/gatkPythonPackageArchive.zip'

Moving gatkcondaenv.yml to the GATK root solves the issue. We can either change the yml location or modify the readme.

[cmnbroad]

@markw The error message above looks like the one I'd expect you'd get if you haven't built the python archive (./gradlew createPythonPackageAchive), as discussed in the Slack thread. I assume you get a different error message once you've done that (which was in turn resolved by your moving the .yml file). If thats right, can you update this ticket with that message/output, and the conda version you're using.

[mwalker174]

@cmnbroad Apologies that I didn't specify that I did run ./gradlew createPythonPackageArchive first. You can see from the error that it's looking for the archive in gatk/scripts/build/ (which doesn't exist) instead of gatk/build. It seems to be searching for ./build/gatkPythonPackageArchive.zip relative to the scripts sub-directory rather than the root (also didn't mention I ran conda from the gatk root dir).

Running conda v4.5.2, python v3.6.4. I run everything from a conda env created with: conda create -n py36 python=3.6 anaconda

[cmnbroad]

Oh yeah, I see. I did upgrade to latests minconda and now I see the same issue. So something changed in conda.

[cmnbroad]

It does look like there are some references to similar issues(see comments in https://stackoverflow.com/questions/35245401/combining-conda-environment-yml-with-pip-requirements-txt) in some stack overflow threads, and some conda tickets that may be related.

Given this issue, and the unrelated but evolving requirement that we have more than a single conda yml (we need a separate one with the intel tensorflow version that supports acceleration on intel hardware), I'm going to make the following changes:

Add a single conda template and a gradle task that generates the two (nearly but not quite identical) yml files as part of the build. These will be stored in the build directory with the python package archive, so the yml file no longer has to specify a path to the zip (which we previously had to strip when we included the file in the zip/tar distribution). The zip and yml files will be copied into the tar/zip distributions as was previously done.
Add a gradle target that creates/updates the local conda dev. This is for local use by devs during iterative development. The yml files will still exist in the build directory, and can be used manually like they previously were, but from the build directory instead of in the root.