Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update to Omnia Linux anvil #761

Merged
merged 37 commits into from
May 15, 2017
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
831fd41
Update docker file
Lnaden Apr 17, 2017
38c83df
Fix openmm
Lnaden Apr 17, 2017
9e3a83e
Change install method for autodoc to `python ...` instead of `pip ...`
Lnaden Apr 17, 2017
3956f2a
Change Travis to use new docker image
Lnaden Apr 17, 2017
7d80095
Update README
Lnaden Apr 17, 2017
81ff08d
Removed the old centos5-build-box folder which has been hanging aroun…
Lnaden Apr 17, 2017
1141426
Make sure `conda-forge` is added
Lnaden Apr 17, 2017
ac1509b
Disable openmm-example-plugin for the purpose of testing this PR
Lnaden Apr 17, 2017
a13342a
Try to fix tar problem on el_capitan
Lnaden Apr 17, 2017
d00429b
Try bumping to Xcode8.1 (OSX 10.12) to fix the mac `sudo` bug on `tar`
Lnaden Apr 17, 2017
da91ebc
Downgrade back to xcode6.4
Lnaden Apr 17, 2017
3301a06
Reverse conda command order on windows to ensure we get version requi…
Lnaden Apr 17, 2017
44df8ce
Disable coverage for builds to make windows builds go
Lnaden Apr 17, 2017
8972bbe
Added flags for remove, review, and a few notes
Lnaden Apr 17, 2017
90df632
Stop building ecos since its on the default channel.
Lnaden Apr 17, 2017
522650c
Remove some windows builds
Lnaden Apr 18, 2017
fb6a60f
Try to fix scs on windows
Lnaden Apr 18, 2017
2bf4719
Fix some packages which rely on cvxopt that cant build due to mingwpy…
Lnaden Apr 18, 2017
3af0ab8
Update readme
Lnaden Apr 18, 2017
4ea36e8
More updates to README
Lnaden Apr 20, 2017
1f9242f
Paranoia safety check. DO NOT MERGE
Lnaden Apr 20, 2017
e87a6b6
Revert the safety check
Lnaden Apr 20, 2017
bac81af
Update the README again for another pass
Lnaden Apr 20, 2017
6faabf9
Update readme again
Lnaden Apr 21, 2017
2dcba3a
OpenMM does not need clang on windows
Lnaden Apr 21, 2017
f3cc6b0
OpenMM skipped on windows
Lnaden Apr 21, 2017
bdb2f26
Actually add conda-forge to windows, remove salilab channel from windows
Lnaden Apr 21, 2017
43965be
Doing some debugging of openmm on mac
Lnaden Apr 21, 2017
9381644
Make sure AppleClang is not overwritten (test)
Lnaden Apr 21, 2017
c8a6c72
Change osx conda operations to ensure the conda binary updates come f…
Lnaden Apr 21, 2017
99f0e53
Try and mimic conda-forge
Lnaden Apr 21, 2017
a4e05d1
make sure conda and conda-env update right
Lnaden Apr 21, 2017
a2bfaff
Try pinning the conda-build to 1 version previous
Lnaden Apr 21, 2017
d91aab9
Try doing the conda updates first then adding the extra channels after
Lnaden Apr 21, 2017
9835c81
Set the MACOSX_DEPLOYMENT_TARGET in OpenMM
Lnaden Apr 24, 2017
5e51a3c
Roll back debug line
Lnaden Apr 24, 2017
0b181fe
disable openmm example plugin on py36 for now
Lnaden Apr 25, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ install:
# The Dockerfile that defines the image that for the build environment is
# available in this repo at devtools/omnia-build-box/Dockerfile
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then
docker pull jchodera/omnia-build-box:cuda${CUDA_SHORT_VERSION}-amd30-clang38;
docker pull jchodera/omnia-linux-anvil:texlive-amd30-cuda${CUDA_SHORT_VERSION};
fi

script:
Expand All @@ -37,7 +37,7 @@ script:
- if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then

docker run -e UPLOAD -e BINSTAR_TOKEN -e TRAVIS_PULL_REQUEST
-t -i --rm -v `pwd`:/io jchodera/omnia-build-box:cuda${CUDA_SHORT_VERSION}-amd30-clang38
-t -i --rm -v `pwd`:/io jchodera/omnia-linux-anvil:texlive-amd30-cuda${CUDA_SHORT_VERSION}
bash /io/devtools/docker-build.sh;

elif [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
Expand Down
95 changes: 84 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,92 @@
* Appveyor-CI `win` builds [![AppVeyor Build status](https://ci.appveyor.com/api/projects/status/fyjgl66t943tf2yg/branch/master?svg=true)](https://ci.appveyor.com/project/jchodera/conda-recipes/branch/master)


omnia-md/conda-recipes
----------------------
# omnia-md/conda-recipes

The recipes here create conda packages for scientific and numerical software components associated with the [`omnia`](http://omnia.md) project.
The packages built from these recipes are shared with the community on [anaconda.org](https://anaconda.org/omnia).
The packages built from these recipes are shared with the community on [anaconda.org](https://anaconda.org/omnia).
These packages also depend on the `conda-forge` conda channel.

### Installing packages from omnia
## Installing packages from omnia

To install a package
To install a package (`mdtraj` for example)
```bash
# Set the channel priority behavior
conda config --set channel_priority false
# Add conda-forge as the lower priority (FILO format for priority)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this is true anymore, since we're using channel_priority false. Maybe combine the two channel add commands with a single

conda config --add channels conda-forge omnia

?

conda config --add channels conda-forge
# Add the omnia channel
$ conda config --add channels omnia
conda config --add channels omnia
# Install the 'mdtraj' a package
$ conda install mdtraj
conda install mdtraj
```

The `channel_priority` behavior we configure gives priority to the highest version of the package over all channels.

The default behavior (`channel_priority true`) pulls packages from the highest priority chanel, independent of package
version numbers. i.e. an older version package on a higher priority channel will be installed over a newer version
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove "i.e." and capitalize the sentence.

of the same package on a lower priority channel.

### Important!
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change "Important!" to "Important: Channel priority behavior must be disabled for omnia to work"


These packages must use the old style of conda channel priority until omnia is entierly on conda-forge!
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

entirely

The instructions below enforce the old channel priority format, please visit
[the conda documentations on channel priority](https://conda.io/docs/channels.html)
for more information.

## Migration to conda-forge
The Omnia project has started migrating to [`conda-forge`](https://conda-forge.github.io/). New packages that
do not depend on OpenMM should be developed on `conda-forge` and exiting packages which do not depend on OpenMM
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"exiting" -> "existing"

should start migrating if possible.

### Planed Migration Stages

1. (**Current**) Update build image to CentOS 6 from CentOS 5

The base Docker image for linux builds will be updated to CentOS 6 with its new glibc. The base image is the
`conda-forge` anvil, with some [custom addons](https://hub.docker.com/r/jchodera/omnia-linux-anvil/~/dockerfile/)
to include things like the AMD SDK, TexLive, and CUDA for GPU builds. The updated version will ensure packages
can work on the `conda-forge` platform which is CentOS 6 based.

1. Stop building packages that appear in `conda-forge`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"For packages that appear in conda-forge, remove the corresponding recipes in omnia"


We want to minimize the amount of work we have to do as maintainers. To that end, we will stop building things
which freely appear on `conda-forge` and maintained by someone other than us!
For reproducibility purposes, we will keep our previously compiled versions, but they will not longer be updated.

1. Begin migration of "nearly-pure Python" and non-OpenMM dependent packages
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Allow recipes that do not depend on OpenMM to migrate from omnia to conda-forge


* Packages which do not depend on OpenMM and can be run on CPUs only should start migrating over to `conda-forge`
in preparation for the total migration.
* Packages which can compile with just the `conda-forge` linux-anvil should also start migrating.
* *We highly encourage devs of individual packages to start migrating now.*
* **Once a package is on conda-forge, it should no longer depend packages from omnia!**

1. Begin migration of `omnia` dependent (but not OpenMM dependent) packages to `conda-forge`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this stage is meaningful. We don't want users to migrate recipes that depend on other omnia recipes until their dependencies have migrated over---do we?


* As more of the `omnia` dependencies appear on `conda-forge`, more packages can start moving over.
* This stage can

1. Determine the appropriate way to build packages which require more than the `conda-forge` linux-anvil can provide

* The `conda-forge` linux-anvil does not support some things such as some LaTeX packages, AMD SDK, and CUDA files.
* We will need to reach out to the conda-forge people to see what the best course of action is

1. Move OpenMM to `conda-forge`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Migrate OpenMM to conda-forge"


* This will highly depend on the previous point and will probably have the longest time frame.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just say "This requires us to identify the best way to add CUDA and AMD APP SDK developer tools to the conda-forge linux-anvil docker image. We have some ideas, but no concrete solution yet."


1. Move packages which depend on OpenMM to `conda-forge`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"Migrate remaining packages to conda-forge"

This can be combined with the final "Finish the move"


1. Finish the move of all packages to `conda-forge`
* Also ensure that all former omnia packages can be installed without the `omnia` conda channel

1. Change this repo into an archive for reproducibility.

### How to migrate to `conda-forge` (for existing packages)

PLACEHOLDER

### Supported versions

Python packages are built against latest two releases of python (3.5 and 3.6) and python 2.7.
Expand All @@ -31,16 +101,19 @@ Packages which have a binary dependency on [numpy](http://www.numpy.org/) are bu

The recipes here are automatically built using [Travis-CI](https://travis-ci.org/) for `linux` and `osx` and [Appveyor-CI](http://www.appveyor.com/) for `win`.

For `linux` builds, we use a modified version of the [Holy Build Box](http://phusion.github.io/holy-build-box/), available [here](https://github.com/omnia-md/omnia-build-box), to ensure that the packages are fully compatible across multiple linux distributions and versions.
For `linux` builds, we use a modified version of the
[conda-forge linux-anvil](https://github.com/omnia-md/omnia-linux-anvil/blob/master/Dockerfile),
to ensure that the packages are fully compatible across multiple linux distributions and versions.
This build image contains the additional tools:
* [clang](http://clang.llvm.org/) 3.8.1
* [TeXLive](https://www.tug.org/texlive/) 2015
* [TeXLive](https://www.tug.org/texlive/) 2016
* The [CUDA](https://developer.nvidia.com/cuda-toolkit) Toolkit version 8.0
* The [AMD APP SDK](http://developer.amd.com/tools-and-sdks/opencl-zone/amd-accelerated-parallel-processing-app-sdk/) 3.0

There is an additional image which has [clang](http://clang.llvm.org/) 3.8.1

To build a package yourself, run `conda build <package_name>`, or `./conda-build-all ./*` to build multiple packages across each of the supported python/numpy configurations.

### Contributing a recipe
### Contributing a recipe (this has not been updated to reflect the conda-forge changes)

1. Fork this repo
2. Add your `conda` recipe for building your package `packagename` in a subdirectory called `packagename`. Feel free to use other recipes here as examples.
Expand Down
Empty file added appdirs/FLAG_FOR_REMOVAL
Empty file.
5 changes: 3 additions & 2 deletions appveyor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,10 +57,11 @@ environment:
install:
- "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%"
- set PATH=C:\MinGW\msys\1.0\bin;%PATH%
- conda update -yq --all
- conda config --set channel_priority false
- conda config --add channels omnia
- conda config --add channels salilab
- conda update -yq --all
- conda install -yq conda-build=2.1.5 jinja2 anaconda-client
- conda install -yq conda-build jinja2 anaconda-client
- powershell .\\devtools\\appveyor\\missing-headers.ps1
# conda-build for some inane reason, effectively puts C:\cygwin\bin at the
# front of the PATH, ahead of everything else, regardless. See
Expand Down
Empty file added arpack/FLAG_FOR_REMOVAL
Empty file.
Empty file added autograd/FLAG_FOR_REMOVAL
Empty file.
Empty file added behave/FLAG_FOR_REMOVAL
Empty file.
Empty file added bhmm/FLAG_FOR_REVIEW
Empty file.
Empty file added bison/FLAG_FOR_REMOVAL
Empty file.
Empty file added ccache/FLAG_FOR_REMOVAL
Empty file.
Empty file added clapack/FLAG_FOR_REVIEW
Empty file.
1 change: 1 addition & 0 deletions conda-build-all
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,7 @@ def main():
help="Set the Python version used by conda build",
metavar="PYTHON_VER",
default='27,34,35,36',

)
p.add_argument(
'--numpy',
Expand Down
Empty file added corner/FLAG_FOR_REMOVAL
Empty file.
Empty file added coverage/FLAG_FOR_REMOVAL
Empty file.
1 change: 1 addition & 0 deletions coverage/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ build:
number: 0
entry_points:
- coverage = coverage.cmdline:main
skip: True

requirements:
build:
Expand Down
Empty file.
Empty file added cvxopt/FLAG_FOR_REMOVAL
Empty file.
Empty file added cvxpy/FLAG_FOR_REVIEW
Empty file.
2 changes: 2 additions & 0 deletions cvxpy/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ package:

build:
number: 0
# No mingwpy on python 3.5+ which is needed by cvxp[t
skip: True # [win and (py35 or py36)]

source:
git_url: https://github.com/cvxgrp/cvxpy
Expand Down
40 changes: 0 additions & 40 deletions devtools/centos5-build-box/Dockerfile

This file was deleted.

78 changes: 0 additions & 78 deletions devtools/centos5-build-box/texlive.profile

This file was deleted.

14 changes: 2 additions & 12 deletions devtools/docker-build.sh
Original file line number Diff line number Diff line change
@@ -1,19 +1,9 @@
#!/bin/bash
set -e
# Activate Holy Build Box environment.
source /hbb_exe/activate

# Disable PYTHONPATH
unset PYTHONPATH

set -x
curl -s -O https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -p /anaconda
PATH=/opt/rh/devtoolset-2/root/usr/bin:/anaconda/bin:$PATH
conda config --set channel_priority false
conda config --add channels omnia
conda install -yq conda-build=2.1.5 jinja2 anaconda-client
# Install missing LaTeX docs
tlmgr install fncychap tabulary capt-of eqparbox environ trimspaces
conda install -yq conda-build jinja2 anaconda-client

/io/conda-build-all -vvv $UPLOAD -- /io/*

Expand Down
4 changes: 3 additions & 1 deletion devtools/osx-build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,11 @@ brew tap -y caskroom/cask
curl -s -O https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh;
bash Miniconda3-latest-MacOSX-x86_64.sh -b -p $HOME/anaconda;
export PATH=$HOME/anaconda/bin:$PATH;
conda config --set channel_priority false;
conda config --add channels conda-forge;
conda config --add channels omnia;
conda config --show;
conda install -yq conda-build=2.1.5 jinja2 anaconda-client;
conda install -yq conda-build jinja2 anaconda-client;

#export INSTALL_CUDA=`./conda-build-all --dry-run -- openmm`
export INSTALL_OPENMM_PREREQUISITES=true
Expand Down
Empty file added docopt/FLAG_FOR_REMOVAL
Empty file.
Empty file added doxygen/FLAG_FOR_REMOVAL
Empty file.
Empty file added ecos/FLAG_FOR_REVIEW_DEFAULT
Empty file.
5 changes: 2 additions & 3 deletions ecos/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,8 @@ source:
md5: 39cc4c4a5598cc51ca4dc9aec810f41a

build:
skip:
# ecos/src/ecos.c(1093): error C2065: '_TWO_DIGIT_EXPONENT': undeclared identifier
- [win and py35]
skip: True # [win and (py35 or py36)]
# Disable for now under certain windows conditions

requirements:
build:
Expand Down
Empty file added emcee/FLAG_FOR_REMOVAL
Empty file.
Empty file added fastcluster/FLAG_FOR_REMOVAL
Empty file.
3 changes: 3 additions & 0 deletions fftw3f/FLAG_FOR_REVIEW_FFTW
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
So this one may have a name colision with FFTW on conda-forge. It appears to be the same thing.

People will probably need to rename their dependencies
Empty file added flex/FLAG_FOR_REMOVAL
Empty file.
Empty file added freeglut/FLAG_FOR_REMOVAL
Empty file.
Empty file added glew/FLAG_FOR_REVIEW
Empty file.
Empty file added graphviz/FLAG_FOR_REMOVAL
Empty file.
Empty file added latexcodec/FLAG_FOR_REMOVAL
Empty file.
Empty file added m4/FLAG_FOR_REMOVAL
Empty file.
Empty file added mdtraj/FLAG_FOR_REMOVAL
Empty file.
Empty file added mingwpy/FLAG_FOR_REMOVAL
Empty file.
Empty file added mpich/FLAG_FOR_REMOVAL
Empty file.
Empty file added msmbuilder/FLAG_FOR_REMOVAL
Empty file.
Empty file added msmexplorer/FLAG_FOR_REMOVAL
Empty file.
Empty file added msmtools/FLAG_FOR_REMOVAL
Empty file.
Empty file added multiprocess/FLAG_FOR_REMOVAL
Empty file.
Empty file added nglview/FLAG_FOR_REMOVAL
Empty file.
Empty file added nose-timer/FLAG_FOR_REMOVAL
Empty file.
Empty file added openblas/FLAG_FOR_REMOVAL
Empty file.
1 change: 1 addition & 0 deletions openforcefield/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ source:
build:
preserve_egg_dir: True
number: 0
skip: True # [win]

requirements:
build:
Expand Down
2 changes: 1 addition & 1 deletion openmm-example-plugin/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ source:
git_rev: f1e12743d7c5403bb6d7ad8da69af6a2f27e1b26

build:
skip: True # [win]
skip: True # [win]

requirements:
build:
Expand Down
Loading