Update to Omnia Linux anvil #761

Lnaden · 2017-04-17T16:36:10Z

Start the process for updating to the CentOS 6 image.

This is very much a work in progress since we cannot reasonably test all the packages at once. So this initial commit will be to make sure the basics of the new docker image work.

This that need to be done in loose order:

Ensure the docker image is pulled correctly, including EPEL 6
Ensure the conda-forge channel and packages are being fetched
Update some non-OpenMM dependent packages' build number to see if they can build on the new image (future PRs)
Remove packages which are in conda-forge that we dont need in omnia anymore (future PRs)

Replaces and closes #667

Tagging @jchodera

Initial README update (still WIP)

Actually update README (still WIP)

Unpin conda-build version

…d for a while

Lnaden · 2017-04-17T16:50:46Z

This keeps xcode on 6.4, no fix is in place

…red for channel_priority first

This is temporary and coverage may be removed outright

Lnaden · 2017-04-17T19:49:57Z

I have added files to serve as flags for removal, review, and other notes on various packages. Most of these are FLAG_FOR_REMOVAL files to indicate that the package can probably be removed since the package already exists as part of conda-forge. These should be reviewed and discussed though.

Allow openmoltools to build on windows

… missing on 3.5+

Attempt to get clang from clangdev of conda-forge for OpenMM TEMPORARY: Made OpenMM build only on Linux and Python 3.5 for rapid testing REMOVE THIS TEMP CHANGE BEFORE MERGE

Lnaden · 2017-04-19T18:11:28Z

Good news! The new docker image can successfully build OpenMM. I have also confirmed that it is correctly using clang as the compiler (which now comes from conda-forge and not an addon to the image, its finding and building the OpenMMCUDA targets so the CUDA toolkits are installed correctly, its finding and building the OpenMMOpenCL targets from the AMD SDK so those are also installed. The docs are being built which means the TeXLive kit is installed correctly as well.

This validates the docker image is constructed correctly and packages can be built on it. I think it we are ready to move onto testing other packages, removing my debug lines, start reviewing the packages for removal.

Also it would be good if someone else look at least at the docker file to make sure it gets everything we need.

peastman · 2017-04-19T19:40:59Z

Yay!

Lnaden · 2017-04-19T19:45:55Z

@peastman is there a particular reason we need to keep OpenMM pinned to clang 3.8.1? They have clang 3.9.1 on conda-forge as well so this may be a decent time to reassess the compiler version if there is not some hard restriction.

peastman · 2017-04-19T19:54:08Z

In general I'm fine with using the most recent version, but for releases it's important to know and control what compiler is used. For example, when we release a patch update to fix bugs, it needs to be built with the same compiler version as the release it's a patch to. Compiler updates have been known to introduce bugs.

Lnaden · 2017-04-19T19:59:17Z

Okay. I'll keep it pinned to 3.8.1 in the main omnia channel since we are not actually releasing a new version yet. We can reconsider the version for any planned major releases going forward. Thanks!

Removed debug lines

Lnaden · 2017-04-20T15:05:31Z

Okay, I have updated the README (could use some review @jchodera ), removed the debugging line, and set all the packages I think are ready for removal and review.

OpenMM builds on the new image, so I think we are ready to go with the migration to CentOS 6 based and conda-forge. I need to reach out to the CF people yet and figure out how to get things like the AMD SDK and the CUDA tools for actually taking OpenMM to CF, but one step at a time here.

When this PR merges: `conda-forge` will become a requirement for packages built after this PR

Lnaden · 2017-04-21T16:06:26Z

Okay, I fixed the OSX builds by not overwriting AppleClang with clangdev. But now there is a new problem.

OSX OpenMM build appears to complete, but not install correctly so the test fails to find the simtk package like it was not installed right. The linux builds are fine, but there should not have been a change to the osx side. Has anyone ever seen this issue? file here, search for run_test to get to the failure since its the raw log.

peastman · 2017-04-21T17:02:29Z

No idea. I don't see any indications in the log of what the problem is. It runs the PythonInstall target, and appears to complete successfully:

[100%] Built target PythonInstall

When it gets to the test, it claims it's about to install the package it just built:

TEST START: /Users/travis/anaconda/conda-bld/osx-64/openmm-7.1.0rc1-py35_1.tar.bz2
...
The following NEW packages will be INSTALLED:
...
    openmm:          7.1.0rc1-py35_1  local

It doesn't report any errors while installing. Yet it then reports that OpenMM isn't installed:

===== testing package: openmm-7.1.0rc1-py35_1 =====
running run_test.py
Traceback (most recent call last):
  File "/Users/travis/anaconda/conda-bld/openmm_1492787167059/test_tmp/run_test.py", line 7, in <module>
    from simtk import openmm
ImportError: No module named 'simtk'
TESTS FAILED: openmm-7.1.0rc1-py35_1
ERROR: File does not exist: /Users/travis/anaconda/conda-bld/osx-64/openmm-7.1.0rc1-py35_1.tar.bz2

Possibly that last line is a clue? Should that file have been created?

Lnaden · 2017-04-21T18:11:30Z

Everything I see tells me it installed correctly, but it looks like the files just were not copied correctly.

Possibly that last line is a clue? Should that file have been created?

Sadly no, that is just the generic error that conda-build throws as the last step as it checks for the tarbal as its last action, independent of if something else failed.

I think this might be conda problem, I have some things I can test to find out.

…rom the same channel

Lnaden · 2017-04-21T19:50:58Z

I cannot find a way to build on OSX without conda-build just failing for some unknown reason. Moving the channel priority to later appears to break the conda-build-all script. There are no helpful errors. Will keep debugging

Lnaden · 2017-04-21T21:40:21Z

So the problem I am seeing on the OSX build is exactly the same problem we are seeing on the OpenMM Python 3.6 windows builds here: openmm/openmm#1758 (comment)

How to fix it though....

Lnaden · 2017-04-24T14:43:14Z

Okay, here is what I can tell from digging around on the net.....

The problem comes from the fact that python is built against a certain MACOSX_DEPLOYMENT_TARGET minimum which is separate from the environment variable. This appears to be tied to the python executable itself from what I can read (I think, there may be a way to change this).

When building new packages, the environment MACOSX_DEPLOYMENT_TARGET cannot be lower than the python that was used to compile it. So what we are seeing is the python we get on OSX for python versions not 3.4 are built against osx 10.9, then when we set the env MACOSX_DEPLOYMENT_TARGET, it complains and crashes.

There are some oddities with this though. My local version of Python 3.5.3 from default channel was built against macosx 10.7, but the same version of python 3.5.3 from conda-forge is built against osx 10.9. My guess is the version of python 3.4, which stopped getting updates, was built against the 10.6 osx, which is why we see this difference.

So now the question is, can we set the version to built against from the python compiler, or are we going to have to find that magic python version which is 10.7 compiled for each version of python.

On a side note, this is excessively dumb.

jchodera · 2017-04-24T14:44:51Z

On a side note, this is excessively dumb.

+1000

jchodera · 2017-04-24T14:45:14Z

Maybe this is a good time to report it on the conda-forge and possibly anaconda issue trackers to solicit advice?

Lnaden · 2017-04-24T14:50:01Z

Maybe this is a good time to report it on the conda-forge and possibly anaconda issue trackers to solicit advice?

Maybe, the problem is this does not appear to be a conda-related issue. I think this is a pure-python issue since I think setup.py scripts would throw this issue. Evidence here where the Python folk were pointing it out back in Python 3.2/3.3. This is my evidence that it looks like a Python deployed to a min version cannot deploy packages it builds to an even smaller min version.

jchodera · 2017-04-24T14:53:02Z

It's conda-related insofar as the python versions made available in these channels were compiled with specific versions of the OSX frameworks, right?

Are all the conda-forge python versions built with 10.7 or earlier? If so, could we just

add the conda-forge channel
conda clean -pltis --yes to get rid of cached anaconda python installs
create a new env with the conda-forge python
then add other channels and build?

Lnaden · 2017-04-24T14:59:58Z

I just realized this conversation is happening on the wrong issue, linking back to the comment here

Lnaden · 2017-05-01T12:26:04Z

Hold off on merging this until OpenMM 7.1.1 is published at minimum

I think this is otherwise ready to go for the most part (some last builds running)

xref: #768

jchodera · 2017-05-01T12:31:39Z

Should we at least disable the current failing byulds? I'm a separate pr?

Lnaden · 2017-05-01T12:47:42Z

I can and did for most of them already. The remaining errors being raised are legitimate concerns that I don't want to silence. They are mostly caused by lack of OpenMM on Python 3.6 for Windows that we never fixed. When we cut 7.1.1, we will want to fix that first.

Lnaden · 2017-05-01T12:52:44Z

And to be precise, the remaining errors are all related to missing OpenMM builds.

Lnaden · 2017-05-15T17:39:14Z

Guess what! Its merging day!

Lnaden added 7 commits April 17, 2017 11:20

Update docker file

831fd41

Initial README update (still WIP)

Fix openmm

38c83df

Change install method for autodoc to python ... instead of pip ...

9e3a83e

Actually update README (still WIP)

Change Travis to use new docker image

3956f2a

Unpin conda-build version

Update README

7d80095

Removed the old centos5-build-box folder which has been hanging aroun…

81ff08d

…d for a while

Make sure conda-forge is added

1141426

Lnaden mentioned this pull request Apr 17, 2017

[WIP] Sundry changes to work with omnia-linux-anvil image #667

Closed

Lnaden added 7 commits April 17, 2017 12:53

Disable openmm-example-plugin for the purpose of testing this PR

ac1509b

Try to fix tar problem on el_capitan

a13342a

Try bumping to Xcode8.1 (OSX 10.12) to fix the mac sudo bug on tar

d00429b

Downgrade back to xcode6.4

da91ebc

Reverse conda command order on windows to ensure we get version requi…

3301a06

…red for channel_priority first

Disable coverage for builds to make windows builds go

44df8ce

This is temporary and coverage may be removed outright

Added flags for remove, review, and a few notes

8972bbe

Stop building ecos since its on the default channel.

90df632

Allow openmoltools to build on windows

jchodera mentioned this pull request Apr 18, 2017

Centos5 EOL #763

Closed

Lnaden added 4 commits April 18, 2017 13:39

Remove some windows builds

522650c

Try to fix scs on windows

fb6a60f

Fix some packages which rely on cvxopt that cant build due to mingwpy…

2bf4719

… missing on 3.5+

Update readme

3af0ab8

Attempt to get clang from clangdev of conda-forge for OpenMM TEMPORARY: Made OpenMM build only on Linux and Python 3.5 for rapid testing REMOVE THIS TEMP CHANGE BEFORE MERGE

More updates to README

4ea36e8

Removed debug lines

Lnaden added 2 commits April 21, 2017 10:14

Doing some debugging of openmm on mac

43965be

Make sure AppleClang is not overwritten (test)

9381644

Lnaden added 3 commits April 21, 2017 14:17

Change osx conda operations to ensure the conda binary updates come f…

c8a6c72

…rom the same channel

Try and mimic conda-forge

99f0e53

make sure conda and conda-env update right

a4e05d1

Lnaden added 2 commits April 21, 2017 16:46

Try pinning the conda-build to 1 version previous

a2bfaff

Try doing the conda updates first then adding the extra channels after

d91aab9

Lnaden added 2 commits April 24, 2017 16:20

Set the MACOSX_DEPLOYMENT_TARGET in OpenMM

9835c81

Roll back debug line

5e51a3c

Lnaden mentioned this pull request Apr 25, 2017

Organising the conda communities and establishing best practices. conda-forge/staged-recipes#299

Closed

disable openmm example plugin on py36 for now

0b181fe

Lnaden merged commit a21c278 into master May 15, 2017

Lnaden mentioned this pull request May 15, 2017

Added Libnetcdf and netcdf4 to the repo #740

Closed

jchodera deleted the linux-anvil branch December 4, 2017 22:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to Omnia Linux anvil #761

Update to Omnia Linux anvil #761

Lnaden commented Apr 17, 2017 •

edited by jchodera

Loading

Lnaden commented Apr 17, 2017 •

edited

Loading

Lnaden commented Apr 17, 2017

Lnaden commented Apr 19, 2017

peastman commented Apr 19, 2017

Lnaden commented Apr 19, 2017

peastman commented Apr 19, 2017

Lnaden commented Apr 19, 2017

Lnaden commented Apr 20, 2017

Lnaden commented Apr 21, 2017

peastman commented Apr 21, 2017

Lnaden commented Apr 21, 2017

Lnaden commented Apr 21, 2017

Lnaden commented Apr 21, 2017 •

edited

Loading

Lnaden commented Apr 24, 2017 •

edited

Loading

jchodera commented Apr 24, 2017

jchodera commented Apr 24, 2017

Lnaden commented Apr 24, 2017

jchodera commented Apr 24, 2017

Lnaden commented Apr 24, 2017

Lnaden commented May 1, 2017 •

edited

Loading

jchodera commented May 1, 2017

Lnaden commented May 1, 2017

Lnaden commented May 1, 2017

Lnaden commented May 15, 2017

Update to Omnia Linux anvil #761

Update to Omnia Linux anvil #761

Conversation

Lnaden commented Apr 17, 2017 • edited by jchodera Loading

Lnaden commented Apr 17, 2017 • edited Loading

Lnaden commented Apr 17, 2017

Lnaden commented Apr 19, 2017

peastman commented Apr 19, 2017

Lnaden commented Apr 19, 2017

peastman commented Apr 19, 2017

Lnaden commented Apr 19, 2017

Lnaden commented Apr 20, 2017

When this PR merges: conda-forge will become a requirement for packages built after this PR

Lnaden commented Apr 21, 2017

peastman commented Apr 21, 2017

Lnaden commented Apr 21, 2017

Lnaden commented Apr 21, 2017

Lnaden commented Apr 21, 2017 • edited Loading

Lnaden commented Apr 24, 2017 • edited Loading

jchodera commented Apr 24, 2017

jchodera commented Apr 24, 2017

Lnaden commented Apr 24, 2017

jchodera commented Apr 24, 2017

Lnaden commented Apr 24, 2017

Lnaden commented May 1, 2017 • edited Loading

jchodera commented May 1, 2017

Lnaden commented May 1, 2017

Lnaden commented May 1, 2017

Lnaden commented May 15, 2017

Lnaden commented Apr 17, 2017 •

edited by jchodera

Loading

Lnaden commented Apr 17, 2017 •

edited

Loading

When this PR merges: `conda-forge` will become a requirement for packages built after this PR

Lnaden commented Apr 21, 2017 •

edited

Loading

Lnaden commented Apr 24, 2017 •

edited

Loading

Lnaden commented May 1, 2017 •

edited

Loading