Allow using conda environment.yml files to create environments

[antonl]

How would this feature be useful?
Conda has two environment creation commands: conda create and conda env create. They differ in that the latter can reproduce an environment specified in a environment.yml file, including packages that are installed using pip, while the former requires a list of packages.

I would like to create specific frozen environments using conda env create and then run tests (or additional commands) in them.

Describe the solution you'd like
One could create a CondaEnv subclass accepts a list of environment files and uses them to clone the environment. A side effect would be that it would ignore the python version.

Describe alternatives you've considered
It is possible to just create the environments in two steps by parsing each environment.yml file, installing the conda dependencies, and then installing the pip requirements.

[tswast]

This feels like too far from how the virtualenvs work, which is to create an empty environment and then install the needed packages inside the session.

What if in addition to conda_install, there was a conda_env function? Then you could call:

# --prefix is auto-populated to point to the environment used by the test session
session.conda_env("update", "--file", "environment.yml", " --prune")

https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#updating-an-environment

[antonl]

Cool! I didn't know about the conda env update function! I developed a workaround in the meantime, which is to just parse the yaml file myself, and then call conda install and pip install in sequence, but this is much better.

As a workaround, I'll just call session._run() with conda env update command for the time being:

session._run(*[
        'conda',
        'env',
        'update',
        '--prefix',
        session.virtualenv.location,
        '--file',
        str(path),
        '--prune',], silent=True)

[salotz]

+1 for loading env.yaml files. I use them quite a bit for pinning versions and such.

[theacodes]

I'm happy to see support for this, but I don't know much about Conda to be honest. If Tim is willing to review I'd more than welcome a PR.

…

On Tue, Apr 7, 2020 at 9:00 AM salotz ***@***.***> wrote: +1 for loading env.yaml files. I use them quite a bit for pinning versions and such. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#260 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB5I42EEKE4DESYVALQMTTRLNEZ7ANCNFSM4JGNQJQA> .

[salotz]

Its likely that I'll swing around to this at some point since I was basically building the same thing ad hoc in my own projects before I found nox.

[tswast]

I can review a PR when it comes

[salotz]

I have been using something similar to what @antonl gave above just to see if there are any problems.

        session.run(
            'conda',
            'env',
            'update',
            '--prefix',
            session.virtualenv.location,
            '--file',
            str(conda_env),
            # options
            silent=False)

I just remove the --prune option so you don't get rid of other things you installed.

The only issue so far (and in my experience is a problem with conda and perhaps not really fixable with nox) is that the order of installation matters.

If you install some things with pip in the session before running the above command, weird things will happen that break your env. I had to put this as the first thing.

Otherwise it seems to work. From looking at the code for specific commands like session.conda_install is that some checks are done to make sure the env backend is the right one.

Again I might get around to making a PR at some point, but I will run it like this for a while and see if there are any other issues.

[smarie]

It looks good @salotz ! Do you have feedback so far about such usage ?

[smarie]

My personal experience with the workaround provided by @salotz is that it takes far too much time to execute the update command everytime the session is run. So I would rather vote for the possibility to declare the environment file (this can be a conda yaml file or a pip requirements.txt file) in some new argument of @nox.session(), and have some kind of mechanism checking if the file changed or not since last run. If it changed, the environment would be recreated.

What do you think ?

[antonl]

That would be nice. One way to do this is to run something like conda env export and hash that, look for differences from creation. There is a problem though; if you pip install anything, that the environments will always compare not equal.

[tswast]

some kind of mechanism checking if the file changed or not since last run

This feels a bit outside of how nox is currently architected. Is there an FR to conda itself to be made here for a "fast env update" that does a faster check to see if any changes need to be made?

[smarie]

not that I know unfortunately but I am not very familiar with the conda evolutions roadmap - I rather consider it as marginally evolving now

[salotz]

This feels a bit outside of how nox is currently architected.

I agree with this.

And yes doing anything with conda is painfully slow other than keeping the same env and jamming lots of crap into it one at a time whenever an import fails. Which I think is the intended use case (not for specifying groups of dependencies for specific purposes).

I'm actually just fine with the way things are and not having nox support them at all. Since there is a lot specific interactions between the order of install between conda and pip that I know how to deal with in my specific workflows, but that I am not curious in figuring out in general. Especially since I'm not particularly confident that this won't change very rapidly on conda's end.

have some kind of mechanism checking if the file changed or not since last run. If it changed, the environment would be recreated.

Like tswast was saying this would be a totally different architecture. Yes that would be ideal, but it would be a different project. I've hand-rolled my own nox-like env-creator/task-runner just using invoke (https://github.com/salotz/jubeo/blob/master/modules/env.py) and while not trivial its not too onerous. My current project is to upgrade from invoke to pydoit, which would give you the incremental build aspect.

[smarie]

My current project is to upgrade from invoke to pydoit, which would give you the incremental build aspect.

In my current project the reason why I can not use nox even if I would love to, is precisely this incremental+task dependency aspect ; this is why I end up using pydoit like you. But since doit is a bit hard to get right and use efficiently, I use doit-api. I encourage you to check it out ! (I'm the author :) )

[salotz]

@smarie oh cool! ya I haven't actually got into the meat of using doit for a lot of stuff yet. I agree that doit doesn't feel very pythonic, I'll check your thing out.

But relevant to envs (and this thread somewhat) maybe we can work together on getting an env manager similar to nox et al. (dox??). Looks like its out of scope and interest to nox project. Feel free to email me, I don't want to hijack this issue any further.

[smarie]

@salotz thanks for the proposal and feedback! Unfortunately my bandwidth will be extremely limited in the upcoming months for family reasons - not really the time for opening new projects :) . However I'll be happy to contribute with ideas/PRs if the ground is solid. Note that maybe the nox internal API is stable enough so that its environment management features can be reused as is - indeed it would be sad to reinvent the wheel / duplicate efforts. Let me know through email if you start something and would like to get some feedback/review at some point! Sorry for not having more time to devote to such interesting projects.

[salotz]

I wouldn't be starting soon either, but I'll DM you when/if I do. TBH there isn't too much wheel to reinvent. I think we can all coexist in peace since its fully developer side stuff. :)

[theacodes]

There's a solution proposed here: #346 (comment)

I'm still happy to review a PR, but again, I don't know enough about conda so it's really up to y'all. :)

[mforbes]

I am still playing with condo env update to see if it is reliable... will try to pull together a PR at some point, but have other looming deadlines so this probably won't be until the new year.

[choldgraf]

Just a quick note that I got a pretty lightweight script working with simple environment.yml files - it uses pyyaml and assumes that the final entry in dependencies is a pip entry, but could be made a bit smarter with one or two extra lines.

Sharing here in case it's useful for others!

from yaml import safe_load
from pathlib import Path

# Parse the environment files we'll need later
environment = safe_load(Path("environment.yml").read_text())
conda = environment.get("dependencies")
requirements = conda.pop(-1).get("pip")


def install_environment(session):
    for conda_pkg in conda:
        session.conda_install(conda_pkg)
    for pkg in requirements:
        # We split each line in case there's a space for `-r`
        session.install(*pkg.split())
    session.install("-e", ".")


@nox.session(venv_backend="conda")
def build(session):
    install_environment(session)
    ...now do other stuff

[edublancas]

thanks for sharing this, @choldgraf!

I modified it a bit so all the conda/pip dependencies are installed in a single call, this is faster because each it only invokes conda's and pip's dependency resolver once. I'm using this:

import nox

from yaml import safe_load
from pathlib import Path

environment = safe_load(Path("environment.yml").read_text())
conda = environment.get("dependencies")
requirements = conda.pop(-1).get("pip")


def install_environment(session):
    session.conda_install(*conda)
    session.install(*requirements)


@nox.session(venv_backend="conda")
def build(session):
    install_environment(session)