When to package software which is already in the default conda channel

[ocefpaf]

(@JanSchulz brought this up in #16 (comment))

I Agreed with @JanSchulz that we should avoid as much as possible to add packages in conda-forge that are available in the default channel.

However, we already have a few redundant packages (pyproj, shapely, geos, and more to come soon). The reason for th1 redundancy is that those packages are partially broken in the default channel.
(And we could not find a proper channel of communication to send the recipe patch back to them.)

Maybe, when fixing a default channel package we should allow the package addition here as long as there is a plan to send that fix back to the default channel, and to remove the package from conda-forge once that happens.

[jankatins]

Maybe register a new channel: "temporary-fixes"?

[jankatins]

Maybe @mcg1969 has some idea how this could be handled?

[ChrisBarker-NOAA]

I'm not sure it's worth a different channel.

But I wonder if we should give the package a different name, otherwise things can get pretty tangled up:

gdal-cf
or ????

(cf for conda forge...)

That means that anyone using it has to change the dependency, but are there going to be any packages outside of conda-forge that depend on a conda-forge special build???

[ocefpaf]

But I wonder if we should give the package a different name

👎 that can be confusing. We can maybe "sign" the packages with a build string like

build:
    string: conda-forge

But IMO just having the package in a different channel should be enough to disambiguate.

[jankatins]

The reason for a different channel is IMO that I suspect that at some point users will add conda-forge to their default channels and a different channel than conda-forge means that users can get the updated/fixed version with conda install -c whatever matplotlib but not with a simple conda update --all.

[ChrisBarker-NOAA]

I'm not sure the channel disambiguates -- does conda prioritize default or your other channels?

But I wonder where all this goes -- with PyPi, it's up to each package maintainer to keep things up to date. with anaconda, it's up to third parties -- mainly continuum. so if they don't, then what?

I'm hoping continuum will adopt a more community model, where folks can easily provide PRs -- it seems it would only save them work. So we'll see.

In the meantime, conda-forge may become the community channel, and I"d say if you want the latest and greatest, then you add that channel -- and you'll get the new MPL, or whatever, if conda-forge provides a newer one than default.

We will probably want to clean things out as continuum catches up.

I'm still wondering about the naming though:

continuum builds package-1.2.1

a new version comes out, and folks want it -- but continuum is being slow on teh draw.

conda-forge provides package-1.3.1

all is good.

now continuum catches up and builds a package-1.3.1 -- now there are two, with the same version. And maybe they are even incompatible in some way. This could make a mess for our users.

If we go with Jan's approach, then this would be cleaner, but users would have to explicilty make a point of getting a newer version -- I think that would be awkward and often missed as an option.

[jankatins]

debian has a similar problem when using backports, but could manage that with a special version suffix (~) which sorts after the version without the suffix:

https://www.debian.org/doc/debian-policy/footnotes.html#f37

One common use of ~ is for upstream pre-releases. For example, 1.0beta1svn1245 sorts earlier than 1.0~beta1, which sorts earlier than 1.0

As this probably requires changes in conda, I would vote for 1.3.0_real_1.3.1 which sorts before 1.3.1, but after 1.3.0.

[jankatins]

Hah, it seems conda already implements such a scheme per default, so 1.3.1.cf is lower than 1.3.1
https://github.com/conda/conda/blob/2ba04a6b2617227de578f4af54ff11115f97ca5c/conda/version.py#L81

[ChrisBarker-NOAA]

nice!, maybe we can use that, then.

-CHB

[ocefpaf]

I'm not sure the channel disambiguates -- does conda prioritize default or your other channels?

Disambiguate? Yes! Solve the install/update problem? No. I even saw an e-mail from Travis Oliphant today discouraging the use of conda update --all because of how conda solves dependencies.

I'm hoping continuum will adopt a more community model, where folks can easily provide PRs -- it seems it would only save them work. So we'll see.

👍

[jankatins]

discouraging the use of conda update --all because of how conda solves dependencies

This is probably conda/conda#1967

[pelson]

Yes. The conda solver has recently been overhauled, and my experience with it is that the performance has been dramatically improved.

If two channels are configured to be used by conda, and both provide the same package name, version and build number, then conda chooses the package from the channel defined first in the config.

now there are two, with the same version. And maybe they are even incompatible in some way.

This is the real problem here. We have, in the past, fixed packages (on IOOS and SciTools) which Continuum package, often by releasing a newer version. The problem comes when Continuum update the version of the software they package, but don't actually fix the problem. This has happened on several occasions with packages such as Shapely and pyproj. From a user's perspective, they are just updating their software and it goes from a functional state to a non-functional state - not really ideal. Because of the lack of a repository of canonical recipe source, all we have been able to do is report a problem with the package, not actually fix it (i.e. in the form of a PR).

---

@jakirkham, @tacaswell and @stefanv have all expressed an opinion on the subject of this issue in the past. Do any of you have comments on when it should be the place of conda-forge to package software which is already being packaged by Continuum?

[mcg1969]

@JanSchulz thanks for pinging me. There's not a whole lot I can officially say yet, except that we recognize the need to support alternatives to Continuum's default channels. We're actively working on a particular community channel solution, but it is not the only way forward, and it shouldn't be. We've been watching the Conda Forge project with enthusiasm. As we talk more we may be able to come up with some specific ways Continuum can help with it. But having an effort like this that Continuum does not control is beneficial to the Python community at large, so I'm grateful you're working on it.

I see three different problems with the default channel:

1. Not every package that people want can be, or should be, available there.
2. Continuum updates some of those packages less frequently than the primary developers of the packages.
3. The packages may not be compiled with the same options that some users like to see.

All three of these issues are an inevitable consequence of Continuum's finite resources for building and supporting packages. We certainly acknowledge that this isn't going to satisfy people who regularly bump up against one of these three problems. Heck I bump up against all three of these problems myself.

My particular perspective, as many of you know who have been watching my conda fixes recently, is on the dependency solver. I've been spending time overhauling it, and it's certainly going to fix some of the issues like conda update --all being slow, conda remove potentially breaking installs, etc. I'm glad @pelson is confirming that my improvements are beginning to make a difference.

But honestly, the mathematics of the solver isn't really the issue here, at least not directly. What you are discussing in this thread is basically the challenge of channel clashes. That is: how should conda handle things when two or more channels release versions of the same package? At the moment, conda effectively "merges" the channels together, so that the packages interleave with each other purely based on version and build numbers. That's clearly not a workable solution. For one thing, build numbers don't have meaning across channels; so for instance, build 1 from channel A may actually be newer than build 2 from channel B, and conda doesn't know that. This is something we need to decide on a fix for.

What we need, it seems to me, is to identify specific improvements to conda that would greatly improve the ability to use alternate, community-driven package channels. For instance:

1. A fix for conda that untangles packages/channel conflicts. For instance, we could say that the highest-priority channel is always preferred for a package, and any packages by the same name in lower-prioirity channels are ignored. But I could see a variety of other strategies, and perhaps conda should adopt several, choose on as a default, but make the others available by configuration.

2. An enhancement to conda that allows channel preferences to be adopted on a per-package and or per-environment basis. For example, perhaps I add the conda-forge channel as a lower-priority channel, but I actually prefer one of its packages to the one provided in default. There should be a way to specify that priority preference and persist it across updates and later installations.

This is the kind of thinking that would be very helpful for me personally. We really do want conda to be adopted more widely---heck, we'd be pleased if someone built their own Python distribution that used conda as a packaging model. And we'd like to find ways to enable groups like Conda Forge to flourish without having to wait on us. I actually do think that there are some changes to conda that we can push through in the short term that will greatly improve our ability to work in parallel.

[ocefpaf]

Having a way to specify channel preference globally or per env would be a really good addition to conda.

👍

[mcg1969]

I would really rather we tackle the channel collision problem correctly than to utilize weird version numbers or (even worse) track_features to disambiguate.

[stefanv]

If we could get Continuum to open recipes (or adopt community recipes upon submission, and then open those up), then perhaps much of the problem can be avoided? Ideally, we do not want multiple versions of numpy with the same version tag floating around. An alternative path is to build everything you need into your own channel. For mixed channels, I don't see a straightforward way of resolving what to install without additional meta-data. In Debian there is the concept of "pinning", which allows you to fix certain packages in place.

[ChrisBarker-NOAA]

@mcg1969: absolutely -- but we need help from conda itself to do it "right" -- are you speaking for continuum, hard to tell :-)

@ocefpaf wrote: "Having a way to specify channel preference globally or per env would be a really good addition to conda."

I think that would actually be a simple solution that would mostly solve the problem at hand -- folks could put the IOS channel, or conda-forge channel at their first preference, and then they would get the latest and greatest.

Granted, the default channel may get updated in a way that leapfrogs conda-forge, but I think it will be up to whoever is maintaining the conda-forge package to keep an eye on that.

And the default channel is clearly the upstream one -- conda-forge will be following its lead, so that could work.

[ChrisBarker-NOAA]

I've been thinking a bit about how this works with PyPi (and PyPi does work well, for the things i works well for, i.e. pure-python packages)

It is a totally different model -- PyPa only provides the infrastructure -- each and every package is maintained by individual package maintainers. Ideally, conda packaging could go that way, but it's going to be a long time (or never) before package authors in general support conda. (never mind non-python stuff....)

By my idea, at least, is that conda-forge becomes the PyPi-like place for conda packages -- it will start (has started) with groups of packages that are not in the default channel being maintained by a third party, but hopefully individual package authors will start to maintain their own packages. So we need to design the infrastructure to support that.

In fact, as a package author steps up to maintain a package, maybe it could even be removed from the default channel. In the long run, maybe continuum will need to maintain few packages, and rather, have Anaconda be the "curated" selection, but much of it would be pulled in from the authors' builds (OK, maybe that's a fantasy).

Anyway, what all this means is that it should be very easy for a package author to push builds to conda-forge, like it is now with PyPi being integrated into the PyPa stack (distutils, setuptools, pip, I"ve lost track...)

[jankatins]

One easy idea would be to add somethign like this:

conda install -c conda-forge --pin-channel matplotlib

That would add an entry to the config file that matplotlib should be taken from the conda-forge and all other packages with the same name form other channels should be discarded (e.g. simple add a step-0 to the solver which removes all matplotlib packages from other channels from the list of available packages).

This will help with the problem of "fixing" packages in the default channel (and IMO this should be the only part where conda-forge should package packages in the default channel).

Another step would be to configure the "default" channel, so that conda does not see the anaconda/Continuum packages at all. Not sure if that is possible today?

[ChrisBarker-NOAA]

@stefanv: Absolutely!

If the default channel was built from (mostly) recipes maintained in a public gitHub project(s), it would be monstrously easier to keep everything up to date and in-sync. We could/would do a lot of the work for continuum.

And they could start one package at a time (shapely?).. it wouldn't have to be a wholesale, all at once move.

I can imagine it's inertia more than anything else that's prevented this from happening so far, but it's a bit frustrating from outside.

-CHB

[jankatins]

If we could get Continuum to open recipes (or adopt community recipes upon
submission, and then open those up), then perhaps much of the problem can
be avoided?

I think it would already be enough if Continuum would add all their package recipes (as they are currently used -> the matplotlib recipe in the conda-recipes repo is out of date) and accept PRs for already included packages. I would be happy to add patches there if I know that they land on my HD a day after they are merged... Contiuum still would have the final say, it would speed up the updates on new upstream releases, and Continuum would have less work... (cc: @mcg1969 :-) )

[tacaswell]

@mcg1969 articulated many of my thoughts more coherently than I would have, I think channel-level precedence is the probably the right way to fix this, but I would like a way to control (maybe at the package level) if it goes with newest-possible or prefers a specific channel.

I was also thinking of the debian idea of 'pinning' as a model for how to do this.

For day-job we have been making aggressive use of 'postN' versioning (pulled directly from git describe via versioneer) which helps the case where the issue is fixes from up-stream project is adding/fixing things. Although, this can get funny if you are packing commits from side-branches and definitely does not help if the difference is different sets of locally applied patches or build configuration.

[jakirkham]

Pinning channel might be ok as long as it is per environment.

[ocefpaf]

@mcg1969 articulated many of my thoughts more coherently than I would have, I think channel-level precedence is the probably the right way to fix this, but I would like a way to control (maybe at the package level) if it goes with newest-possible or prefers a specific channel.

@tacaswell my Linux distro (OpenSUSE) does exactly that. I can set repository preferences that will be used when updating the system, but I can also do a "distribution upgrade" that will get the newest-possible from all repositories. This operations issues warnings stating that the user is responsible for the system stability when adding third party repository and performing the distribution upgrade. Conda is dangerously confusing with that! I can see tons of users breaking their system using conda-forge packages and going to Continuum mailing list to complain.

With that said. This behavior might be a long-term goal. Right now I believe that a global channel preference is already a big win.

[ocefpaf]

Pinning channel might be ok as long as it is per environment.

Agreed.

conda/conda#982

[jankatins]

For an idea on how to mark package/channel combinations as good/bad: conda/conda#2067

[jjhelmus]

We really do want conda to be adopted more widely---heck, we'd be pleased if someone built their own Python distribution that used conda as a packaging model.

That sounds like a challenge! Accepted, acpd, Another Conda-base Python Distribution.

On that topic and related to packaging software already in the default conda channel, would it be possible for someone from Continuum to clarify the license of the recipes in the ContinuumIO/anaconda-recipes repository? A number of those are prime candidates for use in conda-forge.

[jakirkham]

Nice! Do you have any thoughts on their integration into conda-forge, @jjhelmus?

[mcg1969]

I think that channels and subchannels in particular will become very powerful once something like 2323 is implemented. I think that may be the proper way to host multiple variants of the same package.

[ocefpaf]

I think that channels and subchannels in particular will become very powerful once something like 2323 is implemented. I think that may be the proper way to host multiple variants of the same package.

👍

[jakirkham]

Hmm...not sure I see how subchannels work or how that will fit into our infrastructure yet.

[jakirkham]

@gpilab, I noticed your channel recently and noticed that we have a lot of overlap in terms of packages we provide. Maybe you would be interested in getting packages from conda-forge. Also, as those packages are some of your dependencies, maybe being added as maintainers to the would be useful to you. I would be really interested in helping you figure your way around conda-forge. Feel free to give me a ping. 😄

[jakirkham]

@NLeSC @remenska, noticed that you have a variety of interesting packages some present here and some not yet present (though we are eager to add). Given this is quickly becoming the place to get packages that may not yet be packaged by Continuum and we do the builds in automated VMs in very clean environments, I think you might benefit by adding some of your packages here. Also, feel free to sign up for packages that are valuable to your effort. If you need any help figuring out what is going on, please feel free to ping me and I will be happy to get you started. 😄

[danielfrg]

I am not catch up on all this discussion around conda-forge so I am not sure if this is the best place to do it sorry if not. I am very exited to see progress on this, great work!

With the new conda constructor its really easy to make a custom conda distribution with custom packages from a conda channel. I just tested it with a file like this:

name: centonda
version: 1.0.0

channels:
  - http://repo.continuum.io/pkgs/free/
  - https://conda.anaconda.org/conda-forge

specs:
  - python
  - conda
  - anyjson

At the moment you still need http://repo.continuum.io/pkgs/free/ in the channels list to have python and conda but you can see the idea, if these packages are on the conda-forge channel it would be possible to create a distribution with community created packages.

It would also be possible to make that custom distribution point to the conda-forge channel by default. Not as straight forward but possible, see conda/constructor#16.

Just wanted to mention this as a possibility because I haven't seen anybody discuss this option.

[pelson]

Neat idea @danielfrg. I have scripts which already make the self extracting tarballs (such as miniconda is for Linux and OSX) but not a windows installer. 👍

I think this probably deserves its own issue in this repo though. Happy to open it?

if these packages are on the conda-forge channel it would be possible to create a distribution with community created packages.

Why wouldn't they be - this is a community packaging project 😉 😄

[danielfrg]

I opened a new issue in #90 for tracking.

Why wouldn't they be - this is a community packaging project 😉 😄

Definitely! Thats what I meant, a distribution with only community created packages, all open :)

[pelson]

all open :)

To be fair, this repo does now contain all of the anaconda recipes which are in the conda-build form: https://github.com/ContinuumIO/anaconda-recipes

But I still don't know if that is the canonical repository...

[jakirkham]

@pkgw, noticed that you have a variety of interesting packages some present here and some not yet present (though we are eager to add). Given this is quickly becoming the place to get packages that may not yet be packaged by Continuum and we do the builds in automated CI VMs in very clean environments, I think you might benefit by adding some of your packages here. Also, feel free to sign up for packages that are valuable to your effort. If you need any help figuring out what is going on, please feel free to ping me and I will be happy to get you started. 😄