Resurrecting libconda for better downstream stories

[jaimergp]

Checklist

• I added a descriptive title
• I searched open requests and couldn't find a duplicate

What is the idea?

With the introduction of CalVer based releases, there was a discussion about API guarantees for projects that depend on conda as a library, not as a CLI tool. While the deprecation cycle can be used as an approximation of API changes (6 months or so) for dependency pinning, it's not an ideal solution.

One potential idea was to create a library libconda to encapsulate most of the objects that might be required downstream. This is not a new idea, though, since libconda already existed for conda v4.0:

• Archived repo
• Packages at free (old defaults)
• Packages at conda-forge

However, it was slowly abandoned. libconda was never a dependency of conda and it had to manually kept in sync (source).

What I am suggesting now is to recover this idea but, this time, use it within conda! Thanks to the vendoring mechanism implemented à la pip, conda can vendor libconda as it does with other libraries. Projects will be able to pin libconda as a dependency more freely since it won't affect conda itself.

For this to make sense maintenance-wise, libconda would need to follow semver. It's ok if it breaks often and as needed by conda, since now downstream can pin to whatever version is needed (or even vendor, like conda-lock does), but it needs to follow the semver rules.

I am not sure what would go into libconda, but the previous version had basically everything but the CLI. I'd start with:

• conda.base.context
• conda.models
• conda.core
• ... and their dependencies

This might mean "almost everything but the CLI (and shell stuff?)" again. This would suggest that most of the conda/conda repo would indeed become conda/libconda. In that case it might be easier to extract the non-libconda parts to a conda/conda-app repo or something, but that's a decision about an implementation detail that can wait.

Why is this needed?

Better maintenance story downstream. Note this includes both community projects (mamba, conda-lock, grayskull) but also Anaconda projects (conda-build, constructor), to name a few!

What should happen?

A project should be able to depend on libconda in a API-predictable way or, at least, in a way that doesn't restrict conda's release cycle.

Additional Context

• bash: __add_sys_prefix_to_path: command not found... when running any mamba command mamba-org/mamba#1950
• Remove conda SemVer checks conda-forge/conda-smithy#1678
• If conda is installed in a non-base environment, Windows CMD will treat that environment it as base once active #11519 (example of conda dep issues outside base)
• Split out MatchSpec into a new library conda-matchspec #11234

Tagging people who were part of the discussion or might have a say as maintainers of downstream tools that depend on conda: @beckermr @wolfv @marcelotrevisani @mariusvniekerk

[beckermr]

Thanks @jaimergp! I have one question at a technical level. Is most of the technical debt with respect to deprecations in the CLI sections of the code or in the parts of the code that would go into libconda? I ask because if we move most of the code that has pending deprecations back to SemVer, then we've sort of gone full circle.

[jaimergp]

I'd say there's technical debt all over the place 😬

[jezdez]

For the record, conda releases were not SemVer (e.g. see number of 4.x releases and the fabled 5.0). Or at least for quite a while from what I can read in the release notes. This would have implied an actual deprecation policy, which was not clearly spelled out (AFAIK) until very recently (currently being voted on in conda/ceps#27!).

Also, while I wished we would have had this discussion while reviewing and voting on CEP 8, I'm glad for bringing this up, @jaimergp and @wolfv!

I'm yet to see a clearly spelled out reasoning why the CalVer releases can't be used as a basis for 3rd party tools. I'm inviting you to read other people's opinion on why SemVer will not save you from Hyrum's Law.

Besides the abstract discussion, there is an added complication: With the work on plugin hooks, conda would be tricky to decompose in a way that allows plugins to exist at the same time next to libconda. (E.g. where would the plugin hooks live, in libconda, or conda?)

Both strategies are sensible on their own, but follow different goals:

• plugins: to enable shipping additional and/or extending code via 3rd party packages
• libconda: to enable 3rd party projects to reuse core functionality from conda

We've known about plugin use cases for a while and have received many requests over the years, which triggered the work on them (now being in phase 2).

For a reusable version of conda, I don't think I've seen a list of use cases though, which is why I'd suggest to NOT start from the past libconda (which seemed to have taken a broad approach). But instead start collecting actual use cases we need today and the future, before we overcommit to decomposition into "library" and "UI" just for simplicity's sake.

[jezdez]

This is somewhat related to #11234 (adding to the context above)

[wolfv]

I think if the deprecation cycle is properly followed, pinning ~6 months into the future might work well enough for mamba's use case. If there is a plugin API defined, then that is probably going to have a positive impact on conda's "stability" and deprecation cycles.

So I was actually somewhat convinced that CalVer might work just as well as SemVer in this case.

[dholth]

Nice idea. Getting there (libmamba) from here is the question, we don't have a superb set of objects to expose to the outside world the way that conda is factored now.

[jezdez]

I'm putting this into the backlog to have a larger discussion around this at some point.

[github-actions]

Hi there, thank you for your contribution!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this issue to remain open please:

1. Verify that you can still reproduce the issue at hand
2. Comment that the issue is still reproducible and include:
   - What OS and version you reproduced the issue on
   - What steps you followed to reproduce the issue

NOTE: If this issue was closed prematurely, please leave a comment.

Thanks!

[jezdez]

not stale

[jaimergp]

Continues at #13789, where we can focus on the implementation details.