Add support for managing Conda environments and packages

[matterhorn103]

I'm very excited by uv and the possibilities it hints at for the future of one-stop packaging and environment management.

As for a vast number of scientists, though, conda is pretty much essential to many of my projects. A package manager limited to PyPI may replace pip and venv for pure Python projects for me but will always have to sit alongside conda with conda-forge.

I get that it's still very early days, but are there any ambitions/plans to truly turn Python packaging into a single effective ecosystem and unite the science and programming camps? Might uv in future also manage conda environments or use packages from conda-forge/other repos?

Or should I reign in my excitement a little?

[zanieb]

This might be out of scope for us. I'm not sure about the long-term future, but definitely in the short term. Prefix.dev is more focused on conda integration.

[matterhorn103]

Ok, thanks. I understand, but still a pity. A single Python management tool for both camps would be nice one day.

Hopefully the collaboration between you and the pixi team continues and they become complementary tools following the same standards :)

[forrestfwilliams]

Wanted to echo my support for this question as well. Many AI/ML and scientific workflows rely on Python packages that have complex C/C++ dependencies. GDAL is a good example of a package that is essential for many Python applications, but installing it via Pip requires that you have already installed the GDAL C library your machine.

Conda-based tools generally do a good job of managing both the Python and C/C++ dependencies, which is why many scientists use this ecosystem. Unfortunately, using a package manager that cannot manage these lower-level dependencies is a non-starter for many of us.

Would love to see the uv team try to come up with their own solution for this problem sometime in the future!

[sam-goodwin]

Just ran into this with a requirement to use mambda to use a specific version of tensorflow on a Mac. It would be great if UV could support conda environments.

[den-run-ai]

just had an issue that uv does not work in a conda environment, but installation is ok. uv binary is missing.

[marcelroed]

With the latest stabilizations uv has covered a majority of my needs for a Python packaging solution. However, the needs of Python users in the scientific space go a little further than the stated goal of being to Python what Cargo is to Rust. In scientific Python, environments are expected to include system libraries and other important packages that don't mesh well with PyPI.

Currently my environments require something like micromamba or pixi for system packages and binaries, and then uv to install PyPI packages that either can't be sourced from Conda repos, or are easier to manage in PyPI. For example, I might need to install a modern C++ compiler and some system libraries with conda before using uv to install the rest of my requirements. A key issue with this is that solving is not possible across this boundary, leading to environments that deteriorate over time, and the requirements for an install script that orders installs instead of just a single requirements.txt (or environment.yml) that can be solved.

In short, supplanting pip has been a brilliant beginning for uv to take off, but I can't see a world where scientific Python has its needs met by a solution that can only solve using packages on PyPI. Python is not Rust, and users expect an all-encompassing Python package manager to provide more than what pip currently does.

@zanieb, have there been any developments in thinking about expanding capabilities in this direction, or is this still out of scope? I'm partially served by pixi for my use-cases, but I still find just using uv from the terminal to be superior when dealing with the PyPI side of package management.

Thanks for the great work in this space! uv has been a huge boost to my morale and productivity.

[zanieb]

Thanks for the well written comment.

We care about scientific Python (several of us worked in that domain before Astral), but right now we're focused on polishing the project management experience and working on correctness. Hopefully we'll be able to figure out a good story for Conda integration, but it's not in the immediate scope.

Thanks for the great work in this space! uv has been a huge boost to my morale and productivity.

That's great to hear <3

[shuowpro]

Is it possible to export a Conda environment YAML file? It already supports exporting a requirements.txt file, so adding support for exporting a Conda environment file shouldn't be difficult. As a temporary workaround, we can still export the environment file while using pip as the package manager.

[matterhorn103]

@zanieb related to your response in #6989:

I think it's okay to support if it works, but I'm not sure how well it will translate since Conda has, for example, different names for some packages. We can't actually perform a resolution with Conda.

It has felt to me for a while that the unknown correspondence between PyPI and Conda packages is the biggest thing keeping the ecosystems separate. When looking into the topic many months ago I came across Grayskull, this thread discussing the exact problem, and this bot-generated mapping effort. By no means exhaustive, but I imagine some mapping like that would be a necessary starting point for Conda/PyPI interop, so figured I'd leave the links here.

[rosmur]

@matterhorn103 alternatively, is there any (technical) reason to continue using anything in the conda ecosystem? I've made the decision to never install anaconda or anything related on my new MacBook and im all in on uv - its the future.

[matterhorn103]

Like you, I don't use Conda personally at all any more. But I know that colleagues do and are unlikely to be budged. Part of that is that although PyPI can now, thanks to wheels, match Conda in the historically difficult things like numpy, in my field at least programs are often distributed via conda-forge that have no Python content whatsoever and will therefore never be on PyPI.

[tpanum]

I try to avoid conda in favor of uv whenever possible. However, certain packages have so complex external dependencies its is not always feasible.

[bart-maykin]

There's https://github.com/nextgis/pygdal which installs without problems with pip, but uv pip fails.
pip install pygdal==3.4.1.10 install fine.
uv pip install pygdal==3.4.1.10 gives this output: (the version is needed because of the C library version I have installed)

Resolved 2 packages in 5ms
error: Failed to prepare distributions
  Caused by: Failed to fetch wheel: pygdal==3.4.1.10


[stdout]
running bdist_wheel
running build
running build_py
copying ./osgeo/gnm.py -> build/lib.linux-x86_64-cpython-311/osgeo
copying ./osgeo/gdalnumeric.py -> build/lib.linux-x86_64-cpython-311/osgeo
copying ./osgeo/gdalconst.py -> build/lib.linux-x86_64-cpython-311/osgeo
copying ./osgeo/gdal_array.py -> build/lib.linux-x86_64-cpython-311/osgeo
copying ./osgeo/utils.py -> build/lib.linux-x86_64-cpython-311/osgeo
copying ./osgeo/gdal.py -> build/lib.linux-x86_64-cpython-311/osgeo
copying ./osgeo/osr.py -> build/lib.linux-x86_64-cpython-311/osgeo
copying ./osgeo/__init__.py -> build/lib.linux-x86_64-cpython-311/osgeo
copying ./osgeo/ogr.py -> build/lib.linux-x86_64-cpython-311/osgeo
running egg_info
writing pygdal.egg-info/PKG-INFO
writing dependency_links to pygdal.egg-info/dependency_links.txt
writing requirements to pygdal.egg-info/requires.txt
writing top-level names to pygdal.egg-info/top_level.txt

[stderr]
Traceback (most recent call last):
  File "<string>", line 11, in <module>
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/build_meta.py", line 435, in build_wheel
    return _build(['bdist_wheel'])
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/build_meta.py", line 426, in _build
    return self._build_with_temp_dir(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/build_meta.py", line 407, in _build_with_temp_dir
    self.run_setup()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/build_meta.py", line 522, in run_setup
    super().run_setup(setup_script=setup_script)
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/build_meta.py", line 320, in run_setup
    exec(code, locals())
  File "<string>", line 225, in <module>
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/__init__.py", line 117, in setup
    return distutils.core.setup(**attrs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 183, in setup
    return run_commands(dist)
           ^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 199, in run_commands
    dist.run_commands()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 954, in run_commands
    self.run_command(cmd)
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/dist.py", line 995, in run_command
    super().run_command(command)
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/command/bdist_wheel.py", line 381, in run
    self.run_command("build")
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/dist.py", line 995, in run_command
    super().run_command(command)
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/dist.py", line 995, in run_command
    super().run_command(command)
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/command/build_py.py", line 78, in run
    self.build_package_data()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/command/build_py.py", line 169, in build_package_data
    for target, srcfile in self._get_package_data_output_mapping():
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/command/build_py.py", line 161, in _get_package_data_output_mapping
    for package, src_dir, build_dir, filenames in self.data_files:
                                                  ^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/command/build_py.py", line 87, in __getattr__
    self.data_files = self._get_data_files()
                      ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/command/build_py.py", line 93, in _get_data_files
    self.analyze_manifest()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/command/build_py.py", line 191, in analyze_manifest
    self.run_command('egg_info')
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
    self.distribution.run_command(command)
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/dist.py", line 995, in run_command
    super().run_command(command)
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 973, in run_command
    cmd_obj.run()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 313, in run
    self.find_sources()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 321, in find_sources
    mm.run()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 544, in run
    self.add_defaults()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/command/egg_info.py", line 582, in add_defaults
    sdist.add_defaults(self)
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/command/sdist.py", line 109, in add_defaults
    super().add_defaults()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 238, in add_defaults
    self._add_defaults_ext()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/command/sdist.py", line 322, in _add_defaults_ext
    build_ext = self.get_finalized_command('build_ext')
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 303, in get_finalized_command
    cmd_obj.ensure_finalized()
  File "/home/user/.cache/uv/builds-v0/.tmpkEGGLO/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 111, in ensure_finalized
    self.finalize_options()
  File "<string>", line 129, in finalize_options
  File "<string>", line 30, in get_numpy_include
AttributeError: 'dict' object has no attribute '__NUMPY_SETUP__'. Did you mean: 'get_data_files'?

[zanieb]

@bart-maykin please open a new issue, this does not seem like the right place for that problem.

[tobiasdiez]

The following is a concrete proposal how initial conda support may look like in uv:

• Add support for PEP 725 External dependencies. This allows to specify non-python dependencies in pyproject.toml. For example, scipy may declare

  [external]
  build-requires = [
    "virtual:compiler/c",
    "virtual:compiler/cpp",
    "virtual:compiler/fortran",
    "pkg:generic/ninja",
    "pkg:generic/pkg-config",
  ]
  host-requires = [
    "virtual:interface/blas",
    "virtual:interface/lapack",
  ]

  "Support" here would mean at least that one could add external dependencies via uv add and that uv can read these information from pyproject.toml. This should actually be straight-forward and not too involved.

  At this point, the pyproject.toml file contains all the information to create a conda environment, including python and non-python dependencies.

• Render these information as a conda environment file. For this, the python dependencies and the external dependencies need to be translated into the correct conda packages. As @matterhorn103 mentioned above, grayskull maintains such a mapping that could be reused for this purpose. Then the user can use the generated conda environments files via mamba/conda.

• Bonus: Streamline the process into one command. For example, uv sync in an activated conda environment should also update the non-python dependencies.

• End goal: Use https://github.com/conda/rattler to directly and transparently solve the conda environment, bypassing the export step completely.

In sagemath/sage#37447, I've implemented this strategy in a custom python script (not using uv at all). It works quite well.

[xinyu-dev]

I also support this. There are some biology packages that are only conda installable.

[hoaihuongbk]

+1000 for this support. This simplifies the operational of managing the python dependencies on a large scale.

[fepegar]

Is it possible to export a Conda environment YAML file? It already supports exporting a requirements.txt file, so adding support for exporting a Conda environment file shouldn't be difficult. As a temporary workaround, we can still export the environment file while using pip as the package manager.

@shuowpro, I've written uv2conda for that. You might find it useful.

[tupui]

Adding my upvote here. We are adding some recommendations to uv in SciPy's doc but we also need to still recommend things like pixi until we have a solution for Conda.

@charliermarsh I believe support for non Python deps is around the corner?

[zanieb]

@tupui Happy to review those SciPy changes, if you want to ping me.

I don't think support for non-Python dependencies is around the corner though :)

[binbjz]

I've been using pyenv to manage python versions and miniconda and anaconda versions. I really need support for anaconda and miniconda, this is so critical.

[notatallshaw]

I think Astral are aware people would really like this feature, I don't think there's need for any more "me too" comments, add a thumbs up to the original post. Unless you have some novel use case not already described here.

Also, for those who really need conda packages and/or environments, and are not already aware, pixi provides project management, like uv, and does both conda and standard Python packages: https://github.com/prefix-dev/pixi

[matterhorn103]

Thought I'd just drop a link here to raise awareness of a PR I have open over at the conda repo which would add support for creating a conda environment from a specification in a pyproject.toml file.

I like to think that it would be a good first step towards better PyPI-conda interop and would help a little in enabling projects like uv to achieve this feature in future.

[tupui]

Thought I'd just drop a link here to raise awareness of a PR I have open over at the conda repo which would add support for creating a conda environment from a specification in a pyproject.toml file.

I like to think that it would be a good first step towards better PyPI-conda interop and would help a little in enabling projects like uv to achieve this feature in future.

Yeah that might be the most sensible way forward tbh. I hate to have a specific file for the conda env while the whole ecosystem of tools transitioned from having their custom file to using the pyproject. Everything conda could very well be defined there.