Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document the channel logic #610

Merged
merged 2 commits into from
Jan 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 88 additions & 0 deletions docs/advanced/channel_priority.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
All logic regarding the decision which dependencies can be installed from which channel is done by the instruction we give the solver.

The actual code regarding this is in the [`rattler_solve`](https://github.com/mamba-org/rattler/blob/02e68c9539c6009cc1370fbf46dc69ca5361d12d/crates/rattler_solve/src/resolvo/mod.rs) crate.
This might however be hard to read.
Therefore, this document will continue with simplified flow charts.

# Channel specific dependencies
When a user defines a channel per dependency, the solver needs to know the other channels are unusable for this dependency.
```toml
[project]
channels = ["conda-forge", "my-channel"]

[dependencies]
packgex = { version = "*", channel = "my-channel" }
```
In the `packagex` example, the solver will understand that the package is only available in `my-channel` and will not look for it in `conda-forge`.

The flowchart of the logic that excludes all other channels:

``` mermaid
flowchart TD
A[Start] --> B[Given a Dependency]
B --> C{Channel Specific Dependency?}
C -->|Yes| D[Exclude All Other Channels for This Package]
C -->|No| E{Any Other Dependencies?}
E -->|Yes| B
E -->|No| F[End]
D --> E
```

# Channel priority
Channel priority is dictated by the order in the `project.channels` array, where the first channel is the highest priority.
For instance:
```toml
[project]
channels = ["conda-forge", "my-channel", "your-channel"]
```
If the package is found in `conda-forge` the solver will not look for it in `my-channel` and `your-channel`, because it tells the solver they are excluded.
If the package is not found in `conda-forge` the solver will look for it in `my-channel` and if it **is** found there it will tell the solver to exclude `your-channel` for this package.
This diagram explains the logic:
``` mermaid
flowchart TD
A[Start] --> B[Given a Dependency]
B --> C{Loop Over Channels}
C --> D{Package in This Channel?}
D -->|No| C
D -->|Yes| E{"This the first channel
for this package?"}
E -->|Yes| F[Include Package in Candidates]
E -->|No| G[Exclude Package from Candidates]
F --> H{Any Other Channels?}
G --> H
H -->|Yes| C
H -->|No| I{Any Other Dependencies?}
I -->|No| J[End]
I -->|Yes| B
```

This method ensures the solver only adds a package to the candidates if it's found in the highest priority channel available.
If you have 10 channels and the package is found in the 5th channel it will exclude the next 5 channels from the candidates if they also contain the package.

# Use case: pytorch and nvidia with conda-forge
A common use case is to use `pytorch` with `nvidia` drivers, while also needing the `conda-forge` channel for the main dependencies.
```toml
[project]
channels = ["nvidia/label/cuda-11.8.0", "nvidia", "conda-forge", "pytorch"]
platforms = ["linux-64"]

[dependencies]
cuda = {version = "*", channel="nvidia/label/cuda-11.8.0"}
pytorch = {version = "2.0.1.*", channel="pytorch"}
torchvision = {version = "0.15.2.*", channel="pytorch"}
pytorch-cuda = {version = "11.8.*", channel="pytorch"}
python = "3.10.*"
```
What this will do is get as much as possible from the `nvidia/label/cuda-11.8.0` channel, which is actually only the `cuda` package.

Then it will get all packages from the `nvidia` channel, which is a little more and some packages overlap the `nvidia` and `conda-forge` channel.
Like the `cuda-cudart` package, which will now only be retrieved from the `nvidia` channel because of the priority logic.

Then it will get the packages from the `conda-forge` channel, which is the main channel for the dependencies.

But the user only wants the pytorch packages from the `pytorch` channel, which is why `pytorch` is added last and the dependencies are added as channel specific dependencies.

We don't define the `pytorch` channel before `conda-forge` because we want to get as much as possible from the `conda-forge` as the pytorch channel is not always shipping the best versions of all packages.

For example, it also ships the `ffmpeg` package, but only an old version which doesn't work with the newer pytorch versions.
Thus breaking the installation if we would skip the `conda-forge` channel for `ffmpeg` with the priority logic.
6 changes: 5 additions & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,10 @@ markdown_extensions:
- pymdownx.inlinehilite
- pymdownx.snippets
- pymdownx.details
- pymdownx.superfences
- pymdownx.superfences:
custom_fences:
- name: mermaid
class: mermaid
- pymdownx.tabbed:
alternate_style: true
- toc:
Expand Down Expand Up @@ -77,6 +80,7 @@ nav:
- Tasks: advanced/advanced_tasks.md
- Multi Platform: advanced/multi_platform_configuration.md
- Info command: advanced/explain_info_command.md
- Channel Logic: advanced/channel_priority.md
- Examples:
- C++/Cmake: examples/cpp-sdl.md
- OpenCV: examples/opencv.md
Expand Down
Loading