Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove conda.models.dist.Dist usage #5069

Open
3 tasks done
Tracked by #13192
mbargull opened this issue Nov 13, 2023 · 2 comments · Fixed by #5074
Open
3 tasks done
Tracked by #13192

Remove conda.models.dist.Dist usage #5069

mbargull opened this issue Nov 13, 2023 · 2 comments · Fixed by #5074
Assignees
Labels
epic a highlevel collection of smaller related issues source::contributor created by a frequent contributor type::tech-debt identifies or resolves some technical debt

Comments

@mbargull
Copy link
Member

mbargull commented Nov 13, 2023

Summary

Tracking issue to purge Dist from conda-build.


Motivation

A couple of days ago I got a bit annoyed by conda-build's slow (and memory intensive) ramp up before it gets to its actual first environment solving.

The mayor underlying issue is of course conda's continued use of auxlib entities for everything in the package index.
(auxlib is actually not too bad conceptually -- but it's only viable for a few thousand (top-level) entries; and a conda-forge repodata with linux-64+noarch is past half a million records by now...)
Fixing that (=IMO, move the whole object creation/validation to compiled code) would be a thing I'd welcome much; but it's also a major endeavor to tackle.

An orthogonal thing to do in conda-build is to work on purging legacy behavior/code such that we could at least prepare to make use of the deferred creation of package records.
A first step in that direction would be to finally remove conda-build's usage of the old Dist objects (removed from conda's core workings in 2018).


Impact

  • Prepare usage of newer/more performant package index handling in conda-build.
  • Removal and possible removal tech debt not only in conda-build but also conda.
  • Breaking changes for library uses of conda_build:
    • not only might we remove Dist-related functions which might be used downstream,
    • but also not using a Dist-based package index anymore could break downstream expectations.

Since this introduces these breaking changes, a major version bump seems sensible.


Progress

I started to work in this 2 days ago and made good progress.
Half way through it, I noticed some defunct or unused functions/parts.
Turns out a few of them have been defunct for years, and apparently @kenodegard recently made some preparations to lessen/remove usage of Dist 🎉.
I'll open a draft pull request in the next days.

Tasks

Preview Give feedback
  1. cla-signed locked
    kenodegard
  2. cla-signed locked
    kenodegard
  3. cla-signed locked
    kenodegard
@mbargull mbargull added the epic a highlevel collection of smaller related issues label Nov 13, 2023
@github-project-automation github-project-automation bot moved this to 🆕 New in 🧭 Planning Nov 13, 2023
@mbargull mbargull changed the title Remove conda.dist.Dist usage Remove conda.models.dist.Dist usage Nov 13, 2023
@kenodegard
Copy link
Contributor

kenodegard commented Nov 13, 2023

Xref conda/conda#13192

@kenodegard kenodegard added type::tech-debt identifies or resolves some technical debt source::community catch-all for issues filed by community members labels Nov 13, 2023
@kenodegard kenodegard changed the title Remove conda.models.dist.Dist usage Remove conda.models.dist.Dist usage Nov 13, 2023
@kenodegard
Copy link
Contributor

Since this introduces these breaking changes, a major version bump seems sensible.

FYI #5064

@kenodegard kenodegard moved this from 🆕 New to 🏗️ In Progress in 🧭 Planning Nov 13, 2023
@mbargull mbargull mentioned this issue Nov 15, 2023
3 tasks
@mbargull mbargull added source::contributor created by a frequent contributor and removed source::community catch-all for issues filed by community members labels Nov 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
epic a highlevel collection of smaller related issues source::contributor created by a frequent contributor type::tech-debt identifies or resolves some technical debt
Projects
Status: 🏗️ In Progress
Development

Successfully merging a pull request may close this issue.

2 participants