Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support CEP-16 sharded repodata #182

Open
2 tasks done
dholth opened this issue Aug 23, 2024 · 0 comments
Open
2 tasks done

Support CEP-16 sharded repodata #182

dholth opened this issue Aug 23, 2024 · 0 comments
Labels
source::anaconda created by members of Anaconda, Inc. type::feature request for a new feature or capability

Comments

@dholth
Copy link
Contributor

dholth commented Aug 23, 2024

Checklist

  • I added a descriptive title
  • I searched open requests and couldn't find a duplicate

What is the idea?

CEP-16 defines a new repodata format, storing an index and per-project files. This helps with network and disk bandwidth. conda-index should generate this format.

Work has been started in this pull request.

In general, features from https://github.com/dholth/conda-index/blob/sharded-repodata/conda_index/index/shards.py should be merged into the CLI as a --write-shards option, into https://github.com/dholth/conda-index/blob/sharded-repodata/conda_index/index/__init__.py and into https://github.com/dholth/conda-index/blob/sharded-repodata/conda_index/index/sqlitecache.py

We should track the feature's performance over time. So far it takes about the same amount of time to generate shards, versus monolithic repodata.json. We will have to continue to generate both at the same time.

@dholth dholth added the type::feature request for a new feature or capability label Aug 23, 2024
@github-project-automation github-project-automation bot moved this to 🆕 New in 🧭 Planning Aug 23, 2024
@dholth dholth added the source::anaconda created by members of Anaconda, Inc. label Aug 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
source::anaconda created by members of Anaconda, Inc. type::feature request for a new feature or capability
Projects
Status: 🆕 New
Development

No branches or pull requests

1 participant