This repository has been archived by the owner on Jan 30, 2023. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 1
To do sharding or not #1
Comments
Meta:
|
Another argument for sharding:
git has a build in setting |
Closed
We had a poll to probe how many people relied on the GitHub code navigation feature to find packages. Turns out a lot of people do, which is a strong argument in favor of sharding. |
Depends on #18 |
The general consensus is that we should do sharding. PR #20 changes the draft accordingly. We discussed this a lot in Meeting #18 |
Depends on #17 |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
We had some good discussion about this RFC on Matrix. Mainly around whether sharding (splitting the auto-called directory up based on prefixes) is a good idea. Involved were @alyssais, @roberth, @adisbladis and @infinisil. The strength of arguments were also discussed.
Additional motivations for this RFC in general:
git status
performance (good with less recursive structures)These are additional arguments for sharding:
git
performance, big directories are not great for git trees (can we quantify this? @roberth says that git packfiles can delta encode trees)nix edit .#hello
works in any case, no need to manually type package pathsThese are some additional arguments against sharding:
cd pkgs/auto/hello
orcd pkgs/hello
ifauto
isn't needed anymore, which mirrorspkgs.hello
very wellR
,h
,j
,o
,q
andt
would need special handling (suggested was e.g.pkgs/auto/R/R/default.nix
)Benchmarking:
Sorted 1- and 2-prefix sharding counts by @infinisil: https://gist.github.com/infinisil/0afcae04298390b7d02f91fca4a22219
Python script by @adisbladis with counts based on directory names:
Source
Result (sorted)
About 21.7% of nixpkgs files are inaccessible by clicking on directories in GitHub (@alyssais). Though a lot of these (5456) are in Python which currently isn't in scope for the approach of the RFC
How would sharding be done?
li
(because oflib*
packages) with a bit over 1000 packages. This also is more complicated for handling single-letter packages.Meta:
git
CLI with nixpkgs?". Not "Do you prefer sharding?"The text was updated successfully, but these errors were encountered: