python3Packages.torch: redundancy and fragility fetching source

[zeuner]

Describe the bug

By now, the recursive Git source has to retrieve 71 repository copies, of which there are e.g. 6 copies of googletest and 4 copies of pybind11. This could not only be considered as containing much redundant data, it is also not very robust considering that it suffices to have fetch failures on one of the 71 repository clones to have the whole source derivation fail. On less robust uplinks, this can make it prohibitively unreliable to retrieve a source tree for building. Appropriate behaviour also in presence of transmission errors should be expected, as fetchurl also handles these cases.

Steps To Reproduce

Steps to reproduce the behavior:

1. Trigger a build from source on an unreliable uplink
2. See it fail when fetching one of the submodules can't be fetched

Expected behavior

Provide a reasonable number of retries in proportion to the number of files to fetch.

Additional context

Obviously, this issue is not directly caused by torch, but since it surfaces much more severely on this package when compared to other packages, I'm reporting it here.

I would suggest splitting the source fetch derivation into non-recursive derivations for the submodules. This would allow to use the specialized fetchFromGitHub and fetchFromGitLab fetchers, and also to continue when a part of the submodules have already been fetched.

Furthermore, doing so could serve as a starting point to explore whether the submodules actually require different versions of dependencies like googletest (they currently point to different commits), or even to replace some of them by their counterparts from nixpkgs.

Notify maintainers

Will do.

Metadata

Please run nix-shell -p nix-info --run "nix-info -m" and paste the result.

[user@system:~]$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"x86_64-linux"`
 - host os: `Linux 5.15.0-91-generic, Ubuntu, 22.04.3 LTS (Jammy Jellyfish), nobuild`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.17.0`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/root/.nix-defexpr/channels/nixpkgs`

---

Add a 👍 reaction to issues you find important.

[zeuner]

@tscholak