Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update cargo builder to fetch registry dynamically. #27964

Closed
wants to merge 1 commit into from
Closed

Update cargo builder to fetch registry dynamically. #27964

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion pkgs/applications/misc/alacritty/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ buildRustPackage rec {
sha256 = "0h5hrb2g0fpc6xn94hmvxjj21cqbj4vgqkznvd64jl84qbyh1xjl";
};

depsSha256 = "1pbb0swgpsbd6x3avxz6fv3q31dg801li47jibz721a4n9c0rssx";
depsSha256 = "broken see https://github.com/alexcrichton/cargo-local-registry/issues/9";

buildInputs = [
cmake
Expand Down
33 changes: 33 additions & 0 deletions pkgs/build-support/rust/cargo-local-registry.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{ fetchurl, stdenv }:

let
version = "0.1.4";
platform =
if stdenv.system == "x86_64-linux"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will also need aarch64. Maybe we can work-around getting cargo build in bootstrap phase without this package and build cargo-local-registry from source.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are a lot of dependencies and other stuff to manage. I don't think I know enough about the rust setup to do it myself but if someone can take it on that would be great. I was focused here on showing that cargo-local-registry is probably the way forward.

However building might not actually be necessary if we can rely on the cached deps pacakge. This package is archetecture independant so once we have them we can just build cargo-local-registry from source. I don't know if there is a policy about relying on the cache for things like this.

Copy link
Member

@Mic92 Mic92 Aug 10, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we can use this one for bootstrapping: #24991
I have not found yet find to the time to inspect it closer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I prefer this approach is it doesn't require any preprocessing or extra files. It simply reads the Cargo.toml file. I do think that those are fixable but this is also a nice quick improvement on the current state.

Copy link
Member

@Mic92 Mic92 Aug 11, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But a solution for aarch64 must be found first. If upstream adds support for this architecture as well, I am ok with that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or we could just build our own boostrap binary 🤷‍. Once the bootstrap is in place we can just grab a couple of versions back.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Mic92 As far as i can tell there is currently no support for rust bootstrap for aarch64 either? There are aarch64 builds available on the official rust installer page. Adding a binaries for aarch64 to upstream cargo-local-registry would be easy tough.

@kevincox I was working on something similar but based on cargo-vendor. Not sure yet which one is better suited but according to the authore (which is the same for both) cargo-vendor is getting more attention at the moment.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kevincox I forget to mention that cargo-vendor might be easier to bootstrap soon as it will soon have a tarball with vendored dependencies available. See: alexcrichton/cargo-vendor#51

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I'm not an expert on the differences but it seems like cargo-local-registry is designed to "mirror" the required dependencies whereas cargo-vendor is supposed to the upstream project to vendor dependencies themselves. Either will probably work in practice but cargo-local-registry looks like a better fit for our use case.

And it seems like the bootstrapping issue shouldn't be a major concern as cargo can cross-compile easily. Someone just needs to but in some effort to bootstrap the first version.

That being said I'm happy for whatever solution works.

then "x86_64-unknown-linux-musl"
else if stdenv.system == "x86_64-darwin"
then "x86_64-apple-darwin"
else throw "missing bootstrap url for platform ${stdenv.system}";

# fetch hashes by patching print-hashes.sh to not use the "$DATE" variable
# then running `print-hashes.sh 1.16.0`
hash =
if stdenv.system == "x86_64-linux"
then "7716524846297abc6e8a7b26c57f56b1e0c2b261a6d0583e78f2a8fd60b33695"
else if stdenv.system == "x86_64-darwin"
then "439ff0303bc17fd8587196f0658ae4430850a906e67648f9fde047c9c7c75998"
else throw "missing bootstrap hash for platform ${stdenv.system}";
in stdenv.mkDerivation {
name = "cargo-local-registry-${version}";

src = fetchurl {
url = "https://github.com/alexcrichton/cargo-local-registry/releases/download/${version}/cargo-local-registry-${version}-${platform}.tar.gz";
sha256 = hash;
};

phases = "unpackPhase installPhase";

installPhase = ''
install -Dm755 cargo-local-registry $out/bin/cargo-local-registry
'';
}
61 changes: 15 additions & 46 deletions pkgs/build-support/rust/default.nix
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,6 @@
{ stdenv, callPackage, path, cacert, git, rust, rustRegistry }:
{ fetchurl, stdenv, callPackage, path, cacert, git, rust }:

let
rustRegistry' = rustRegistry;
in
{ name, depsSha256
, rustRegistry ? rustRegistry'
, src ? null
, srcs ? null
, sourceRoot ? null
Expand All @@ -19,7 +15,7 @@ let
lib = stdenv.lib;

fetchDeps = import ./fetchcargo.nix {
inherit stdenv cacert git rust rustRegistry;
inherit fetchurl stdenv cacert git rust;
};

cargoDeps = fetchDeps {
Expand All @@ -28,7 +24,7 @@ let
};

in stdenv.mkDerivation (args // {
inherit cargoDeps rustRegistry;
inherit cargoDeps;

patchRegistryDeps = ./patch-registry-deps;

Expand All @@ -43,57 +39,30 @@ in stdenv.mkDerivation (args // {
postUnpack = ''
eval "$cargoDepsHook"

echo "Using cargo deps from $cargoDeps"

cp -a "$cargoDeps" deps
chmod +w deps -R

# It's OK to use /dev/null as the URL because by the time we do this, cargo
# won't attempt to update the registry anymore, so the URL is more or less
# irrelevant
mkdir .cargo
cat >.cargo/config <<-EOF
[source.crates-io]
registry = 'https://github.com/rust-lang/crates.io-index'
replace-with = 'local-registry'

cat <<EOF > deps/config
[registry]
index = "file:///dev/null"
[source.local-registry]
local-registry = '$cargoDeps'
EOF

export CARGO_HOME="$(realpath deps)"
export CARGO_HOME="$(pwd)/deps"
export RUST_LOG=${logLevel}
export SSL_CERT_FILE=${cacert}/etc/ssl/certs/ca-bundle.crt

# Let's find out which $indexHash cargo uses for file:///dev/null
(cd $sourceRoot && cargo fetch &>/dev/null) || true
cd deps
indexHash="$(basename $(echo registry/index/*))"

echo "Using indexHash '$indexHash'"

rm -rf -- "registry/cache/$indexHash" \
"registry/index/$indexHash"

mv registry/cache/HASH "registry/cache/$indexHash"

echo "Using rust registry from $rustRegistry"
ln -s "$rustRegistry" "registry/index/$indexHash"

# Retrieved the Cargo.lock file which we saved during the fetch
cd ..
mv deps/Cargo.lock $sourceRoot/

(
cd $sourceRoot

cargo fetch
cargo clean
)
# Unpack crates. This is only needed for $patchRegistryDeps
cargo fetch --frozen --verbose --manifest-path "$sourceRoot/Cargo.toml"
'' + (args.postUnpack or "");

prePatch = ''
# Patch registry dependencies, using the scripts in $patchRegistryDeps
(
set -euo pipefail

cd $NIX_BUILD_TOP/deps/registry/src/*
cd "$CARGO_HOME/registry/src/"*

for script in $patchRegistryDeps/*; do
# Run in a subshell so that directory changes and shell options don't
Expand All @@ -107,7 +76,7 @@ in stdenv.mkDerivation (args // {
buildPhase = with builtins; args.buildPhase or ''
runHook preBuild
echo "Running cargo build --release ${concatStringsSep " " cargoBuildFlags}"
cargo build --release ${concatStringsSep " " cargoBuildFlags}
cargo build --release --frozen ${concatStringsSep " " cargoBuildFlags}
runHook postBuild
'';

Expand Down
209 changes: 0 additions & 209 deletions pkgs/build-support/rust/fetch-cargo-deps
View file
Open in desktop

This file was deleted.

Loading