capture_lockfile offline if available #178

Eh2406 · 2018-01-20T22:30:26Z

This is a start on fixing #96

I could use some advice on how to update the registry, before capture_lockfile is called.

Mark-Simulacrum · 2018-01-20T22:35:28Z

We could probably run cargo search serde or something like that to get the registry to update.

Eh2406 · 2018-01-20T22:37:18Z

Perfect! Will do momentarily.

Mark-Simulacrum · 2018-01-20T22:44:56Z

src/ex.rs

@@ -457,9 +459,13 @@ fn capture_lockfile(
    path: &Path,
    toolchain: &Toolchain,
 ) -> Result<()> {
-    let args = &["generate-lockfile", "--manifest-path", "Cargo.toml"];
+    let args = &["generate-lockfile", "--manifest-path", "Cargo.toml", "-Z", "offline"];


I expect we'll want to store whether -Zoffline works somewhere -- probably in toolchain; we could make it "probably work" for CI and starts_with("nightly") for Toolchain::Dist. That way we avoid the somewhat expensive failure here (docker container creation, primarily).

That makes sense, I assumed run_cargo was a low cost command. If it is not, than how best to test if cargo has -Z offline and where to sotor that knowledge?

Well, "low cost" to an extent -- it creates a docker container. So the ~500 milliseconds or so (no idea, just assumption) that this will waste is 1.4 hours of runtime over the 10,000 crates that we have (worst case).

I think the best thing to do is to use a heuristic as I said above; CI will probably work, nightly should, and beta/stable will not. I'd add a function to Toolchain that returns a bool (something like fn use_unstable_features(&self) -> bool).

Does that help?

aidanhs · 2018-01-21T02:10:54Z

Looks like master of rust-lang has got -Z offline now - it was merged into master of cargo in rust-lang/cargo@3bb0536 (11 days ago), and the latest cargo is master of rust is rust-lang/cargo@6a8eb71 (7 days ago). However, we do need to support stable for beta runs...

Maybe we could pick a particular crate to probe against for each toolchain and if it exits successfully then we know we can use -Z offline - I would prefer not to be trying and falling back for every single crate if heuristics fail. Maybe lazy_static? https://github.com/rust-lang-nursery/crater/blob/368210c/src/ex.rs#L90-L109

One question: will cargo generate a different lockfile with -Z offline than if that flag was not set? I don't know if generate-lockfile -Z offline considers what's in the cache before generating (I sort of hope it doesn't, as that's the ideal behavior here, but also hope it does, because that's better for my personal use).

Mark-Simulacrum · 2018-01-21T02:13:43Z

A single crate to try against seems fine. I'm not sure where we'd store the result though.. we could add a RefCell to toolchain, I guess.

Eh2406 · 2018-01-22T02:02:14Z

Storing the probes result in a local variable in capture_lockfiles then passing it as an arg to capture_lockfile looks like it will work. I just pushed, let me know how it looks.

Leaving me with 2 questions:

What path can I pass to run_cargo to get lazy_static's Cargo.toml?
More generally, If we update the registry in the docker container created by one call to run_cargo will it be updated in the docker container created by subsequent calls?

Mark-Simulacrum · 2018-01-22T02:08:19Z

src/ex.rs

+    // This nop cargo command is to update the registry
+    // so we don't have to do it for each crate.
+    let args = &["cargo", "search", "lazy_static"];
+    let _ = toolchain.run_cargo(&ex.name, Path::new("."), args, CargoState::Unlocked);


I would probably do something like ex.crates().into_iter().find(...) to get the crate, then call with_work_crate like we do in the main loop.

We should assert that this is successful. If it wasn't then the index may not have updated...

added a ? instead of let _ = . so now to get ci green.

Mark-Simulacrum · 2018-01-22T02:09:18Z

More generally, If we update the registry in the docker container created by one call to run_cargo will it be updated in the docker container created by subsequent calls?

I would hope the answer is yes. I'm not certain, though -- maybe worth testing or trying to figure it out based on the arguments to Cargo?

Mark-Simulacrum · 2018-01-22T03:01:17Z

src/ex.rs

+                false
+            }
+        })
+        .unwrap();


Could you use expect here? I believe we compile with debug info enabled, but either way, that'll be cleaner than having to hunt this down later...

Will do, just sparring with ci. I wish I could build locally. Thank you for the ci.

Mark-Simulacrum · 2018-01-22T03:02:31Z

src/ex.rs

+    // This nop cargo command is to update the registry
+    // so we don't have to do it for each crate.
+    let args = &["cargo", "search", "lazy_static"];
+    let _ = toolchain.run_cargo(&ex.name, Path::new("."), args, CargoState::Unlocked);


We should assert that this is successful. If it wasn't then the index may not have updated...

Eh2406 · 2018-01-22T04:24:17Z

Ok, ci is green! And, I tried to include your suggestions. How does this look?

Mark-Simulacrum · 2018-01-22T19:52:37Z

Are you on Windows and as such can't build locally? We should probably fix that... cargo check should nearly universally work, I would hope. Maybe cargo check --target x86_64-unknown-linux-gnu?

Mark-Simulacrum · 2018-01-22T19:53:37Z

Changes look good to me. @aidanhs should look too, though. We can try these next crater run.

Eh2406 · 2018-01-22T20:39:37Z

Ya I am a windows based lifeform, and cargo check did not work. cargo check --target x86_64-unknown-linux-gnu gives:

...
   Compiling clap v2.29.2
   Compiling lzma-sys v0.1.9
   Compiling tempdir v0.3.5
   Compiling uuid v0.5.1
   Compiling rustc_version v0.2.1
error: failed to run custom build command for `backtrace-sys v0.1.16`
process didn't exit successfully: `C:\MyProjects\crater\target\debug\build\backtrace-sys-4fc6aa82abf3bb67\build-script-build` (exit code: 101)
--- stdout
OPT_LEVEL = Some("0")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-pc-windows-msvc")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-pc-windows-msvc")
CC_x86_64-unknown-linux-gnu = None
CC_x86_64_unknown_linux_gnu = None
TARGET_CC = None
CC = None
HOST = Some("x86_64-pc-windows-msvc")
CROSS_COMPILE = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-pc-windows-msvc")
CFLAGS_x86_64-unknown-linux-gnu = None
CFLAGS_x86_64_unknown_linux_gnu = None
TARGET_CFLAGS = None
CFLAGS = None
DEBUG = Some("true")
running: "sh" "C:/Users/finkelman/.cargo/registry/src/github.com-1ecc6299db9ec823/backtrace-sys-0.1.16/src/libbacktrace/configure" "--with-pic" "--disable-multilib" "--disable-shared" "--disable-host-shared" "--host=x86_64-unknown-linux-gnu"

--- stderr
thread 'main' panicked at '

failed to execute command: The system cannot find the file specified. (os error 2)
Is `sh` not installed?

', C:\Users\finkelman\.cargo\registry\src\github.com-1ecc6299db9ec823\backtrace-sys-0.1.16\build.rs:168:12
note: Run with `RUST_BACKTRACE=1` for a backtrace.

warning: build failed, waiting for other jobs to finish...
error: failed to run custom build command for `libz-sys v1.0.18`
process didn't exit successfully: `C:\Users\finkelman\Documents\MyProjects\crater\target\debug\build\libz-sys-8c171b473aa2f5a4\build-script-build` (exit code: 1)
--- stdout
OPT_LEVEL = Some("0")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-pc-windows-msvc")
TARGET = Some("x86_64-unknown-linux-gnu")
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-pc-windows-msvc")
CC_x86_64-unknown-linux-gnu = None
CC_x86_64_unknown_linux_gnu = None
TARGET_CC = None
CC = None
HOST = Some("x86_64-pc-windows-msvc")
CROSS_COMPILE = None
TARGET = Some("x86_64-unknown-linux-gnu")
HOST = Some("x86_64-pc-windows-msvc")
CFLAGS_x86_64-unknown-linux-gnu = None
CFLAGS_x86_64_unknown_linux_gnu = None
TARGET_CFLAGS = None
CFLAGS = None
DEBUG = Some("true")
running: "./configure" "--prefix=C:\\Users\\finkelman\\Documents\\MyProjects\\crater\\target\\x86_64-unknown-linux-gnu\\debug\\build\\libz-sys-4140099cca4c2b5b\\out"


failed to execute command: The system cannot find the file specified. (os error 2)
Is `sh` not installed?



warning: build failed, waiting for other jobs to finish...
error: build failed

aidanhs · 2018-01-22T21:23:47Z

Unfortunately this PR isn't going to work. I decided to look into my concerns above:

One question: will cargo generate a different lockfile with -Z offline than if that flag was not set? I don't know if generate-lockfile -Z offline considers what's in the cache before generating (I sort of hope it doesn't, as that's the ideal behavior here, but also hope it does, because that's better for my personal use).

cargo generate-lockfile -Z offline does consider the cache before generating, which means if you have an foo = "0.1" dependency with no versions of "foo" cached, it will fail to generate the lockfile. Sadly this means it's not suitable for crater purposes. The only way I can think of this working is if we do the original approach of falling back to -Z offline...in which case we're not really gaining much on speed, and we're not testing the latest version of each dependency in cases where -Z offline does work.

What we really want is a --assume-registry-is-up-to-date flag to generate-lockfile, but I can't think of any reason why anybody except crater would need this flag (though...cargo metadata generates a lockfile and I can imagine wanting to query information about a fully resolved dep graph without it updating the registry every time). One option is to cache all of crates.io and use -Z offline then, but git repos are still an issue.

Maybe worth raising an issue on the cargo repo to see what they think.

Mark-Simulacrum · 2018-01-22T23:24:27Z

Hm. I wonder if at this point it might be worth directing calling into to Cargo's APIs. Presumably we could pass the right mix of options somewhere to make this feasible.... I'd rather not, though, if we can avoid it.

Eh2406 · 2018-01-31T22:30:52Z

My pr to add a no-index-update was just accepted, so I updated this to use it.

aidanhs · 2018-02-03T18:11:31Z

This looks good to me! When we're next able to get a crater run in with these changes we can merge.

aidanhs · 2018-02-03T18:46:49Z

(specifically, the crater run should include this commit of cargo in the toolchains)

aidanhs · 2018-02-14T00:35:57Z

Status: still waiting on a version of rustc with a recent enough cargo

Eh2406 · 2018-03-18T23:57:59Z

-Z no-index-update now works on nightly

Mark-Simulacrum · 2018-04-13T03:28:27Z

I believe that stable 1.25.0 supports -Zno-index-update though we won't be able to use it since it requires Cargo built with dev/nightly channel (unless we use hacks, which... we might want to do?). I don't know how @alexcrichton feels about using Cargo's environment variable to force it to think we are on nightly. I'm somewhat on the side of not doing it but don't feel too strongly.

alexcrichton · 2018-04-13T14:20:56Z

@Mark-Simulacrum it's probably fine for crater's usage, we manage it all anyway :)

Eh2406 · 2018-05-08T16:50:47Z

What is the status on this?

pietroalbini · 2018-05-19T21:55:09Z

We should probably go ahead and implement this with the environment variable to override the cargo channel. Registry updates are starting to become a (small) bottleneck with run-graph.

I looked at the code and it's good for the legacy workflow (with some parts of it adapted to the recent refactorings), but it needs to be reworked a bit to be used within run-graph: due to how run-graph works, you can't run generate-lockfile on the lazy_static crate before starting the run, because it won't be available locally (its own prepare stage will be executed later). Using cargo search as suggested before should work fine though.

Eh2406 · 2018-05-19T23:23:37Z

It's been a long time since I had this loaded into my memory. Based on the merge conflicts it's going to take a fair bit of reworking. Without some hand-holding it's going to be awhile before I have the spare Cycles.

Where, given the new graph algorithms, should we check to see if cargo supports this feature? Or at this point should we just assume that all cargos do? And, where should we make sure the registry is up-to-date.

What is the magic environment variable and what does it need to be set to? Where's the best place in the code is to set it?

Where did the invitation of cargo get moved to?

These are all straightforward questions I could research and answer myself, I'm not sure when I will have time. Someone who knows the answer could give me a cheat sheet that would save time, otherwise I will get to it eventually.

pietroalbini · 2018-05-20T07:56:30Z

Where, given the new graph algorithms, should we check to see if cargo supports this feature? Or at this point should we just assume that all cargos do?

I'd say all of the builds we're going to use in the future will have the feature, so it's not a problem.

And, where should we make sure the registry is up-to-date.

Possibly as soon as the graph runner starts.

What is the magic environment variable and what does it need to be set to? Where's the best place in the code is to set it?

The env var is __CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS=nightly. To set it, I'd add an enable_unstable_cargo_features bool in the docker env struct, and add the var to the list if that bool is true. Then, you can set the bool to true in run_cargo, if -Z is in the args list.

Where did the invitation of cargo get moved to?

It's still (almost) in the same place, the only difference is the lockfile generation is done separately for each crate. I'd wait for #234 to be merged first, it contains another bit of refactoring of that part of code.

If you have any other question please ping me here or on IRC.

Eh2406 · 2018-05-20T14:25:00Z

Thank you for all the pointers! I will wait for #234, then attempt to update this code.

If we make the environment variable optional, then we need to add back in code to determine whether to use the unstable argument. But I guess it's not complicated, just whether that environment variable was set. ( this paragraph was going to be a question but I think I answered it.)

pietroalbini · 2018-07-09T09:46:20Z

This is not blocked anymore.

Eh2406 · 2018-07-09T16:29:24Z

Rebuilt! One question; Dose prep_offline_registry need to use docker::run instead of RunCommand?

pietroalbini · 2018-07-11T09:35:35Z

Rebuilt! One question; Dose prep_offline_registry need to use docker::run instead of RunCommand?

Nah, there is no need to sandbox this command. To be fair, even capture_lockfile could be run outside of Docker, since it doesn't execute user code.

pietroalbini · 2018-07-11T09:35:52Z

Thanks!

Mark-Simulacrum reviewed Jan 20, 2018

View reviewed changes

Mark-Simulacrum reviewed Jan 22, 2018

View reviewed changes

Eh2406 mentioned this pull request Jan 22, 2018

generate-lockfiles without updating registry rust-lang/cargo#3479

Closed

Eh2406 mentioned this pull request Jan 31, 2018

Please provide a subcommand to refresh the crates.io index rust-lang/cargo#3377

Closed

Eh2406 added 3 commits July 9, 2018 13:21

capture_lockfile with -Zno-index-update

6863d5d

only use the override when really needed

4e1fe3e

make ci happy

b7703b3

pietroalbini merged commit 7a9ef08 into rust-lang:master Jul 11, 2018

pietroalbini mentioned this pull request Sep 5, 2018

Enable yanked dependencies with an unstable flag rust-lang/cargo#5967

Closed

capture_lockfile offline if available #178

capture_lockfile offline if available #178

Conversation

Eh2406 commented Jan 20, 2018

Mark-Simulacrum commented Jan 20, 2018

Eh2406 commented Jan 20, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aidanhs commented Jan 21, 2018

Mark-Simulacrum commented Jan 21, 2018

Eh2406 commented Jan 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Mark-Simulacrum commented Jan 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eh2406 commented Jan 22, 2018

Mark-Simulacrum commented Jan 22, 2018

Mark-Simulacrum commented Jan 22, 2018

Eh2406 commented Jan 22, 2018

aidanhs commented Jan 22, 2018

Mark-Simulacrum commented Jan 22, 2018

Eh2406 commented Jan 31, 2018

aidanhs commented Feb 3, 2018

aidanhs commented Feb 3, 2018

aidanhs commented Feb 14, 2018

Eh2406 commented Mar 18, 2018

Mark-Simulacrum commented Apr 13, 2018

alexcrichton commented Apr 13, 2018

Eh2406 commented May 8, 2018

pietroalbini commented May 19, 2018

Eh2406 commented May 19, 2018

pietroalbini commented May 20, 2018

Eh2406 commented May 20, 2018

pietroalbini commented Jul 9, 2018

Eh2406 commented Jul 9, 2018 • edited Loading

pietroalbini commented Jul 11, 2018

pietroalbini commented Jul 11, 2018

Eh2406 commented Jul 9, 2018 •

edited

Loading