cargo sometimes fails to run a build.rs that generates the src/lib.rs

[pnkfelix]

This came up in the context of tango, specifically the tango-demo project, which attempts to show off that one can have all checked-files under src/ be markdown source rather than Rust source.

The intention is that the build script will always regenerate the src/lib.rs if it is out of date or has been deleted.

However, this sometimes does not happen. It is hard for me to tell exactly what the scenario is that is causing it to fail.

Here is a sample project that hopefully captures the essence of what I am trying to do:

Two files (only) at outset:

gen-lib/
|
+- Cargo.toml
|
+- build.rs

Cargo.toml:

[package]
name = "gen-lib"
version = "0.1.0"
authors = ["Felix S. Klock II <pnkfelix@pnkfx.org>"]

[dependencies]

[lib]
name = "gen_lib"
path = "src/lib.rs"

build.rs:

use std::fs::{self, File};
use std::io::{self, Write};

fn main() {
    generate_lib_rs().unwrap();
}

fn generate_lib_rs() -> Result<(), io::Error> {
    match fs::create_dir("src") {
        Ok(_) => (), // great, we built it.
        Err(_) => (), // okay, lets assume that was a file-exists error...
    }

    let mut f = File::create("src/lib.rs")?;
    write!(f, "{}", "\
#[cfg(test)]
mod tests {
    #[test]
    fn it_works() {
        assert_eq!(2 + 2, 4);
    }
}
")
}

Here is an example of a troubling interaction. Sometimes cargo build seems to know to run the build script, but other times it gives up as soon as it sees that the src/lib.rs does not exist. And I do not see any hints as to why there is a difference in the --verbose output.

% rm -rf src target build.rs~ Cargo.lock Cargo.toml~
% ls && echo && date && echo && cargo build --verbose -vv && echo && date
build.rs  Cargo.toml

Mon Sep  4 15:44:05 CEST 2017

   Compiling gen-lib v0.1.0 (file:///home/pnkfelix/Dev/Rust/gen-lib)
     Running `rustc --crate-name build_script_build build.rs --crate-type bin --emit=dep-info,link -C debuginfo=2 -C metadata=66f9d79c9066950c -C extra-filename=-66f9d79c9066950c --out-dir /home/pnkfelix/Dev/Rust/gen-lib/\
target/debug/build/gen-lib-66f9d79c9066950c -L dependency=/home/pnkfelix/Dev/Rust/gen-lib/target/debug/deps`
     Running `/home/pnkfelix/Dev/Rust/gen-lib/target/debug/build/gen-lib-66f9d79c9066950c/build-script-build`
     Running `rustc --crate-name gen_lib src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=8fb3f0fed1d71daa -C extra-filename=-8fb3f0fed1d71daa --out-dir /home/pnkfelix/Dev/Rust/gen-lib/target/de\
bug/deps -L dependency=/home/pnkfelix/Dev/Rust/gen-lib/target/debug/deps`
    Finished dev [unoptimized + debuginfo] target(s) in 0.45 secs

Mon Sep  4 15:44:06 CEST 2017
% ls && echo && date && echo && cargo build --verbose -vv && echo && date
build.rs  Cargo.lock  Cargo.toml  src  target

Mon Sep  4 15:44:10 CEST 2017

   Compiling gen-lib v0.1.0 (file:///home/pnkfelix/Dev/Rust/gen-lib)
     Running `/home/pnkfelix/Dev/Rust/gen-lib/target/debug/build/gen-lib-66f9d79c9066950c/build-script-build`
     Running `rustc --crate-name gen_lib src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=8fb3f0fed1d71daa -C extra-filename=-8fb3f0fed1d71daa --out-dir /home/pnkfelix/Dev/Rust/gen-lib/target/de\
bug/deps -L dependency=/home/pnkfelix/Dev/Rust/gen-lib/target/debug/deps`
    Finished dev [unoptimized + debuginfo] target(s) in 0.12 secs

Mon Sep  4 15:44:10 CEST 2017
% rm src/lib.rs # can we recover from removing the generated file first?
% ls && echo && date && echo && cargo build --verbose -vv && echo && date
build.rs  Cargo.lock  Cargo.toml  src  target

Mon Sep  4 15:44:14 CEST 2017

   Compiling gen-lib v0.1.0 (file:///home/pnkfelix/Dev/Rust/gen-lib)
     Running `/home/pnkfelix/Dev/Rust/gen-lib/target/debug/build/gen-lib-66f9d79c9066950c/build-script-build`
     Running `rustc --crate-name gen_lib src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=8fb3f0fed1d71daa -C extra-filename=-8fb3f0fed1d71daa --out-dir /home/pnkfelix/Dev/Rust/gen-lib/target/de\
bug/deps -L dependency=/home/pnkfelix/Dev/Rust/gen-lib/target/debug/deps`
    Finished dev [unoptimized + debuginfo] target(s) in 0.12 secs

Mon Sep  4 15:44:14 CEST 2017
% rm src/lib.rs # We happened to recover above. But does a second attempt fail?
% ls && echo && date && echo && cargo build --verbose -vv && echo && date
build.rs  Cargo.lock  Cargo.toml  src  target

Mon Sep  4 15:44:18 CEST 2017

   Compiling gen-lib v0.1.0 (file:///home/pnkfelix/Dev/Rust/gen-lib)
     Running `rustc --crate-name gen_lib src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=8fb3f0fed1d71daa -C extra-filename=-8fb3f0fed1d71daa --out-dir /home/pnkfelix/Dev/Rust/gen-lib/target/de\
bug/deps -L dependency=/home/pnkfelix/Dev/Rust/gen-lib/target/debug/deps`
error: couldn't read "src/lib.rs": No such file or directory (os error 2)

error: Could not compile `gen-lib`.

Caused by:
  process didn't exit successfully: `rustc --crate-name gen_lib src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=8fb3f0fed1d71daa -C extra-filename=-8fb3f0fed1d71daa --out-dir /home/pnkfelix/Dev\
/Rust/gen-lib/target/debug/deps -L dependency=/home/pnkfelix/Dev/Rust/gen-lib/target/debug/deps` (exit code: 101)
%

[pnkfelix]

(My first guess would be that this bug is somewhat related to #2426 but I have not yet confirmed this...)

[alexcrichton]

Hm I think this may be an unfortunate interaction with assumptions about build scripts today. Right now Cargo assumes that the output of a build script is immutable in the sense that if the inputs haven't changed the output is guaranteed to have not changed either. I believe that means that in this case where src/lib.rs is an "output" of the build script when that changes (aka deleted) Cargo doesn't realize that because it doesn't actually know what the output is and because no inputs changed it assumes nothing happened.

You can perhaps fix this though with some cargo:rerun-if-changed annotations though? I'm not 100% sure that'll fix things, but maybe it'll get closer?

[pnkfelix]

But if that's the case, why does it rebuild some times and not others? (My money's still on time stamp oddities...)

[alexcrichton]

Yeah that's a good point. My guess though is the same, having to do basically with the resolution of the clock.

[pnkfelix]

(For those watching the dialogue...)

@alexcrichton wrote:

You can perhaps fix this though with some cargo:rerun-if-changed annotations though? I'm not 100% sure that'll fix things, but maybe it'll get closer?

Indeed, adding the line:

    println!("cargo:rerun-if-changed=\"src/lib.rs\"");

to the build.rs script does seem to cause the example interaction to work.

(I still do not completely understand the underlying model here, but I just wanted to get that extra data point in there.)

[BruceChapmanNec]

Having a similar problem using built crate https://docs.rs/built/0.3.0/built/
(all that changes are timestamps and git tags - but I want those changes into the generated code).

In my case adding

println!("cargo:rerun-if-changed=\"whateverYouLikeEvenAnAbsentDirectoryOrFile\"");

forced the build script to be run each build.

When Alex said ( #4468 (comment))

Right now Cargo assumes that the output of a build script is immutable in the sense that if the inputs haven't changed the output is guaranteed to have not changed either.

That assumption is not correct in my case. We need a way for the build script to say that the build script should run whether or not its assumed inputs have changed - or a way to specify the inputs (in my case .git directory)

[alexcrichton]

@BruceChapmanNec the rerun-if-changed directive can be used for the contents of .git I believe?

[frehberg]

AFAICS,(please correct me if I am wrong)

if there is a build.rs-script, but the script does not communicate rerun-if-changed back to the cargo process, all file-items (present at startup of the process cargo) are monitored for changes, but not timestamps of directories. (lib.rs did no exist at first startup, and removing a file will only change the timestamp of the remaining directory src/)

This default behavior is disabled, if build.rs communicates at least a single rerun-if-changed back to cargo process, eg. for this specific file "src/lib.rs" and the directory "src/" itself.

I assume the erratic rebuild was caused by this default behavior, probably the timestamp of any other file did change. Also, in larger projects this default behavior will cause kind of monitoring-overhead.

Please see the crate https://crates.io/crates/build-deps

[clearloop]

error: couldn't read build.rs: No such file or directory (os error 2)

error: aborting due to previous error

error: failed to verify package tarball

Same error on cargo publish, even the build.rs doesn't generate src/lib.rs.

---

cargo build works fine.

[rami3l]

Same problem here.

My project pacaptr has a build.rs that just prints a pretty compatibility table to docs/compatibility_table.md.

Don't know if this is counted as a hack or not, but the problem is the same: cargo build works fine, while cargo publish doesn't.

Update: I managed to bypass this problem with macro-generated comments in rustdoc. Thanks @therealbnut!

[therealbnut]

I came across this looking into another issue, in the cargo book it says:

Build scripts may save any output files in the directory specified in the OUT_DIR environment variable. Scripts should not modify any files outside of that directory.

If I'm understanding correctly that might be why you're having this issue. If you want to generate files you might need to put them into the OUT_DIR.

[rami3l]

cc @pnkfelix @clearloop

As @alexcrichton has pointed out, we cannot try to modify anything out of OUT_DIR in the build script.

However, starting from Rust 1.54, it's possible to declare a module as:

// This won't work...
// #[path = concat!(env!("OUT_DIR"), "/generated.rs")]
// mod generated;

// ... but this will.
mod generated {
    include!(concat!(env!("OUT_DIR"), "/generated.rs"));
}

(For a complete example, see here.)

... which makes it possible to leverage build.rs to generate a source file and use it without violating the rules.

Thus, I believe this issue has been resolved.

[jdx]

error: couldn't read build.rs: No such file or directory (os error 2)

error: aborting due to previous error

error: failed to verify package tarball

Same error while cargo publish, even the build.rs doesn't generate src/lib.rs.

cargo build works fine.

I had this issue as well. My problem was that /build.rs was missing from the files. See the fix here: jdx/mise@41160c2

[CobaltCause]

However, starting from Rust 1.54, it's possible to declare a module as:

#[path = concat!(env!("OUT_DIR"), "/generated.rs")]
mod generated;

... which makes it possible to leverage build.rs to generate a source file and use it without violating the rules.

This is actually not possible. Per rust-lang/rust#83366 (comment):

Notably, the attributes path, crate_type, and recursion_limit do not support this because they need their values before or during expansion. It is also not supported with proc-macros.

[rami3l]

@CobaltCause Thanks a lot for pointing it out!

I've changed my (previously misleading) comment accordingly.

[epage]

Considering this is running counter to documented practices and there seem to be ways to make those documented practices work in these situations, I lean towards closing this issue. If there is a reason for us to keep it open, let us know!