Building with local dependencies #388

Piripant · 2020-03-10T21:23:06Z

It seems that when building with a local dependency cross fails to load the source for that local dependency. If this is my Cargo.toml:

[dependencies]
nasatopo = { path = "../nasatopo" }

The output of cross build is:

error: failed to load source for a dependency on `nasatopo`

Caused by:
  Unable to update /nasatopo

Caused by:
  failed to read `/nasatopo/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

Which is technically correct as the is not /nasatopo, but it the Cargo.toml I specified a local path, not a global one, as cross seems to understand.
Furthermore if I try to replace the relative path with a global one, the same error still happens, this time pointing to a path which is actually valid and takes to the location of the nasatopo source folder.

I think this has to do with how a docker container interfaces with the file-system, but I am not familiar and cannot tell.

The text was updated successfully, but these errors were encountered:

velvia · 2020-03-19T23:18:44Z

This is hitting me as well. I even tried symlink but it doesn't work either.

zekronium · 2020-04-01T15:17:58Z

Cross only mounts the cargo project.

I had this issue as well and resolved it by moving the dependency into the project dir.

The path then has to be changed to
nasatopo = { path = "nasatopo" }

reitermarkus · 2020-04-07T12:18:53Z

Yeah, we would need to parse the Cargo.toml and find all path dependencies, then mount them inside the container.

kornelski · 2020-05-29T11:28:08Z

Could you mount root of the workspace?

I'm trying to build one of crates in a workspace, but running into the same problem - that crate gets moved out of its workspace.

brahmlower · 2020-10-31T22:53:44Z

I'm bumping into this as well, but I was hoping to solve it using the volume mount feature. I tried adding a Cross.toml file with the following content:

[build.env]
volumes = [
    "MY_DEP",
]

In my Cargo.toml I have:

[dependencies]
my_dep = { path = "../my-dep" }

However when I run cross with that environment variable set, it fails with the same error reported by @Piripant. Verbose output shows that the volume mount isn't being added:

$ MY_DEP=../my-dep cross build --target x86_64-unknown-linux-gnu --verbose
+ "rustc" "--print" "sysroot"
+ "rustup" "toolchain" "list"
+ "rustup" "target" "list" "--toolchain" "stable-x86_64-unknown-linux-gnu"
+ "rustup" "component" "list" "--toolchain" "stable-x86_64-unknown-linux-gnu"
+ "/usr/local/bin/docker" "run" "--userns" "host" "-e" "PKG_CONFIG_ALLOW_CROSS=1" "--rm" "--user" "501:20" "-e" "XARGO_HOME=/xargo" "-e" "CARGO_HOME=/cargo" "-e" "CARGO_TARGET_DIR=/target" "-e" "USER=me" "-e" "CROSS_RUNNER=" "-v" "/Users/me/.xargo:/xargo:Z" "-v" "/Users/me/.cargo:/cargo:Z" "-v" "/cargo/bin" "-v" "/Users/me/development/my-project/backend-api:/project:Z" "-v" "/Users/me/.rustup/toolchains/stable-x86_64-unknown-linux-gnu:/rust:Z,ro" "-v" "/Users/me/development/my-project/backend-api/target:/target:Z" "-w" "/project" "-i" "-t" "rustembedded/cross:x86_64-unknown-linux-gnu-0.2.1" "sh" "-c" "PATH=$PATH:/rust/bin cargo build --target x86_64-unknown-linux-gnu --verbose"
error: failed to get `my_dep` as a dependency of package `backend_api v0.1.0 (/project)`

Caused by:
  failed to load source for dependency `my_dep`

Caused by:
  Unable to update /my-dep

Caused by:
  failed to read `/my-dep/Cargo.toml`

Caused by:
  No such file or directory (os error 2)

Am I using the volume feature wrong here? As far as I can tell, I'm using it as described by the documentation.

Beyond my issue of using the volume feature, it looks like the volumes don't let you change where the path gets mounted in the container. The code for this is here:

if let Ok(val) = env::var(var) {
    let host_path = Path::new(&val).canonicalize()?;
    let mount_path = &host_path;
    docker.args(&["-v", &format!("{}:{}", host_path.display(), mount_path.display())]);
    docker.args(&["-e", &format!("{}={}", var, mount_path.display())]);
}

It looks like the path in the container will be the same as the canonical path on the host, which obviously won't work for us. For example, my attempted work around would result in the docker flag -v /Users/me/development/my-project/my-dep:/Users/me/development/my-project/my-dep, but we know cargo will be looking for the dependency at /my-dep.

We could expand the volume feature to respect host/container volume paths such that the environment variable MY_DEP=../my-dep:/my-dep would translate to the docker flag -v /Users/me/development/my-project/my-dep:/my-dep. @reitermarkus and @malbarbo Does that sound like a change you'd be comfortable with? If so, then I can get a PR put together for this.

wngr · 2020-12-02T10:40:10Z

The other possibility would be to run cross from the root dir, but be be able to give an additional working dir parameter (like docker -w).

Edit: https://github.com/rust-embedded/cross/pull/438/files solves the issue for me.

…ker root This commits adds support for parsing the `--manifest-path` option to cross. So far, the `Cargo.toml` manifest of the crate (or its Cargo workspace) to compile has been assumed to be in the current working directory. This means, that relative crate dependencies were not supported, because those paths were not mapped into the docker container. Take the following example structure, where `my-bin` depends on `my-lib`: . ├── my-bin │ ├── Cargo.toml │ └── src │ └── main.rs └── my-lib ├── Cargo.toml └── src └── lib.rs This commits enables such scenarios, by running cross from `.` like so: `cross build --manifest-path=my-lib/Cargo.toml --target x86_64-pc-windows-gnu`, as `.` is mapped as the container's root, and the options passed through to Cargo. Related cross-rs#388 cross-rs#139 cross-rs#277 cross-rs#78 Co-authored-by: Kviring Alexey <alex@kviring.com>

TarekkMA · 2021-03-21T10:24:59Z

@bplower I did face issues with the volume feature as well. But I realized the code you linked is newer than the latest release.

I installed cross from git directly and the option worked
cargo install cross --git https://github.com/rust-embedded/cross/

Edit: I did add a PR to add the option to specify the mount path #540

…ker root This commits adds support for parsing the `--manifest-path` option to cross. So far, the `Cargo.toml` manifest of the crate (or its Cargo workspace) to compile has been assumed to be in the current working directory. This means, that relative crate dependencies were not supported, because those paths were not mapped into the docker container. Take the following example structure, where `my-bin` depends on `my-lib`: . ├── my-bin │ ├── Cargo.toml │ └── src │ └── main.rs └── my-lib ├── Cargo.toml └── src └── lib.rs This commits enables such scenarios, by running cross from `.` like so: `cross build --manifest-path=my-lib/Cargo.toml --target x86_64-pc-windows-gnu`, as `.` is mapped as the container's root, and the options passed through to Cargo. Related cross-rs#388 cross-rs#139 cross-rs#277 cross-rs#78 Co-authored-by: Kviring Alexey <alex@kviring.com>

root This commits adds support for parsing the `--manifest-path` option to cross. So far, the `Cargo.toml` manifest of the crate (or its Cargo workspace) to compile has been assumed to be in the current working directory. This means, that relative crate dependencies were not supported, because those paths were not mapped into the docker container. Take the following example structure, where `my-bin` depends on `my-lib`: . ├── my-bin │ ├── Cargo.toml │ └── src │ └── main.rs └── my-lib ├── Cargo.toml └── src └── lib.rs This commits enables such scenarios, by running cross from `.` like so: `cross build --manifest-path=my-lib/Cargo.toml --target x86_64-pc-windows-gnu`, as `.` is mapped as the container's root, and the options passed through to Cargo. Related cross-rs#388 cross-rs#139 cross-rs#277 cross-rs#78 Co-authored-by: Kviring Alexey <alex@kviring.com>

mkhattab · 2022-03-23T22:01:10Z

I think have a work around for this that perhaps may help someone. The cross command accepts a DOCKER_OPTS environment variable, so you can use that to mount volumes into the container.

For example, assuming you have a workspace like the following:

.
├── Cargo.toml
├── foo                       # `foo` has a dependency on `../bar`
│   └── Cargo.toml
└── bar
    └── Cargo.toml

# in foo/
DOCKER_OPTS="-v $(realpath ..)/bar:/bar" cross build --target x86_64-unknown-linux-gnu

On macOS the above command works for me.

root This commits adds support for parsing the `--manifest-path` option to cross. So far, the `Cargo.toml` manifest of the crate (or its Cargo workspace) to compile has been assumed to be in the current working directory. This means, that relative crate dependencies were not supported, because those paths were not mapped into the docker container. Take the following example structure, where `my-bin` depends on `my-lib`: . ├── my-bin │ ├── Cargo.toml │ └── src │ └── main.rs └── my-lib ├── Cargo.toml └── src └── lib.rs This commits enables such scenarios, by running cross from `.` like so: `cross build --manifest-path=my-lib/Cargo.toml --target x86_64-pc-windows-gnu`, as `.` is mapped as the container's root, and the options passed through to Cargo. Related cross-rs#388 cross-rs#139 cross-rs#277 cross-rs#78 Co-authored-by: Kviring Alexey <alex@kviring.com>

Emilgardis · 2022-04-29T16:24:07Z

I've implemented this with some help from @wngr in #684, please try it out to see if it works. You should not have to do any changes except changing cargo with cross

install the new version from the pr with cargo install cross --git https://github.com/Emilgardis/cross --branch cargo-workspace and try it out.

neilvv · 2022-05-06T12:24:42Z

@Emilgardis - tried with #684 and solves the local dependency problem for me. Thanks.

root This commits adds support for parsing the `--manifest-path` option to cross. So far, the `Cargo.toml` manifest of the crate (or its Cargo workspace) to compile has been assumed to be in the current working directory. This means, that relative crate dependencies were not supported, because those paths were not mapped into the docker container. Take the following example structure, where `my-bin` depends on `my-lib`: . ├── my-bin │ ├── Cargo.toml │ └── src │ └── main.rs └── my-lib ├── Cargo.toml └── src └── lib.rs This commits enables such scenarios, by running cross from `.` like so: `cross build --manifest-path=my-lib/Cargo.toml --target x86_64-pc-windows-gnu`, as `.` is mapped as the container's root, and the options passed through to Cargo. Related cross-rs#388 cross-rs#139 cross-rs#277 cross-rs#78 Co-authored-by: Kviring Alexey <alex@kviring.com>

Alexhuszagh · 2022-05-27T14:52:36Z

This is also partially related to #405, however, #684 fixes the issue much more robustly.

root This commits adds support for parsing the `--manifest-path` option to cross. So far, the `Cargo.toml` manifest of the crate (or its Cargo workspace) to compile has been assumed to be in the current working directory. This means, that relative crate dependencies were not supported, because those paths were not mapped into the docker container. Take the following example structure, where `my-bin` depends on `my-lib`: . ├── my-bin │ ├── Cargo.toml │ └── src │ └── main.rs └── my-lib ├── Cargo.toml └── src └── lib.rs This commits enables such scenarios, by running cross from `.` like so: `cross build --manifest-path=my-lib/Cargo.toml --target x86_64-pc-windows-gnu`, as `.` is mapped as the container's root, and the options passed through to Cargo. Related cross-rs#388 cross-rs#139 cross-rs#277 cross-rs#78 Co-authored-by: Kviring Alexey <alex@kviring.com>

684: enable cargo workspace and path dependencies to work seamlessly r=Alexhuszagh a=Emilgardis This will allow cargo workspaces to work seamlessly with cross, assuming they don't reference anything outside the current workspace root. This change also automatically mounts (if needed) path dependencies. If they do reference things outside the workspace root, and that fact is not know to `cargo metadata` (e.g, it's not a workspace member or a path dependency), you'll have to mount it with ```toml [build.env] volumes = ["THING"] ``` and set `$THING="/path/to/thing"` (we need to work on accessibility for this feature too, a lot of reported issues are solved by it) I'm sure there is an issue directly related to workspaces, but I can't find any. fixes #388 Co-authored-by: Emil Gardström <emil.gardstrom@gmail.com> Co-authored-by: wngr <oliver@wngr.de> Co-authored-by: Alexander Huszagh <ahuszagh@gmail.com>

wngr mentioned this issue Dec 4, 2020

feat: Parse Cargo's --manifest-path option to determine mounted docker root #494

Closed

AlterionX mentioned this issue Sep 20, 2021

Local dependencies are not accessible. #595

Closed

Emilgardis mentioned this issue Apr 7, 2022

enable cargo workspace and path dependencies to work seamlessly #684

Merged

Alexhuszagh added the enhancement label May 27, 2022

bors bot closed this as completed in 0c1d76d Jun 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Building with local dependencies #388

Building with local dependencies #388

Piripant commented Mar 10, 2020

velvia commented Mar 19, 2020

zekronium commented Apr 1, 2020

reitermarkus commented Apr 7, 2020

kornelski commented May 29, 2020

brahmlower commented Oct 31, 2020

wngr commented Dec 2, 2020 •

edited

Loading

TarekkMA commented Mar 21, 2021 •

edited

Loading

mkhattab commented Mar 23, 2022

Emilgardis commented Apr 29, 2022

neilvv commented May 6, 2022

Alexhuszagh commented May 27, 2022

Building with local dependencies #388

Building with local dependencies #388

Comments

Piripant commented Mar 10, 2020

velvia commented Mar 19, 2020

zekronium commented Apr 1, 2020

reitermarkus commented Apr 7, 2020

kornelski commented May 29, 2020

brahmlower commented Oct 31, 2020

wngr commented Dec 2, 2020 • edited Loading

TarekkMA commented Mar 21, 2021 • edited Loading

mkhattab commented Mar 23, 2022

Emilgardis commented Apr 29, 2022

neilvv commented May 6, 2022

Alexhuszagh commented May 27, 2022

wngr commented Dec 2, 2020 •

edited

Loading

TarekkMA commented Mar 21, 2021 •

edited

Loading