Testing whole workspace with features enabled in some crate(s)

[djc]

Problem
I have a workspace with a testing crate and some unit tests in other crates. Some tests need big dependencies, so I'd like to disable these by default and enable them only in CI. In CI, I then want to run all the tests, enabling the features for these tests. Here are some things I've tried:

cargo test --all --features dependency-test
cargo test --all --features testing/dependency-test (testing is the directory the testing crate lives in relative to the workspace root)
cargo test --all --features project_testing/dependency-test (project_testing is the package.name for the testing crate)

However, none of these seem to actually run the tests in question, whereas I'd expect at least one of them would (most likely the latter one?).

Actual use case is https://github.com/djc/askama, the actix feature in the testing crate. (Prototype change is here: https://github.com/djc/askama/compare/actix-off?expand=1).

Testing with stable Rust/Cargo on either macOS or (Linux) Travis.

[ehuss]

Some workarounds:
cargo test -p askama_testing --features "actix" -Z package-features
or cd into each directory and run cargo test within it. AFAIK you can't do all packages at once. Currently in a workspace the features selected depends on which directory you are in (-Z package-features changes this behavior).

[djc]

I understand about the workaround of cding into a particular directory, but that means either making the Travis config more complex by specifying all contained crates separately, or making the test suite slower to run because some tests are run twice.

Since I'm running this at the top of the workspace, I think there are two reasonable conceptual models that could work: either I'm one level "above" the crates, and I should be able to use a crate/feature value to select a feature in the crates, or my command is more or less "copied" into each crate, in which case I should be able to use the bare feature.

If neither of these work, it seems like there is a conceptual hole in the workspace implementation (or how the test command interacts with it).

[ehuss]

Yea, I agree there should be some way to do it. Typically the form foo/bar for features means the bar feature of the foo dependency, but in a virtual workspace there is no foo dependency (there is a foo member). So it may be confusing what that actually means. It may also be confusing on how to select features when building multiple packages (which is why I believe -Z package-features prevents you from trying).

This has been reported several times (#4753, #4106, #4463, #5364, etc.), so you're not alone. I'm trying to collect all the issues with features to see if there is a cohesive way to deal with them all.

[djc]

@ehuss -- more than a year later -- do you have any ideas on how we could move this forward?

It still seems to me that the dependency vs member difference is small enough that we can paper over it in this case; conceptually, a workspace already in some ways acts like a meta-crate that depends on all (or some, with default-members) of its members.