Design document for "features"

[emberian]

The basic idea is to have "feature sets". When writing a package, you can specify a list of "features" for a dependency to be built with. And in the dependency, it can do various operations depending on a feature. So for example, the top-level package:

[dependencies.db]
features = ["postgres"]

Then, in the manifest for db:

[features.postgres]
    [dependencies.postgres]
    git = "https://github.com/sfackler/rust-postgres"

When a dependency is specified as part of a feature, it is only built when that feature is specified.

The "features" for a crate would also be exposed as a --cfg flag to rustc for that package and as a comma-separated list of features in an environment variable, CARGO_FEATURES. The following would work:

#[cfg(cargo_feature = "postgres")]
fn connect() { ... }

(Specifically, a --cfg cargo_feature="$FEAT" would be passed for every feature. See this gist for an example of how this can work)

There is some subtlety in this feature. Consider the following:

Main manifest:

[dependencies.db]
features = ["postgres"]

[dependencies.foo]
git = "..."

Foo manifest:

[dependencies.db]
features = ["sqlite"]

The problem is that transitive dependencies may have a different set of features.
To solve this, a package which supports features can specify a list of mutually exclusive features. When this package is used as a dependency, every feature specified in the dependency graph will be taken as a list and compared against the mutually exclusive feature lists. For example, db could have:

[package]
name = "db"
mutually_exclusive_features = [["postgres", "sqlite", "mysql"], ["awesome", "lame"]]

If not specified, no features are mutually exclusive.

[wycats]

Sadly, you cannot nest in Toml the way you show:

[features.postgres]
    [dependencies.postgres]
    git = "https://github.com/sfackler/rust-postgres"

Instead, it will likely need to be something like:

[dependencies.postgres]
git = "https://github.com/sfackler/rust-postgres"

[features.postgres]
dependencies = ["postgres"]
cfg = ["db_posgres"]

One nice thing about this structure is that you can easily use the same dependency in multiple feature sets.

[emberian]

Addition: dependencies listed as part of a feature are not built if that feature isn't specified. That is, they aren't built by default anymore.

Addition: feature_clusters = [[...]] which has the same syntax as mutually_exclusive_features but requires exactly one feature from each sub-list to be included.

[wycats]

Maybe feature_options and mandatory_feature_options?

[emberian]

feature_clusters and mandatory_feature_clusters: share the root, but don't have enough meaning to just guess what they mean and be surprised, and easy to remember.

[jansegre]

I'm not exactly sure if I understand how this would be used on the code. From the example:

#[cfg(cargo_feature = "postgres")]
fn connect() { ... }

Does it means that the cfg cargo_feature has a value "postgres"? If so it would imply that you can only have one feature, right?

Wouldn't the following idiom be preferred?

#[cfg(cargo_feature_postgres)]
fn connect() { ... }

Or simply

#[cfg(feature_postgres)]
fn connect() { ... }

Additionally I would take a look at gentoo's portage use flags system, it has a powerful and mature implementation of a very similar system, having already designed depending on a package with at least an specific set of features (use flags), default features, deep features, and many of the sort.

[emberian]

@jansegre see the gist I linked for how it works with multiple features.

A complex system like portage is completely unreasonable, both in terms of complexity of solving for dependencies and in terms of needless features.

[jansegre]

Ok, I got it now. Made this simple test based on your gist and it worked as expected:

fn main() {
    if cfg!(cargo_feature = "foo") { println!("foo") }
    if cfg!(cargo_feature = "bar") { println!("bar") }
}

$ rustc --cfg cargo_feature=\"foo\" a.rs && ./a
foo

$ rustc --cfg cargo_feature=\"foo\" --cfg cargo_feature=\"bar\" a.rs && ./a
foo
bar

Although I had to literally escape the " quotes. And it's not intuitive at all that cargo_feature="foo" and cargo_feature="bar" can both be true.

[wycats]

Here is a more fleshed out version of the design: https://gist.github.com/wycats/199f09a3ff1c8e6dcebd. I have embedded it below.

[wycats]

"Features" Design

The goal of "features" is to make it possible to express:

• Optional dependencies, which enhance a package, but are not required
• Clusters of optional dependencies, such as "postgres", that would include the postgres package, the postgres-macros package, and possibly other packages (such as development-time mocking libraries, debugging tools, etc.)

We originally thought that a related goal, the ability to specify a list of "pick exactly one" options to choose from, was in scope for "features", but have since come to believe that those choices are better expressed as "Provides"-style virtual dependencies.

Features

To flesh out the design, we used Rails, which has a combination of mandatory dependencies, optional opt-in dependencies and optional opt-out dependencies.

Rails uses a Bundler-specific structure to express these dependencies (using a generated Gemfile), but we were able to use it as a real-world example of the constraints.

The Rails package, with optional dependencies expressed as features.

[package]

name = "rails"

[features]

# The "default" set of optional packages. Most people will want
# to use these packages, but they are strictly optional
default = ["sass-rails", "uglifier", "jquery-rails", "sdoc"]

# The "omakase" set of optional packages. These are packages
# curated by DHH as desirable for normal usage. Some developers
# will leave the "default" packages, but disable the hand-curated
# Omakase packages.
omakase = ["coffee-rails", "turbolinks", "jbuilder"]

# The "secure-password" feature depends on the bcrypt gem. This
# aliasing will allow people to talk about the feature in a
# higher-level way and allow Rails to add more requirements to
# the feature in the future.
secure-password = ["bcrypt"]

[dependencies]

# These packages are mandatory, and form the core of the Rails
# distribution. They are locked to particular versions to make
# sure that Cargo will only include one, precise version of
# Rails in the compiled binary.
actionmailer = "=4.1.5"
actionpack = "=4.1.5"
actionview = "=4.1.5"
activemodel = "=4.1.5"
activerecord = "=4.1.5"
activesupport = "=4.1.5"
railties = "=4.1.5"

# A few other mandatory dependencies that do not have strict
# version dependencies
bundler = "1.3.0"
sprockets-rails = "2.0.0"

# A list of all of the optional dependencies, some of which
# are included in the above "features". They can be opted
# into by apps.
sass-rails = { version = "4.0.3", optional = true }
uglifier = { version = "1.3.0", optional = true }
coffee-rails = { version = "4.0.0", optional = true }
bcrypt = { version = "3.1.7", optional = true }
unicorn = { version = "*", optional = true }
turbolinks = { version = "*", optional = true }
jbuilder = { version = "*", optional = true }

[dev-dependencies]

sdoc = { version = "0.4.0", optional = true }
debugger = { version = "*", optional = true }
spring = { version = "*", optional = true }

To use Rails,

[package]

name = "my-app"

[dependencies.rails]

version = "4.1.5"
features = ["omakase", "unicorn"]

# do not include the default features, and optionally
# cherry-pick individual features
# default-features = false

Rules

1. Feature names must not conflict with package names. This is because they are opted into via features = [...], which only has a single namespace
2. With the exception of the default feature, all features are opt-in. To opt out of the default feature, use default-features = false and cherry-pick individual features.
3. When a feature is selected, Cargo will call rustc with --cfg cargo_feature=${feature_name}. If a feature group is included, both the group and all of its individual features will be included. This can be tested in code via #[cfg(cargo_feature = "turbolinks")]

General Usage

In End Products

One major use-case for this feature is specifying optional features in end-products. For example, the Servo project may want to include optional features that people can enable or disable when they build it.

In that case, Servo will describe features in its Cargo.toml and they can be enabled using command-line flags:

$ cargo build --release --features "shumway pdf"

Default features could be excluded using --no-default-features.

In Packages

In most cases, the concept of "optional dependency" in a library is best expressed as a separate package that the top-level application depends on.

However, high-level packages, like Iron or Piston, may want the ability to curate a number of packages for easy installation. The current Cargo system allows them to curate a number of mandatory dependencies into a single package for easy installation.

In some cases, packages may want to provide additional curation for optional dependencies:

• Grouping a number of low-level optional dependencies together into a single high-level "feature".
• Specifying packages that are recommended (or suggested) to be included by users of the package (the "default" and "omakase" features in the motivating example).
• Including a feature (like secure-password in the motivating example) that will only work if an optional dependency is available, and would be difficult to implement as a separate package. For example, it may be overly difficult to design an IO package to be completely decoupled from OpenSSL, with opt-in via the inclusion of a separate package.

In almost all cases, it is an antipattern to use these features outside of high-level packages that are designed for curation. If a feature is optional, it can almost certainly be expressed as a separate package.

[huonw]

Feature names must not conflict with package names. This is because they are opted into via features = [...], which only has a single namespace

To clarify, is the uniqueness global (i.e. must not conflict with any package in the registry), or just local (i.e. must not conflict with any dependencies etc.)?

[jansegre]

That is strange, it leaves out features names like "mysql", which is a
reasonable name for a dependency as well.

On Fri, Aug 29, 2014 at 9:47 PM, Huon Wilson notifications@github.com
wrote:

Feature names must not conflict with package names. This is because they
are opted into via features = [...], which only has a single namespace

To clarify, is the uniqueness global (i.e. must not conflict with any
package in the registry), or just local (i.e. must not conflict with any
dependencies etc.)?

—
Reply to this email directly or view it on GitHub
#385 (comment).

Att,
Jan Segre

[gilles-leblanc]

Just a question, since #509 hasn't been merged yet this hasn't landed, right?

I ask because I was looking at the following Servo issue: servo/servo#3278.

[lilyball]

I agree with @jansegre, it seems odd that feature names would conflict with package names. I'm assuming that the intent here is that --features can be used to opt in to optional dependencies without having to actually declare them as a distinct feature group.

The ability to do this strikes me as odd. It means that I cannot have an optional dependency that only makes sense when bundled with another optional dependency. I can use a feature group that includes both, but any client of mine could treat them as distinct features and try to enable one without the other.

Instead, I'd much rather see every feature have to be explicitly listed. Features could either be an array of other feature names, or could be a declaration of dependencies. The former is for feature clusters (such as the omakase feature), and the latter is for actually defining each feature. The example rails manifest would then look like

[features]

# Feature clusters: each of the next 3 features contain arrays
# that refer specifically to other features.

# The "default" set of optional packages. Most people will want
# to use these packages, but they are strictly optional
default = ["sass-rails", "uglifier", "jquery-rails", "sdoc"]

# The "omakase" set of optional packages. These are packages
# curated by DHH as desirable for normal usage. Some developers
# will leave the "default" packages, but disable the hand-curated
# Omakase packages.
omakase = ["coffee-rails", "turbolinks", "jbuilder"]

# The "secure-password" feature depends on the bcrypt gem. This
# aliasing will allow people to talk about the feature in a
# higher-level way and allow Rails to add more requirements to
# the feature in the future.
secure-password = ["bcrypt"]

[features.sass-rails]
dependencies = ["sass-rails"]

[features.uglifier]
dependencies = ["uglifier"]

[features.jquery-rails]
dependencies = ["jquery-rails"]

[features.sdoc]
dependencies = ["sdoc"]

[features.coffee-rails]
dependencies = ["coffee-rails"]

[features.turbolinks]
dependencies = ["turbolinks"]

[features.jbuilder]
dependencies = ["jbuilder"]

[features.bcrypt]
dependencies = ["bcrypt"]

This looks kind of verbose, although I think that the example here of rails wanting to provide an easy way to get external packages is kind of unusual, as I would normally expect the clients of rails to simply pull in these other packages themselves. More likely, I would think, is something like

[features.python]
dependencies = ["python", "py-awesomepackage"]

where the one feature named python needs to pull in two separate dependencies.

I also question whether "feature clusters" are common enough to warrant having the simplest syntax. An alternative approach would look like

[feature-clusters]

# The "default" set of optional packages. Most people will want
# to use these packages, but they are strictly optional
default = ["sass-rails", "uglifier", "jquery-rails", "sdoc"]

# The "omakase" set of optional packages. These are packages
# curated by DHH as desirable for normal usage. Some developers
# will leave the "default" packages, but disable the hand-curated
# Omakase packages.
omakase = ["coffee-rails", "turbolinks", "jbuilder"]

# The "secure-password" feature depends on the bcrypt gem. This
# aliasing will allow people to talk about the feature in a
# higher-level way and allow Rails to add more requirements to
# the feature in the future.
secure-password = ["bcrypt"]

[features]

sass-rails = ["sass-rails"]
uglifier = ["uglifier"]
jquery-rails = ["jquery-rails"]
sdoc = ["sdoc"]
coffee-rails = ["coffee-rails"]
turbolinks = ["turbolinks"]
jbuilder = ["jbuilder"]
bcrypt = ["bcrypt"]

This still has the oddity of duplicating each optional package name as a feature, though again, I think the way rails works here is kind of unusual. More generally, feature names are not necessarily going to want to mirror the particular packages they depend on.

[lilyball]

I'm also not sure I agree with the premise of a "default" feature cluster that can be opted out in its entirety. It makes more sense to me to simply have certain features be defaulted on, and to be able to opt-out on an individual feature basis. The reason being that I think it's more common to want to disable specific functionality rather than disabling everything. After all, default features are defaulted because they're expected to be wanted by most people. If I only want to turn off one feature, then I need to know what the complete default feature set is so I can turn everything else back on.

Instead, with the original syntax, we could add a default = true key to each feature, and with the alternative syntax we can just have a default-features key (instead of a cluster). As for actually opting out of default features, I can see two approaches:

1. Have a separate flag and manifest entry for "features to disable". This could be something like --without-features "foo bar" as a flag, and without-features = ["foo", "bar"] in the manifest.
2. Generate a feature name "no-foo" for every default feature "foo". You could then say something like --features "shumway pdf no-nsa" to disable the default "nsa" feature, and features = ["shumway", "pdf", "no-nsa"] in the manifest.

I'm partial to the second approach, but it has the issue of needing to handle the case of a feature named "no-foo". I'm inclined to just declare that features may not have the prefix "no-" and treat the presence of such a feature as an error (just as feature names with spaces need to be disallowed, for compatibility with the --features command-line flag).