Tracel Xtask

---

A collection of easy-to-use and extensible commands to be used in your xtask CLI based on clap.

We rely on these commands in each of our Tracel repositories. By centralizing our redundant commands we save a big amount of code duplication, boilerplate and considerably lower their maintenance cost. This also provides a unified interface across all of our repositories.

These commands are not specific to Tracel repositories and they should be pretty much usable in any Rust repositories with a cargo workspace as well as other repositories where Rust is not necessarily the only language. The commands can be easily extended using handy proc macros and by following some patterns described in this README.

Getting Started

Setting Up a Cargo Workspace with an xtask binary crate

1. Create a new Cargo workspace:

cargo new my_workspace --vcs none
cd my_workspace

2. Create the xtask binary crate:

cargo new xtask --bin --vcs none

3. Configure the workspace:

Edit the Cargo.toml file in the root of the workspace to include the following:

[workspace]
members = ["xtask"]

4. Add the tracel-xtask dependency:

In the xtask/Cargo.toml file, add the following under [dependencies]:

[dependencies]
tracel-xtask = "~1.0"

5. Build the workspace:

cargo build

Your workspace is now set up with a xtask binary crate that depends on tracel-xtask version 1.0.x.

Bootstrap main.rs

1. In the main.rs file of the newly created xtask crate, import the tracel_xtask prelude module and then declare a Command enum. Select the base commands you want to use by adding the macros::base_commands attribute:

use tracel_xtask::prelude::*;

#[macros::base_commands(
    Bump,
    Check,
    Fix
    Test,
)]
pub enum Command {}

2. Update the main function to initialize the xtask parser and dispatch the base commands:

fn main() -> anyhow::Result<()> {
    let args = init_xtask::<Command>()?;
    match args.command {
        // dispatch_base_commands function is generated by the commands macro
        _ => dispatch_base_commands(args),
    }
}

3. Build the workspace with cargo build at the root of the repository to verify that everything is.

4. You should now be able to display the main help screen which lists the commands you selected previously:

cargo xtask --help

Setup aliases for easy invocation

Cargo Alias

Invoking the xtask binary with cargo is very verbose and not really usable as is. Happily we can create a cargo alias to make it really effortless to invoke it.

Create a new file .cargo/config.toml in your repository with the following contents:

[alias]
xtask = "run --target-dir target/xtask --package xtask --bin xtask --"

This saves quite a few characters to type as you can now invoke xtask directly like this:

cargo xtask

Try it with cargo xtask --help.

Shell Alias

We can save even more typing by creating a shell alias for cargo xtask.

For instance we can set the alias to cx. Here is how to do it in various shells.

• For bash:

nano ~/.bashrc

# add this to the file
alias cx='cargo xtask'

# save and source the file or restart the shell session
source ~/.bashrc

• For zsh:

nano ~/.zshrc

# add this to the file
alias cx='cargo xtask'

# save and source the file or restart the shell session
source ~/.zshrc

• For fish:

nano ~/.config/fish/config.fish

# add this to the file
alias cx='cargo xtask'

# save and source the file or restart the shell session
source ~/.config/fish/config.fish

• For powershell:

notepad $PROFILE

# add this at the end of file
function cx {
    cargo xtask $args
}

# save and quit then open a new powershell terminal

Try it with cx --help at the root of the repository.

Conventions

Repository structure

All our repositories follow the same directory hierarchy:

• a crates directory which contains all the crates of the workspace
• an examples directory which holds all the examples crates
• a xtask directory which is the binary crate for our xtask CLI using tracel-xtask

About tests

As per Cargo convention, Integration tests are tests contained in a tests directory of a crate besides its src directory.

Inline tests in src directory are called Unit tests.

tracel-xtask allow to easily execute them separately using the test command.

Interface generalities

Target

There are 4 default targets provided by tracel-xtask:

• workspace which targets the cargo workspace, this is the default target
• crates are all the binary crates and library crates
• examples are all the example crates
• all-packages are both crates and examples targets

workspace and all-packages are different because workspace uses the --workspace flag of cargo whereas all-packages relies on crates and examples targets which use the --package flag. So all-packages executes a command for each crate or example individually.

Here are some examples:

# run all the crates tests
cargo xtask test --target crates all
# check format for examples, binaries and libs
cargo xtask check --target all-packages unit
# build the workspace
cargo xtask build --target workspace
# workspace is the default target so this has the same effect
cargo xtask build

Global options

The following options are global and precede the actual command on the command line:

• Environment (-e, --environment):

cargo xtask -e production build

-e or --environment does not do anything per se in the base commands, it is a flag whose only goal is to inform your custom commands or dispatch functions about the targeted environment which can be development (default), staging or production.

• Execution environment (-E, --execution-environment):

cargo xtask -E no-std build

-E or --execution-environment does not do anything per se in the base commands, it is a flag whose only goal is to inform your custom commands or dispatch functions about the targeted execution environment which can be std or no-std.

• Coverage (-c, --enable-coverage):

-c or --enabled-coverage setups the Rust toolchain to generate coverage information.

Anatomy of a base command

We use the derive API of clap which is based on structs, enums and attribute proc macros. Each base command is a submodule of the base_commands module. If the command accepts arguments there is a corresponding struct named <command>CmdArgs which declare the options, arguments and subcommands. In the case of subcommands a corresponding enum named <command>SubCommand is defined.

Here is an example with a foo command:

#[macros::declare_command_args(Target, FooSubCommand)]
struct FooCmdArgs {}

pub enum FooSubCommand {
    /// A sub command for foo (usage on the command line: cargo xtask foo print-something)
    PrintSomething,
}

Note that it is possible to have an arbitrary level of nested subcommands but deeper nested subcommands cannot be extended, in other words, only the first level of subcommands can be extended. If possible, try to design commands with only one level of subcommands to keep the interface simple.

In the following sections we will see how to create completely new commands as well how to extend existing base commands.

Customization

Create a new command

1. First, we organize commands by creating a commands module. Create a file xtask/src/commands/mycommand.rs as well as the corresponding mod.rs file to declare the module contents.

2. Then, in mycommand.rs define the arguments struct with the declare_command_args macro and define the handle_command function. The declare_command_args macro takes two parameters, the first is the type of the target enum and the second is the type of the subcommand enum if any. If the command has no target or no subcommand then put None for each argument respectively. Target is the default target type provided by tracel-xtask. This type can be extended to support more targets as we will see in a later section.

use tracel_xtask::prelude::*;

#[macros::declare_command_args(Target, None)]
struct MyCommandCmdArgs {}

pub fn handle_command(_args: MyCommandCmdArgs) -> anyhow::Result<()> {
    println!("Hello from my-command");
    Ok(())
}

3. Make sure to update the mod.rs file to declare the command module:

pub(crate) mod my_command;

4. We can now add a new variant to the Command enum in main.rs:

mod commands;

use tracel_xtask::prelude::*;

#[macros::base_commands(
    Bump,
    Check,
    Fix,
    Test,
)]
pub enum Command {
    MyCommand(commands::mycommand::MyCommandCmdArgs),
}

5. And dispatch its handling to our new command module:

fn main() -> anyhow::Result<()> {
    let args = init_xtask::<Command>()?;
    match args.command {
        Command::NewCommand(args) => commands::new_command::handle_command(args),
        _ => dispatch_base_commands(args),
    }
}

6. You can now test your new command with:

cargo xtask my-command --help

cargo xtask my-command

Extend the default Target enum

Let's implement a new command called extended-target to illustrate how to extend the default Target enum.

1. Create a commands/extended_target.rs file and update the mod.rs file as we saw in the previous section.

2. We also need to add a new strum dependency to our Cargo.toml file:

[dependencies]
strum = {version = "0.26.3", features = ["derive"]}

3. Then we can extend the Target enum with the macros::extend_targets attribute in our extended_target.rs file. Here we choose to add a new target called frontend which targets the frontend component we could find for instance in a monorepo:

use tracel_xtask::prelude::*;

#[macros::extend_targets]
pub enum MyTarget {
    /// Target the frontend component of the monorepo.
    Frontend,
}

4. Then we define our command arguments by referencing our newly created MyTarget enum in the declare_command_args attribute:

#[macros::declare_command_args(MyTarget, None)]
struct ExtendedTargetCmdArgs {}

5. Our new target is then available for use in the handle_command function:

pub fn handle_command(args: ExtendedTargetCmdArgs) -> anyhow::Result<()> {
    match args.target {
        // Default targets
        MyTarget::AllPackages => println!("You chose the target: all-packages"),
        MyTarget::Crates => println!("You chose the target: crates"),
        MyTarget::Examples => println!("You chose the target: examples"),
        MyTarget::Workspace => println!("You chose the target: workspace"),

        // Additional target
        MyTarget::Frontend => println!("You chose the target: frontend"),
    };
    Ok(())
}

6. Register our new command the usual way by adding it to our Command enum and dispatch it in the main function:

mod commands;

use tracel_xtask::prelude::*;

#[macros::base_commands(
    Bump,
    Check,
    Fix,
    Test,
)]
pub enum Command {
    ExtendedTarget(commands::extended_target::ExtendedTargetCmdArgs),
}

fn main() -> anyhow::Result<()> {
    let args = init_xtask::<Command>()?;
    match args.command {
        Command::ExtendedTarget(args) => commands::extended_target::handle_command(args),
        _ => dispatch_base_commands(args),
    }
}

7. Test the command with:

cargo xtask extended-target --help

cargo xtask extended-target --target frontend

Extend a base command

To extend an existing command we use the macros::extend_command_args attribute which takes three parameters:

• first argument is the type of the base command arguments struct to extend,
• second argument is the target type (or None if there is no target),
• third argument is the subcommand type (or None if there is no subcommand).

Let's use two examples to illustrate this, the first is a command to extend the build base command with a new --debug argument; and the second is a new command to extend the subcommands of the check base command to add a new my-check subcommand.

Note that you can find more examples in the xtask crate of this repository.

Extend the arguments of a base command

We create a new command called extended-build-args which will have an additional argument called --debug.

1. Create the commands/extended_build_args.rs file and update the mod.rs file as we saw in the previous section.

2. Extend the BuildCommandArgs struct using the attribute macros::extend_command_args and define the handle_command function. Note that the macro automatically implements the TryInto trait which makes it easy to dispatch back to the base command own handle_command function. Also note that if the base command requires a target then you need to provide a target as well in your extension, i.e. the target parameter of the macro cannot be None if the base command has a Target.

use tracel_xtask::prelude::*;

#[macros::extend_command_args(BuildCmdArgs, Target, None)]
pub struct ExtendedBuildArgsCmdArgs {
    /// Print additional debug info when set
    #[arg(short, long)]
    pub debug: bool,
}

pub fn handle_command(args: ExtendedBuildArgsCmdArgs) -> anyhow::Result<()> {
    if args.debug {
        println!("Debug is enabled");
    }
    base_commands::build::handle_command(args.try_into().unwrap())
}

3. Register the new command the usual way by adding it to the Command enum and dispatch it in the main function:

mod commands;

use tracel_xtask::prelude::*;

#[macros::base_commands(
    Bump,
    Check,
    Fix,
    Test,
)]
pub enum Command {
    ExtendedBuildArgs(commands::extended_build_args::ExtendedBuildArgsCmdArgs),
}

fn main() -> anyhow::Result<()> {
    let args = init_xtask::<Command>()?;
    match args.command {
        Command::ExtendedBuildArgs(args) => commands::extended_build_args::handle_command(args),
        _ => dispatch_base_commands(args),
    }
}

4. Test the command with:

cargo xtask extended-build-args --help

cargo xtask extended-build-args --debug

Extend the subcommands of a base command

For this one we create a new command called extended-check-subcommands which will have an additional subcommand.

1. Create a commands/extended_check_subcommands.rs file and update the mod.rs file as we saw in the previous section.

2. Extend the CheckCommandArgs struct using the attribute macros::extend_command_args:

use tracel_xtask::prelude::*;

#[macros::extend_command_args(CheckCmdArgs, Target, ExtendedCheckSubcommand)]
pub struct ExtendedCheckedArgsCmdArgs {}

3. Implement the ExtendedCheckSubcommand enum by extending the CheckSubcommand base enum with the macro extend_subcommands. It takes the name of the type of the subcommand enum to extend:

#[macros::extend_subcommands(CheckSubCommand)]
pub enum ExtendedCheckSubcommand {
    /// An additional subcommand for our extended check command.
    MySubcommand,
}

4. Implement the handle_command function to handle the new subcommand. Note that we must handle the All subcommand as well:

use strum::IntoEnumIterator;

pub fn handle_command(args: ExtendedCheckedArgsCmdArgs) -> anyhow::Result<()> {
    match args.get_command() {
        ExtendedCheckSubcommand::MySubcommand => run_my_subcommand(args.clone()),
        ExtendedCheckSubcommand::All => {
            ExtendedCheckSubcommand::iter()
                .filter(|c| *c != ExtendedCheckSubcommand::All)
                .try_for_each(|c| {
                    handle_command(
                        ExtendedCheckedArgsCmdArgs {
                            command: Some(c),
                            target: args.target.clone(),
                            exclude: args.exclude.clone(),
                            only: args.only.clone(),
                        },
                    )
                })
        }
        _ => base_commands::check::handle_command(args.try_into().unwrap()),
    }
}

fn run_my_subcommand(_args: ExtendedCheckedArgsCmdArgs) -> Result<(), anyhow::Error> {
    println!("Executing new subcommand");
    Ok(())
}

5. Register the new command the usual way by adding it to the Command enum and dispatch it in the main function:

mod commands;

use tracel_xtask::prelude::*;

#[macros::base_commands(
    Bump,
    Check,
    Fix,
    Test,
)]
pub enum Command {
    ExtendedCheckSubcommand(commands::extended_check_subcommands::ExtendedCheckedArgsCmdArgs),
}

fn main() -> anyhow::Result<()> {
    let args = init_xtask::<Command>()?;
    match args.command {
        Command::ExtendedCheckSubcommand(args) => commands::extended_check_subcommands::handle_command(args),
        _ => dispatch_base_commands(args),
    }
}

6. Test the command with:

cargo xtask extended-check-subcommands --help

cargo xtask extended-check-subcommands my-check

Custom builds and tests

tracel-xtask provides helper functions to easily execute custom builds or tests with specific features or build targets (do not confuse Rust build targets which is an argument of the cargo build command with the xtask target we introduced previously).

For instance we can extend the build command to build additional crates with custom features or build targets using the helper function:

pub fn handle_command(mut args: tracel_xtask::commands::build::BuildCmdArgs)  -> anyhow::Result<()> {
    // regular execution of the build command
    tracel_xtask::commands::build::handle_command(args)?;

    // additional crate builds
    // build 'my-crate' with all the features
    tracel_xtask::utils::helpers::custom_crates_build(vec!["my-crate"], vec!["--all-features"], None, None, "all features")?;
    // build 'my-crate' with specific features
    tracel_xtask::utils::helpers::custom_crates_build(vec!["my-crate"], vec!["--features", "myfeature1,myfeature2"], None, None, "myfeature1,myfeature2")?;
    // build 'my-crate' with a different target than the default one
    tracel_xtask::utils::helpers::custom_crates_build(vec!["my-crate"], vec!["--target", "thumbv7m-none-eabi"], None, None, "thumbv7m-none-eabi target")?;
    Ok(())
}

Enable and generate coverage information

Here is a example GitHub job which shows how to setup coverage, enable it and upload coverage information to codecov:

env:
  GRCOV_LINK: "https://github.com/mozilla/grcov/releases/download"
  GRCOV_VERSION: "0.8.19"

jobs:
  my-job:
    runs-on: ubuntu-22.04
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      - name: install rust
        uses: dtolnay/rust-toolchain@master
        with:
          components: rustfmt, clippy
          toolchain: stable
      - name: Install grcov
        shell: bash
        run: |
          curl -L "$GRCOV_LINK/v$GRCOV_VERSION/grcov-x86_64-unknown-linux-musl.tar.bz2" |
          tar xj -C $HOME/.cargo/bin
          cargo xtask coverage install
      - name: Build
        shell: bash
        run: cargo xtask build
      - name: Tests
        shell: bash
        run: cargo xtask --enable-coverage test all
      - name: Generate lcov.info
        shell: bash
        # /* is to exclude std library code coverage from analysis
        run: cargo xtask coverage generate --ignore "/*,xtask/*,examples/*"
      - name: Codecov upload lcov.info
        uses: codecov/codecov-action@v4
        with:
          files: lcov.info
          token: ${{ secrets.CODECOV_TOKEN }}

Special command 'validate'

By convention this command is responsible to run all the checks, builds, and/or tests that validate the code before opening a pull request or merge request.

The command Validate can been added via the macro tracel_xtask_macros::commands like the other commands.

By default all the checks from the check command are run as well as both unit and integration tests from the test command.

You can make your own handle_command function if you need to perform more validations. Ideally this function should only call the other commands handle_command functions.

For quick reference here is a simple example to perform all checks and tests against the workspace:

pub fn handle_command(args: ValidateCmdArgs) -> anyhow::Result<()> {
    let target = Target::Workspace;
    let exclude = vec![];
    let only = vec![];

    // checks
    [
        CheckSubCommand::Audit,
        CheckSubCommand::Format,
        CheckSubCommand::Lint,
        CheckSubCommand::Typos,
    ]
    .iter()
    .try_for_each(|c| {
        super::check::handle_command(CheckCmdArgs {
            target: target.clone(),
            exclude: exclude.clone(),
            only: only.clone(),
            command: Some(c.clone()),
            ignore_audit: args.ignore_audit,
        })
    })?;

    // tests
    super::test::handle_command(TestCmdArgs {
        target: target.clone(),
        exclude: exclude.clone(),
        only: only.clone(),
        threads: None,
        jobs: None,
        command: Some(TestSubCommand::All),
    })?;

    Ok(())
}

Base commands list

Check and Fix

The check and fix commands are designed to help you maintain code quality during development. They run various checks and fix issues, ensuring that your code is clean and follows best practices.

check and fix contains the same subcommands to audit, format, lint or proofread a code base.

While the check command only reports issues, the fix command attempts to fix them as they are encountered.

Each check can be executed separately or all of them can be executed sequentially using all.

Usage to lint the code base:

cargo xtask check lint

cargo xtask fix lint

cargo xtask fix all

Running Tests

Testing is a crucial part of development, and the test command is designed to make this process easy.

This command makes the distinction between unit tests and integrations tests. Unit tests are inline tests under the src directory of a crate. Integration tests are tests defined in files under the tests directory of a crate besides the src directory.

Usage:

# execute workspace unit tests
cargo xtask test unit
# execute workspace integration tests
cargo xtask test integration
# execute workspace both unit tests and integration tests
cargo xtask test all

Note that documentation tests are supported by the doc command.

Documentation

Command to build and test the documentation in a workspace.

Bumping Versions

This is a command reserved for repository maintainers.

The bump command is used to update the version numbers of all first-party crates in the repository. This is particularly useful when you're preparing for a new release and need to ensure that all crates have the correct version.

You can bump the version by major, minor, or patch levels, depending on the changes made. For example, if you’ve made breaking changes, you should bump the major version. For new features that are backwards compatible, bump the minor version. For bug fixes, bump the patch version.

Usage:

cargo xtask bump <SUBCOMMAND>

Publishing Crates

This is a command reserved for repository maintainers and is typically used in publish GitHub workflows.

This command automates the process of publishing crates to crates.io, the Rust package registry. By specifying the name of the crate, xtask handles the publication process, ensuring that the crate is available for others to use.

Usage:

cargo xtask publish <NAME>

As mentioned, this command is often used in a GitHub workflow. We provide a Tracel's reusable publish-crate workflow that makes use of this command. Here is a simple example with a workflow that publishes two crates A and B with A depending on B.

name: publish all crates

on:
  push:
    tags:
      - "v*"

jobs:
  publish-B:
    uses: tracel-ai/github-actions/.github/workflows/publish-crate.yml@v1
    with:
      crate: B
    secrets:
      CRATES_IO_API_TOKEN: ${{ secrets.CRATES_IO_API_TOKEN }}

  # --------------------------------------------------------------------------------
  publish-A:
    uses: tracel-ai/github-actions/.github/workflows/publish-crate.yml@v1
    with:
      crate: A
    needs:
      - publish-B
    secrets:
      CRATES_IO_API_TOKEN: ${{ secrets.CRATES_IO_API_TOKEN }}

Coverage

This command provide a subcommand to install the necessary dependencies for performing code coverage and a subcommand to generate the coverage info file that can then be uploaded to a service provider like codecov. See dedicated section Enable and generate coverage information.

Dependencies

Various additional subcommands about dependencies.

deny make sure that all dependencies meet requirements using cargo-deny.

unused detects dependencies in the workspace that are not in ussed.

Vulnerabilities

This command makes it easier to execute sanitizers as described in the Rust unstable book.

These sanitizers require a nightly toolchain.

Run the specified vulnerability check locally. These commands must be called with 'cargo +nightly'

Usage: xtask vulnerabilities <COMMAND>

Commands:
  all                            Run all most useful vulnerability checks
  address-sanitizer              Run Address sanitizer (memory error detector)
  control-flow-integrity         Run LLVM Control Flow Integrity (CFI) (provides forward-edge control flow protection)
  hw-address-sanitizer           Run newer variant of Address sanitizer (memory error detector similar to AddressSanitizer, but based on partial hardware assistance)
  kernel-control-flow-integrity  Run Kernel LLVM Control Flow Integrity (KCFI) (provides forward-edge control flow protection for operating systems kerneljs)
  leak-sanitizer                 Run Leak sanitizer (run-time memory leak detector)
  memory-sanitizer               Run memory sanitizer (detector of uninitialized reads)
  mem-tag-sanitizer              Run another address sanitizer (like AddressSanitizer and HardwareAddressSanitizer but with lower overhead suitable for use as hardening for production binaries)
  nightly-checks                 Run nightly-only checks through cargo-careful `<https://crates.io/crates/cargo-careful>`
  safe-stack                     Run SafeStack check (provides backward-edge control flow protection by separating stack into safe and unsafe regions)
  shadow-call-stack              Run ShadowCall check (provides backward-edge control flow protection - aarch64 only)
  thread-sanitizer               Run Thread sanitizer (data race detector)
  help                           Print this message or the help of the given subcommand(s)