implement "target" features #63

Gankra · 2022-05-03T00:55:03Z

able to specify that an audit is only valid for a target(s)
able to specify that a policy only applies to a target(s) (package has no reqs for targets that aren't included)

Targets are based on cfg syntax and include:

any (or)
all (and)
not
platform "atoms" like target_arch, target_os
feature "atoms" like my_crate_feature
target "families" like "unix" and "wasm" which overlap with platform atoms
target triples like x86_64-gnu-unknown-linux which overlap with platform atoms

Gankra · 2022-05-03T00:55:57Z

Here are some "interesting" cfg's that show up in firefox's and wasm-time's cargo metadata graphs:

// Stuff in wasm-time:

// Simple
"target": "cfg(unix)",
"target": "cfg(windows)",
"target": "cfg(not(windows))",
"target": "cfg(target_os = \"windows\")",
"target": "cfg(not(target_os = \"wasi\"))",
"target": "cfg(target_arch = \"wasm32\")",
"target": "cfg(not(target_arch = \"wasm32\"))",

// Complex
"target": "cfg(any(target_os = \"macos\", target_os = \"ios\"))",
"target": "cfg(any(target_arch = \"x86_64\", target_arch = \"x86\"))",
"target": "cfg(all(target_arch = \"aarch64\", target_os = \"linux\"))",
"target": "cfg(not(any(windows, target_os = \"hermit\")))",
"target": "cfg(not(any(windows, target_arch = \"s390x\")))",
"target": "cfg(not(all(target_arch = \"arm\", target_os = \"none\")))",
"target": "cfg(not(any(target_os = \"linux\", target_os = \"android\", target_os = \"windows\", target_os = \"macos\", target_os = \"ios\", target_os = \"freebsd\", target_os = \"openbsd\", target_os = \"netbsd\", target_os = \"dragonfly\", target_os = \"solaris\", target_os = \"illumos\", target_os = \"fuchsia\", target_os = \"redox\", target_os = \"cloudabi\", target_os = \"haiku\", target_os = \"vxworks\", target_os = \"emscripten\", target_os = \"wasi\")))",
"target": "cfg(any(target_os = \"linux\", target_os = \"android\", target_os = \"windows\", target_os = \"macos\", target_os = \"ios\", target_os = \"freebsd\", target_os = \"openbsd\", target_os = \"netbsd\", target_os = \"dragonfly\", target_os = \"solaris\", target_os = \"illumos\", target_os = \"fuchsia\", target_os = \"redox\", target_os = \"cloudabi\", target_os = \"haiku\", target_os = \"vxworks\", target_os = \"emscripten\", target_os = \"wasi\"))",

// Evil
"target": "cfg(all(any(target_arch = \"x86_64\", target_arch = \"aarch64\"), target_os = \"hermit\"))",
"target": "cfg(all(not(rustix_use_libc), not(miri), target_os = \"linux\", any(target_arch = \"x86\", all(target_arch = \"x86_64\", not(target_pointer_width = \"32\")), target_arch = \"arm\", target_arch = \"aarch64\", target_arch = \"powerpc64\", target_arch = \"riscv64\", target_arch = \"mips\", target_arch = \"mips64\")))",
"target": "cfg(all(windows, not(target_vendor = \"uwp\")))",

// Triples
"target": "aarch64-apple-darwin",
"target": "aarch64-linux-android",
"target": "wasm32-unknown-unknown",
"target": "asmjs-unknown-emscripten",
"target": "wasm32-unknown-emscripten",

// Totally Custom
"target": "cfg(loom)",
"target": "cfg(tokio_unstable)",
"target": "cfg(tracing_unstable)",
"target": "cfg(fuzzing)",




// Extra Evil Stuff in firefox

"target": "cfg(all(target_arch = \"wasm32\", not(any(target_os = \"emscripten\", target_os = \"wasi\"))))",
"target": "cfg(all(target_arch = \"wasm32\", not(target_os = \"emscripten\")))",
"target": "cfg(any(target_os = \"android\", all(unix, not(target_os = \"macos\"))))",
"target": "cfg(all(not(target_arch = \"wasm32\"), any(target_os = \"ios\", target_os = \"macos\")))",
"target": "cfg(all(not(target_arch = \"wasm32\"), unix, not(target_os = \"ios\"), not(target_os = \"macos\")))",
"target": "cfg(all(not(target_arch = \"wasm32\"), windows))",

Gankra · 2022-05-03T01:09:25Z

In general a cfg expression is a boolean expression with a possible large number of input vars, and answering questions like "is this cfg <= another" is BOOL SAT and therefore NP-complete. Cargo and rustc don't need to answer such questions because the pick a specific target and just plug the inputs in and check if an expr is true or false.

We would be evaluating the exprs more abstractly, putting us more in BOOL SAT territory. The existence of "not" also introduces problematic law-of-the-excluded-middle problems. For instance, checking if an audit with "any(X, Y)" satisfies "NOT Z" requires you to know for certain that the universe is strictly "X, Y, Z". This is not generally possible to know, as rustc can always introduce new OSes/arches.

If you remove "all" and "not" and just support "any" this is relatively tractable because "any" just means "set all these bits" and you can have nice little bitvectors of length N (N=number of atoms) and do subset queries easily. If you introduce either AND or NOT then things quickly become BOOL SAT and you need to have bitvectors of length 2^N to represent the full truth table, in the general case.

"triples" and "families" are however still annoying because we potentially need to "understand" the contents of "unix" or "x86_64-unknown-linux-gnu" to map them to equivalent atoms (which again, may have excluded middle issues).

bholley · 2022-05-17T22:37:39Z

Still not quite sure what to do with this. I think we can work around the lack of this for now, and reevaluate when a strong use-case arises.

conradludgate · 2022-06-17T16:09:55Z

I have a use case for this:

My company only deploys code to linux servers and runs dev environments on linux/macos laptops.

Currently, windows-sys appears in my cargo vet suggest but this never even gets compiled in any or our environments. Ideally, I could have

[policy.'*']
targets = ["unix"]

and it would just ignore windows-sys

bholley · 2022-06-17T16:16:22Z

For sure. We definitely want some kind of affordance for this, we just have to figure out how to make it computationally tractable.

In the mean time, you can add suggest = false to that entry in your unaudited.toml to get it off the radar.

Gankra · 2022-06-17T17:50:10Z

cargo-deny seems to have figured out some tolerable approach to this, we should look into it

https://embarkstudios.github.io/cargo-deny/checks/cfg.html#the-targets-field-optional

(cc @Jake-Shadle who was interested in the two interoping)

bholley · 2022-06-17T18:46:27Z

So I think there are two broad categories of potential feature here.

The first is to exclude crates from the vet graph that don't get built on the platforms of interest. This seems like something we can just borrow cargo-deny's approach for (neat to see that rustc has a set of built-ins). The key point here is that audits don't have to be target-aware at all.

The second is to allow excluding target-specific code at sub-crate granularity, by augmenting audit records to record that the audit is only valid for a subset of the targets for which the crate builds. This is a much-harder problem per all the discussion above.

So it seems like we should perhaps do the first-thing, which has clear use-cases, and punt on the second thing, which may never be needed often enough to justify the complexity.

repi · 2022-11-19T10:16:10Z

would definitely appreciate, esp. as a start, the first and simple approach we have with cargo-deny of set of exact target triples to include in the vetting graph.

here is for example how our main application/server project's deny.toml looks like in the top (and could imagine something quite similar for supply-chain/config.toml:

targets = [
    { triple = "x86_64-unknown-linux-gnu" },
    { triple = "aarch64-apple-darwin" },
    { triple = "x86_64-apple-darwin" },
    { triple = "x86_64-pc-windows-msvc" },
    { triple = "aarch64-linux-android" },
    { triple = "x86_64-unknown-linux-musl" },
]

and a our Wasm-only projects correspondingly have just:

targets = [
    { triple = "wasm32-unknown-unknown" },
]

which drastically helps prune the dependency graph, and exclude the more exotic dependencies for platforms we are not even supporting. would be a good step towards to reduce set of exemptions and be able to eventually get to 0 in some projects.

Jake-Shadle · 2022-11-19T11:13:37Z

Just FYI, one current limitation is laid out here EmbarkStudios/cargo-deny#324, my plan was just do each requested target in parallel and collate diagnostics, but it's not a huge issue.

bholley · 2022-11-23T19:45:02Z

@Gankra @mystor Do y'all have a sense of how straightforward it would be to implement option (1)? Conceptually it seems straightforward to just loop N times when constructing the crate graph but that may or may not line up easily with what we've got.

bholley · 2022-11-23T20:18:07Z

It'd be extra nice if we could use the same target list for cargo-vet and cargo-vendor, which would allow us to fix rust-lang/cargo#6179. Right now we have some gross a-hoc workarounds to avoid vendoring certain crates into mozilla-central.

hpenne · 2023-01-14T15:59:52Z

In the mean time, you can add suggest = false to that entry in your unaudited.toml to get it off the radar.

Is this unaudited.toml file a feature that has been removed? The documentation does not seem to mention it. I'm looking for a way to configure a crate as a permanent excemption (any version), for cases where a crate is pulled in to support a platform that we will never building for.

mystor · 2023-01-14T16:05:14Z

unaudited has been renamed to exemptions since #228 - you can find the entry in your config.toml and add the suggest = false option to it there.

bholley · 2023-01-14T16:08:27Z

If you want a permanent exemption across all versions, you can also add a policy entry for the crate and set criteria = []

hpenne · 2023-01-14T19:22:16Z

If you want a permanent exemption across all versions, you can also add a policy entry for the crate and set criteria = []

Perfect. Thanks! It could be useful to have this in the docs. I think many developers that do not target all platforms will benefit from this trick.

hpenne · 2023-01-16T13:37:28Z

Of course, the perfect solution would be "option 1" mentioned by @bholley above. That would be extremely useful. It seems that without this I'll have to add exemptions for all crates (but not their transitive dependencies) that end up in the dependency graph due to platforms that we're not building for. That creates a little bit of annoying extra work, especially when the tool is introduced to an existing project.

Since we only target Linux x86_64 machines, I ignored all the obvious Windows/WASM/Redox dependencies. In the future it seems like we should be able to set an explicit target for this (mozilla/cargo-vet#63). We import the Firefox audits to help reduce review load; the SecureDrop security model relies on Firefox already via Tor Browser. Refs #6500

djkoloski · 2023-03-14T21:17:26Z

Fuchsia would benefit from having configurable policies and exemptions. We have a set of host-only crates which are roughly our equivalent of dev-dependencies, and it would be nice to select a set of criteria for them like we could with dev-criteria. We don't use any complicated target configuration, just cfg(target_os = "fuchsia") and cfg(not(target_os = "fuchsia")).

Since we only target Linux x86_64 machines, I ignored all the obvious Windows/WASM/Redox dependencies. In the future it seems like we should be able to set an explicit target for this (mozilla/cargo-vet#63). We import the Firefox audits to help reduce review load; the SecureDrop security model relies on Firefox already via Tor Browser. Refs #6500

bholley added the p3 do later label May 17, 2022

bholley added p2 do soon and removed p3 do later labels Jan 16, 2023

legoktm mentioned this issue Jan 30, 2023

Determine plan for auditing Rust dependencies freedomofpress/securedrop#6500

Closed

quic-cbusold mentioned this issue Nov 25, 2024

cargo vet check only checks target-specific dependencies if build.target is used #579

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

implement "target" features #63

implement "target" features #63

Gankra commented May 3, 2022

Gankra commented May 3, 2022

Gankra commented May 3, 2022

bholley commented May 17, 2022

conradludgate commented Jun 17, 2022

bholley commented Jun 17, 2022

Gankra commented Jun 17, 2022 •

edited

Loading

bholley commented Jun 17, 2022

repi commented Nov 19, 2022

Jake-Shadle commented Nov 19, 2022

bholley commented Nov 23, 2022

bholley commented Nov 23, 2022 •

edited

Loading

hpenne commented Jan 14, 2023

mystor commented Jan 14, 2023

bholley commented Jan 14, 2023

hpenne commented Jan 14, 2023

hpenne commented Jan 16, 2023

djkoloski commented Mar 14, 2023

implement "target" features #63

implement "target" features #63

Comments

Gankra commented May 3, 2022

Gankra commented May 3, 2022

Gankra commented May 3, 2022

bholley commented May 17, 2022

conradludgate commented Jun 17, 2022

bholley commented Jun 17, 2022

Gankra commented Jun 17, 2022 • edited Loading

bholley commented Jun 17, 2022

repi commented Nov 19, 2022

Jake-Shadle commented Nov 19, 2022

bholley commented Nov 23, 2022

bholley commented Nov 23, 2022 • edited Loading

hpenne commented Jan 14, 2023

mystor commented Jan 14, 2023

bholley commented Jan 14, 2023

hpenne commented Jan 14, 2023

hpenne commented Jan 16, 2023

djkoloski commented Mar 14, 2023

Gankra commented Jun 17, 2022 •

edited

Loading

bholley commented Nov 23, 2022 •

edited

Loading