Fuzzing of private APIs

[dbrgn]

Currently cargo-fuzz imports the fuzz target crate using extern crate. This exposes only public interfaces.

Is there a way to fuzz internal (pub(crate)) APIs?

[frewsxcv]

The only way I know of would be to have cfg blocks on the visibility keywords so they're conditionally public or private, but that's pretty hacky and messy.

Ideally, Rust would have a clearer story for custom test frameworks, but that's not really a thing right now.

[nathaniel-brough]

For posterity, I went down a bit of a rabbit hole, trying to see the current state of this now. Here are a couple of findings;

There was an attempt made to support custom test frameworks (incomplete/abandoned):

• docs
• Tracking issue for eRFC 2318, Custom test frameworks rust-lang/rust#50297

There is also some interesting work done in defmt-test that seems to support a custom test framework. This is more targetted at bare-metal testing and applications. 1 test goes to a single binary which is uploaded to a microcontroller. It might be possible to adapt the approach taken by defmt-test/probe-rs to work with cargo-fuzz.

[Ekleog]

bolero-based fuzzers have support for this functionality, by leveraging the cargo test binaries and passing libFuzzer arguments through an environment variable rather than on the command line. I'd suggest this as a way to fuzz private functions currently :)

@fitzgen Do you think such a feature would be in scope for cargo fuzz? I can't promise a timeline for when I'd be doing it, but I'd definitely be interested to know if I should think of this feature as a PR to cargo fuzz or as a new project :) (I have my own issues with bolero, that I mostly fixed upstream there, but the remaining ones make me feel like there's too much to be done and it'd be easier to start from a clean state)

I'll add that another feature bolero gets, which is very nice, is the fact that on each unit test run, the fuzzers will then run for a few rounds, including re-running the crashes and corpus folders. This is pretty cool to help detect issues before the changes reach the "undergoing fuzzing" phase :)

[fitzgen]

cargo fuzz builds Rust sources with cfg(fuzzing) enabled, so you can conditionally publicly export APIs based on that config if you want to fuzz them.

[Ekleog]

Sure, but it makes things much less convenient than just writing a proptest-like unit test that'll automatically be turned into a fuzzer.

Should I consider your answer as "implementing something like what bolero does is out of scope for cargo-fuzz"? I'd totally understand, as it's a pretty significant addition, though I personally consider it a required feature for my use cases :)

[fitzgen]

I could see the libfuzzer-sys crate supporting a way to run libfuzzer inside regular cargo test rather than defining main but I don't see cargo fuzz having anything to do with that.

[Ekleog]

Well, the thing is that in order to run the fuzzer inside a cargo test, then you need to change the way you pass the arguments. To give an example, if example-12345678 is the cargo test binary inside target, to run the fuzzer one'd need to run example-12345678 --exact [the fuzzer test name]. So in turn, to run with eg. -j 8, the arguments would need to be passed like (the bolero solution) BOLERO_LIBFUZZER_ARGS="-j 8" example-12345678 --exact fuzzer_test_name.

So you're entirely right that libfuzzer-sys would likely need to change a bit too, but cargo fuzz would also need to be adjusted to call the cargo test binary in the right way.

Does that make sense?