Filter Duplicate Input Execution

[riesentoaster]

See #2759.

Some mutators report MutationResult::Mutated, even if nothing actually changes about the input. HashMutator is a wrapper around other mutators that hashes inputs pre- and post-mutation to ensure MutationResult::Mutated is only reported if something actually changed.

This may be worth using on slow targets, where the hashing is quicker than the unnecessary additional executions of the target for previously tried inputs.

[domenukk]

HashFilterMutator?

Or even HashMutationFilter

[domenukk]

As I stated in the discussion thread, I think a method for rejecting inputs that were already tried would be more useful (but I don't know your use case, so..)
Maybe using a Bloomfilter on the executor, or similar..

[riesentoaster]

As I stated in the discussion thread, I think a method for rejecting inputs that were already tried would be more useful (but I don't know your use case, so..)

I'm targeting the TCP/IP stack of an OS, so each execution takes in the order of magnitude of 1s, although most of that is spent in wait states (hence previous work like overcommit). Even still, the added runtime of this would be nothing compared to the execution, so this felt like an easy win.

Maybe using a Bloomfilter on the executor, or similar..

Something like this would definitely further improve the situation. Do you suggest creating a wrapping executor that returns either ExitKind::Ok or a new ExitKind::Skipped if the input was previously evaluated? This seems like a bodged-on solution as well though, since observers/feedbacks still run — we probably don't even want to call the executor in such cases.

Tracing this back it seems most appropriate in the stage? But that seems not that generic. So maybe in Fuzzer (resp. it's Evaluator impl)?

I'm also not sure if there's an opportunity here to combine this somehow with CentralizedLauncher?

[domenukk]

I think it could simply wrap an executor, yeah. And have an extra observation that's "skipped" -if it's true the testcase isn't interesting. Should be easy enough to do.

\We can still merge this PR as well, but the feedback should be renamed IMHO.

[riesentoaster]

How about something like this?

[riesentoaster]

I'll do some performance comparisons later today. Initial runs suggest that adding even a 10µs sleep to the harness reduces the performance penalty to <5%.

I might also see how many duplicate inputs actually appear. But for now I feel like for slow targets this very well might be worth using.

[riesentoaster]

Alright, some performance tests. Running against the libfuzzer_libpng example fuzzer:

1. Without the bloom filter, I'm getting a throughput of ~100k/s.
2. With the bloom filter, I get ~85k/s.
3. The rate of duplicate vs. new inputs increases over time, after 1min and 2min it was 0.6%, after 3min it was 1%, after 4min 2%, after 5min 4.4%, after ~7min it reaches 10%, after ~13mins 40%. At this point I assume most inputs are going to be duplicates.

All these numbers obviously depend on the exact fuzzers:

1. When the corpus count reaches a plateau, duplicate inputs become increasingly likely
2. When the number of possible mutations is small, duplicate inputs are more likely
3. If the execution time of the target is larger, the added runtime may be less than the runtime saved from not executing an input twice.
4. The bloom filter requires quite a bit of memory, so if that is your limitation, not using it and instead spawning additional instances may be worth it

Overall, I feel like this may be worth having in the library.

Btw: There is no easy way of adding metadata to the state such that it is printed by monitors, right? Otherwise, calculating the number/rate of duplicates may be an interesting addition.

[domenukk]

There is an easy way, using UserStats see... however other things do UserStats. For example the stability in the calibration stage.

[riesentoaster]

Alright, I've implemented some of that. I have changed the following stages to use the new evaluate_filtered:

• GenStage
• PowerMutationalStage
• StdMutationalStage
• MultiMutationalStage
• SimpleConcolicMutationalStage

Notable unchanged stages (please check them yourself, I know only very little about the many different stages):

• StdTMinMutationalStage (it calls evaluate_execution directly, and I'm generally unsure if this should call the filtered or unfiltered method)
• CalibrationStage needs to call the unfiltered version, right?

[domenukk]

I would add a with_input_filter method and then you just provide a BloomInputFilter there (or we keep this one as extra constructor)

[riesentoaster]

I've added a generic version and changed both new and with_bloom_input_filter to use this one.

[domenukk]

Now it looks very good! Just left some minor nitpicks.
Yes, tmin and calibration need to execute unfiltered.

Thank you and merry christmas!

[tokatoka]

merry christmas 🎅

[tokatoka]

is this I: Hash necessary? if not can you delete?
(always, keep the contraints minimal)

[riesentoaster]

Yes, the bloom filter checks presence based on the hash value of the input.

[riesentoaster]

Thank you for the quick responses! I hope you had relaxing holidays anyways :D

The remaining CI issues seem unrelated(?)