feat(fuzz): WorkerCorpus for multiple worker threads

[yash-atreya]

Motivation

towards #8898 (#11842)

Solution

• Replaces CorpusManager with WorkerCorpus
• WorkerCorpus is the corpus to be used by parallel worker threads.
• Each WorkerCorpus has an id: u32. The master worker has the id = 0.
• The WorkerCorpus's share their in_memory_corpus amongst each other via the file system using a star-pattern where the master worker (id = 0) is in the center.
• The corpus_dir is now organized as:

corpus_dir/
    worker0/ - // master
        sync/
        corpus/
    worker1/
        sync/
        corpus/
    worker2/
        sync/

• Each non-master worker exports their corpus to worker0/sync/ - See fn export in 089f30b
• The master worker distributes it's worker0/corpus entries (which includes entries from all workers when synced) to each workers sync/ directory - See fn distribute in d4200e4
• Each worker then pulls the new corpus entries from their respective corpur_dir/workerId/sync dir into corpus_dir/workerId/corpus if it leads to new coverage and updates it's history_map - See fn calibrate in 488d09d
• In fn calibrate we are fetching the new corpus entries from the workers sync/ dir and replaying the tx sequences to check if they lead to new coverage for this particular worker. If it does then we're updating history_map.
• The pub fn sync introduced in e9d8d3c handles all of the above.

Note: This PR does not address parallelizing the fuzz runs, only prepares for it. Opened for initial feedback on the approach.

PR Checklist

• Added Tests
• Added Documentation
• Breaking changes

[0xalpharush]

I think this is fine, but may result in some corpus entries not getting sync'd e.g. due to a crash or ctr+c and restart. If you persist the last sync'd timestamp, it can recover from restarts by checking if there are newer entries written before a sync could occur

[DaniPopes]

please merge master, this is still using old CI runners

[onbjerg]

Do we continually write to the corpus directory? This is very expensive as we not only iterate a directory and read the files, but we also (if gzip is enabled) do decompression over and over, potentially of the same file. It feels like the corpus should be default in-memory, and we only write at the end.

[0xalpharush]

This is just happening at start up. The entries are held in memory so long as they don't exceed a configurable limit and then flushed to disk (and compressed if it's enabled).

Your point does still stand elsewhere. IIUC workers share compressed corpus entries so they potentially are repeatedly decompressing the same files. Moving compression to the very end would resolve this

[0xalpharush]

Checking the entire sequence prior to executing may make sense as I think currently new_coverage_on_sync can be set and a later tx contains an invalid selector. Ideally, we could just drop those txs and see if we can recover and retain the modified, valid corpus entry

[0xalpharush]

Overall looks pretty good to me. I left a few comments and there are some outstanding. I would recommend adding a config to turn syncing off and also a way to make it occur more/less frequently.

[DaniPopes]

Merged into #12713.