PVF: Throw error if worker process cannot spawn necessary threads

[mrcnski]

ISSUE

Overview

This issue is about the following quote from tokio::task::spawn_blocking (used by us to create threads in the worker process):

Tokio will spawn more blocking threads when they are requested through this function until the upper limit configured on the Builder is reached. After reaching the upper limit, the tasks are put in a queue. The thread limit is very large by default, because spawn_blocking is often used for various kinds of IO operations that cannot be performed asynchronously. When you run CPU-bound code using spawn_blocking, you should keep this large upper limit in mind.

Why is this an issue? Because for deterministic results we require that all worker threads are spawned at the same time, and not staggered. For example, if the CPU time monitor thread is late to start, we may timeout later than other validators, and accept candidates that others do not. (Also, we may at some point start using the measurements from the memory stats thread to reject PVFs, though right now this is preparation-only so inaccurate measurements cannot lead to disputes.)

Proposal

If we cannot spawn all the threads immediately, we should return an internal error (i.e. not dispute). This would lead to a retry (since paritytech/polkadot#7011).

That said, as the default limit in tokio is very high, it is not clear how it can ever be reached in practice. Each execution job uses 2 threads, prepare jobs use 3. So I would say this is low priority, but something to keep in mind.

[jpserrat]

Hey @mrcnski, I'm going to start working on this one now. I'm probably gonna take a little longer just to get the context on this one haha. Let me know if you have any suggestions.

[mrcnski]

Hey @jpserrat! As there might be a refactor of relevant code coming soon with this issue, I would suggest waiting on this one right now. If you want, you can try to think about how to approach this in the meantime - I would suggest familiarizing yourself with the current code that deals with threads and learning about Rust's synchronization primitives, that might give you some ideas ;) Otherwise I can help you select a different issue. Up to you!

[jpserrat]

I'm going to follow your advice regarding the current code and Rust threads. Thank you very much for your guidance. It would also be great if you could recommend a different one.

[mrcnski]

@jpserrat Awesome! This one would be pretty quick and easy: #695. It may not end up being needed, but I think it probably will, and definitely wouldn't hurt to have. You can even add a sanity-check before each execution to make sure that the current node version matches the version recorded in the filename.

This is another easy one: #622

I found a couple more but they are somewhat non-trivial, would be a big help if you dove into them though:

1. https://github.com/paritytech/polkadot/issues/7042
2. Preparation errors should no longer be a InvalidCandidate variant #733