Batch semaphore, causality tracking

[Aurel300]

This adds batch semaphores into Shuttle (not originally implemented by me) as the core primitive to be used for other synchronisation primitives. Vector clocks were added to the batch semaphore to track causality across acquire, release, and try_acquire calls.

Implementing other primitives using the batch semaphore will thus also cause causality to be tracked. The eventual goal is for primitives to only rely on the API of batch semaphore, without calls to e.g. ExecutionState. As a result, batch semaphore and the runtime internals can be pulled out of Shuttle into a separate shuttle-core crate, while both the stdlib and tokio primitives can be placed into new sibling crates, shuttle-stdlib and shuttle-tokio, respectively.

An open question is how to proceed with the crate restructuring. One option is for the crate called shuttle now to "become" shuttle-stdlib, as it will contain the same primitives, but this would require various re-exports. We could also publish a wrapper crate called shuttle (intended to be deprecated?) which will re-export items from both shuttle-core and shuttle-stdlib.

Planned for this PR still:

• Test cases.
  • Test case to show where causality tracking is not precise.
• Option for the semaphore to not be strictly fair, by waking all feasible waiters at once, then letting the scheduler resolve the race.

[sarsko]

Sweet.

[...] while both the stdlib and tokio primitives can be placed into new sibling crates, shuttle-stdlib and shuttle-tokio, respectively.

An open question is how to proceed with the crate restructuring. One option is for the crate called shuttle now to "become" shuttle-stdlib, as it will contain the same primitives, but this would require various re-exports. We could also publish a wrapper crate called shuttle (intended to be deprecated?) which will re-export items from both shuttle-core and shuttle-stdlib.

There's more.

Shuttle currently:

1. "Core": all the functionality to run the program under test, ie starting up, pausing and resuming tasks, shutting down, etc.
2. The schedulers (scheduler).
3. std-lib mirrors (sync,thread.rs, hint.rs, thread_local! in lib.rs).
4. future, ie a mix of futures::executor and tokio::task::JoinHandle.
5. rand mirror.
6. lazy_static mirror.
7. The "external" interface: lib.rs (minus thread_local! and lazy_static!) and current.rs

My opinion is that all of these should be its own crate.

The question then becomes whether something which is called Shuttle should stick around. I think the thing called Shuttle will be the crate corresponding to bullet number 7. There is a question of whether it should also provide reexports of some of the other crates. I have not thought about that enough to have a definitive opinion, but I am generally of the opinion that the correct approach is to use the wrapper crates over what is provided by Shuttle directly. I am also of the opinion that future should be deprecated (ie it should not be provided by Shuttle, but may be provided by some crate).

[jorajeev]

Yeah, think we need to add a causal dependency from the close event to subsequent Closed error returns. But we can do that in a followup.

[Aurel300]

Yes. I tried to implement this, but ran into issues with getting a clock in the close method, because it may happen as part of the cleanup process/drop handlers. We talked with @sarsko a bit about re-organising how drops happen, which would unblock this.

[bkragl]

Should we track this in an issue?

[sarsko]

nit: Do we need the batch_semaphore prefix for all of the tests — they are in the batch_semaphore.rs file after all.

[sarsko]

This shouldn't be should_panic, but ignore no ?

[Aurel300]

Currently this specifies how the semaphore actually behaves, i.e. it will panic this way. If it ever starts failing (and the check succeeds), then the behaviour has changed and the test should be updated by removing the should_panic.

[sarsko]

Yes. Why do we want to waste CPU cycles to confirm that we have not improved Shuttle? I like the mechanical-ness of "oh hey btw here's a test which now works", but that could also be gotten by running this test (and optionally other tests) in CI.

Not blocking for me, I'll leave it for others to decide.

[jorajeev]

Actually, I asked Aurel to add this, so we have a scenario showing that our causality tracking is imprecise.

[sarsko]

Actually, I asked Aurel to add this, so we have a scenario showing that our causality tracking is imprecise.

?

I don't disagree on the test but whether it should be ignore (and not should_panic) or should_panic (and not ignore)

[bkragl]

My two cents: with the explanation in the comment, the risk of someone mistaking this test as desirable behavior is pretty low. I think it's better to run the test than not to run it. If the test starts to break without addressing the issue, we'd like to know about it, instead of hoping that it still does what we want when running it manually.