[CBR-475] adds MVar for any sqlite-operation

[kderme]

Description

Multithreading at SQlite is not the greatest. The intermediate libraries we use make things slightly worse. We already suffer from 2 known issues:

• unexpected exceptions because of race conditions (Unreliable sqlErrorDetails in parallelized environments IreneKnapp/direct-sqlite#77). This can make things like this not pattern match
  Left (Sqlite.SQLConstraintError Sqlite.Unique "tx_metas_outputs.meta_id, tx_metas_outputs.output_index") because we get the wrong exception and the node crashes with:

StorageFailure (SQLOtherError SQLite3 returned ErrorConstraint while attempting to perform step: not an error)
cardano-node: StorageFailure (SQLOtherError SQLite3 returned ErrorConstraint while attempting to perform step: not an error)

• withTransaction is not multithreaded. If 2 threads (which for example restore/sync at the same time) tries to use it, the node may crash. I tried this:

thread0 conn = 
    forM_ [1..5000] $ \n -> do
        withTransaction conn (return ())

thread1 conn = 
    forM_ [1..5000] $ \n -> do
        withTransaction conn (return ())

and I get different results like:
SQLite3 returned ErrorError while attempting to perform step: not an error
SQLite3 returned ErrorError while attempting to perform step: cannot start a transaction within a transaction
so even trying to handle the IS_BUSY error, we may get the wrong error because of the previous isssue!

Given the above and despite as cruel as it is, I think we should use our own MVars and use mutually exclusive uses of the connection.

Linked issue

Type of change

• 🐞 Bug fix (non-breaking change which fixes an issue)

How to merge

Send the message bors r+ to merge this PR. For more information, see
docs/how-to/bors.md.

[dcoutts]

LGTM. Serialising access to the connection on our side is much cleaner than relying on the sqlite serialised mode.

[parsonsmatt]

Fantastic, love the tests.

[kderme]

thanks @parsonsmatt for the fix. I cherry-picked only the unit-tests, without the MVars here #3872 and it fails pretty convincingly. I missed the failing stylish-Haskell frm Hydra, now fixed 🙏 . I added bindings for sqlite_threadSafe3 and tests indicate THREADSAFE=1, which means Serialized mode https://www.sqlite.org/threadsafe.html.

[kderme]

🤦‍♂️ 🤦‍♂️