Fix multi-threaded access to non-TLS servers (eg - SNP nodes fetching endorsements from THIM)

[eddyashton]

#6830 includes a bunch of debugging hacks to try and get a repro of the failure in virtual, to isolate the segfault. Includes a tiny THIM shim server.

This is a cleaner version of that, hopefully including the minimal changes for a CI repro (multi-threaded SNP CI) and the minimal fixes (dispatch send_data by threads, and add a Mutex in QuoteEndorsementsClient).

[eddyashton]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[eddyashton]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[eddyashton]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[eddyashton]

2 open questions on this:

I think this is a sufficient fix, we have a green run where we previously had consistent reds. But also had a single red run on this final version of the code. I think that's something unrelatedly flaky in modules_test:

57:   1) polyfill
57:        isValidX509CertChain
57:          returns true for valid cert chains:
57:      Error: Timeout of 2000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/CCF/js/ccf-app/test/polyfill.test.ts)
57:       at process.processImmediate (node:internal/timers:478:21)

But will try a few more runs to confirm.

I was hoping to leave on the -DWORKER_THREADS=2, so that we maintain CI coverage of multi-threaded SNP runs, but it causes a significant slowdown in the tests. The CTest step takes ~35 minutes rather than ~23 minutes, pushing the whole Deploy ACI job worryingly close to the 1hr timeout. Almost all of the e2e tests run slower, I'm guessing because we're overloading the 4-core C-ACI instances with multiple parallel services and multiple nodes. If anyone has a quick bright idea here let me know, otherwise I'll turn off the -DWORKER_THREADS=2 and we'll lose automated test of this again, for now.

[eddyashton]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[eddyashton]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[eddyashton]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).