Limit concurrency

[harmony-ek]

Do not spawn unbounded number of goroutines for parallel message processing. Instead, use a worker pool of 32 goroutines sharing the same intake queue. Log ("rx overrun") and tail-drop messages if the queue is full.

Tests

Local test pass; 100% coverage of the newly added core code (./msgq):

git checkout harmony-one/master
go test -count=1 -cover ./... > coverage-before.txt
git checkout limit_concurrency 
go test -count=1 -cover ./... > coverage-after.txt
sed -i.bak -E 's/  [0-9]+(\.[0-9]*)?s      /       /' coverage-*.txt
diff -U0 coverage-before.txt coverage-after.txt

--- coverage-before.txt 2019-10-08 16:13:42.000000000 -0700
+++ coverage-after.txt  2019-10-08 16:13:42.000000000 -0700
@@ -86 +86,2 @@
-ok     github.com/harmony-one/harmony/node     coverage: 15.4% of statements
+ok     github.com/harmony-one/harmony/msgq     coverage: 100.0% of statements
+ok     github.com/harmony-one/harmony/node     coverage: 14.3% of statements

TODO

• Add UTs

[fxfactorial]

Wondering if we should have go routine wrapper, like withMaxConcurrency(max int, caller_provide_cb)

[harmony-ek]

Wondering if we should have go routine wrapper, like withMaxConcurrency(max int, caller_provide_cb)

Thought about doing so, but didn't go for it because it would lose type safety: the intake channel's type must be made generic (chan interface{}) and the recipient would have to type-assert it from interface{} which opens a possibility of a panic when misused/abused (msg := item.(incomingMessage) where the received item is not an incomingMessage), and even with a type check (if msg, ok := item.(incomingMessage); !ok { ... }) it creates a spurious branch that hurts readability.

This is a cost-benefit trade off. I'd say factor the logic out if we do this maybe the third time.

(O generics, where art thou…?)

[rlan35]

LGTM, [sidenote, it would be better if it support prioritization scheduling of the messages. Consensus should take higher priority]