[Bug/Instability] Nodes running on develop keep getting stuck randomly every day or two. #3476
Description
This issue was initially seen by @UdjinM6 on April 10th, 2020.
When this occurs the node wasn’t receiving blocks and shutdown wasn’t clear
Originally running 26dec64, on ubuntu 16.04
Node was reporting
2020-04-10 00:31:17 socket sending timeout: 1201s
...
2020-04-10 00:31:17 socket sending timeout: 1201s
2020-04-10 00:31:22 socket sending timeout: 1201s
That node while running 52bf616 had earlier reported
2020-04-07 00:07:30 socket sending timeout: 1201s
2020-04-07 00:07:30 socket sending timeout: 1201s
2020-04-07 00:07:34 socket sending timeout: 1201s
...
2020-04-07 00:07:34 socket sending timeout: 1201s
cb33702 was the prior commit that had been being ran since mid January, so the problem(s) was likely introduced between cb33702 and 52bf616
#3456 which fixes some deadlocks was created with the intent to fix this issue, however, the issue persists