Investigate Production Issues and complete Prod Release

[darunrs]

The prod release recently done filed in production. It highlighted three issues:

1. Coordinator logging is incomplete. Many actions are somehow not being logged.
2. Block Streamer ran out of memory.
3. Runner most likely also ran out of memory, though we couldn't see any explicit logs related to that. But, the machine was inaccesible through ssh.

The Prod Release was directly responsible for the third one. It inexplicably increased memory usage by Runner by A LOT. It needs to be investigated what change is causing this problem.

The first two are unrelated to the prod release. We've beefed up the Block Streamer machine for now.

[darunrs]

Issue presented itself in dev. Will do my testing in dev now.

[darunrs]

Tried commit 6038fe6 in dev and it ran, although I did see pretty high usage still. You can see the behavior in the image below. The left spike is when dev failed due to the error we saw in prod. The spike on the right is the above commit.

[darunrs]

The issue is that, while reverting to that commit worked, the spike is still quite high. In production, which has many more indexers, the spike was only to 11GB total in prdo, even with many more Indexers. Both dev and prod ran on the same commit. Although, I did reduce prefetch to 3 from 10 for prod before the revert.

[darunrs]

Increasing commit up to the implementation of Instrumentation reduced the memory usage by 4GB. Not sure why. I am going to pause dev or a couple minutes to let the messages back up and then start it again. I want to see if its the prefetch which is the difference maker here or not.

After some 10 minutes of Runner being off:

[darunrs]

Prod is suddenly working after I tried the prod release commit again. I don't know why. Will look into it more on Monday but I want to leave prod up with the changes since the performance enhancements will help.

[pkudinov]

We also had this ticket talking about memory leaks: #437

[pkudinov]

Next step: do a prod release commit