Improvements to node scheduler

[mwylde]

This addresses a few issues with the node scheduler:

• Previously we were sending the entire pipeline binary as part of a single gRPC message, but as our pipelines binaries have grown in size this can bump up against the max size limit. This PR moves the communication to a multi-part stream of 2MB chunks.
• If a node dies, the jobs running on it going into recovery. The first phase of this is to try to clean up the old cluster, which for the node scheduler involves telling the node to shut down the worker. Except that the node itself is down, and so can't do that. Now we no longer treat this as an error, so that recovery can continue under the assumption that if the node isn't responding any more, the worker probably isn't running either.

[jacksonrnewhouse]

We could probably bump the message size limits up to the point where that isn't an issue. Have a preference?

[jacksonrnewhouse]

If we wanted to be careful about making sure a pipeline was actually dead we'd coordinate this behavior with timeouts/heartbeats. Might not be worth it for what is likely not the main mode of deployment.

[mwylde]

Yeah, I think there are ways of making this more robust but waiting for a heartbeat does mean taking longer to recover failed pipelines.

There are basically two cases:

• The node process has died, in which case the workers will also have died (generally) because they are child processes of the node
• We have a network partition between the controller and the node

In case 1, we're doing the right thing. In case 2, the worker and the node will each die on their own as they fail to talk to the controller.

[mwylde]

We could probably bump the message size limits up to the point where that isn't an issue. Have a preference?

We actually can't, there's a hard limit in tonic and we've always had it configured as high as it can be set. It also looks like that limit has just been lowered to 4MB in the newest version of tonic.