RPC endpoints do not support cancellation

[dnephin]

The RPC server uses net/rpc which does not support cancellation of requests (and also happens to have been deprecated frozen since 2016).

For requests that are consistently short lived, this lack of cancellation is not a significant problem. However, many of the RPC endpoints support blocking queries (long polling) which are expected to be long living requests (up to some timeout, defaults to 10 minutes). If the client performing the request (via the CLI, or HTTP API) attempts to cancel the request, the RPC endpoint keeps blocking until the timeout.

The symptoms of this problem are not obvious. It can look like a file descriptor or goroutine leak because the resources used by the request are held until the timeout. In extreme cases, where a client frequently cancels requests and starts new ones, this could waste a lot of resources on the server.

Some options for addressing this problem are:

1. streaming: Expand Streaming support to all blocking queries #6366 - replace all blocking queries with streaming. Streaming is built on gRPC which supports cancellation.
2. replace net/rpc with gRPC while continuing to use blocking queries (less work overall than using streaming)
3. Server temporarily stops responding - Stopping a blocking query leaves goroutines running and in-use file descriptors in the server #9647 (comment) - replace net/rpc with keegancsmith/rpc, however based on the initial analysis this doesn't seem like it will work with our implementation.

Issues that are likely caused by this problem:

• Server temporarily stops responding - Stopping a blocking query leaves goroutines running and in-use file descriptors in the server #9647
• The monitoring indicators of consul_rpc_queries_blocking maybe not accurate #10193