The monitoring indicators of consul_rpc_queries_blocking maybe not accurate

[lambdaxs]

When filing a bug, please include the following headings if possible. Any example text in this template can be deleted.

Overview of the Issue

A paragraph or two about the issue you're experiencing.

Reproduction Steps

Steps to reproduce this issue, eg:

1.when I watch a key, the consul_rpc_queries_blocking value will +1.

2.when I use Ctrl+c to cancel the watch, the consul_rpc_queries_blocking not -1.

So, Is there a delay in the statistics of this indicator ？Emmm ...How many seconds will this delay?

[dnephin]

Thank you for the question! I believe this is happening because ctrl-c is not actually able to shutdown the blocking queries that are used by the watches. ctrl-c only stops the cli process, but there's no way to propagate that shutdown back to the blocking query. The RPC interface uses https://golang.org/pkg/net/rpc/ , which does not support cancellation.

The blocking query should time out after 10 minutes, at which time you should see the metric decrement.

We are slowly moving to gRPC instead of net/rpc. gRPC does support cancellation, so once all the RPC endpoints are ported, cancellation should work properly, and the metrics should be updated almost immediately.