Improve HTTP metrics to distinguish client cancelled requests in 503 responses

[bartekn]

What version are you using?

2.4.1

What did you do?

If client disconnects the horizon_http_requests_duration_seconds_count metric records this with status=503 label. It's possible that it can affect alerting. We should be able to distinguish client initiated disconnects from actual 503 responses sent because of ctx being cancelled.

What did you expect to see?

Maybe we should add another boolean label to horizon_http_requests_duration_seconds_count to be able to tell which 503 were a result of client disconnects.

[sreuland]

@bartekn , which metrics data model option is more preferable to capture client disconnect, since there is no standard HTTP code for client disconnects:

1. set new metrics label 'client_disconnected=true|false' and set existing label 'status=204'
   or
2. use '499', a non-standard HTTP Status Code proposed by Nginx which represents 'client disconnected', so metrics would just have different 'status=499' label when client disconnected.

[bartekn]

I think option 2 is great, much clearer and no new label dimension. I wasn't aware 499 existed, looks like nginx custom code but should be fine in our context.