Rework Scope metrics according to Prometheus conventions.

[juliusv]

• counters should end with _total
• elaborated and added units to help strings
• recommended for cache hit/miss metrics: track only the total and the
  hits and in separate metrics, since the most common query will be
  "hits / total"
• track all times in seconds (base units), which has become the standard
  recommendation
• other small changes

There could be more changes that would require more thinking (what
dimensions to use, summaries vs. histograms, etc.), but this is probably
enough controversial material already :)

[juliusv]

@tomwilkie

[tomwilkie]

Can you update sqs_control_router.go to use timeRequestStatus? Or would you mind if I do?

Will also need to redo our dashboards in a separate PR.

[juliusv]

Will do!

[juliusv]

Did that and fixed a bug in an existing metric (please double check):

scope/app/multitenant/sqs_control_router.go

Line 255 in c882882

sqsRequestDuration.WithLabelValues("GetQueueUrl", "500").Observe(float64(duration.Nanoseconds()))

was reusing both a duration and method string from an Observe() call above.

[tomwilkie]

Nice catch!