Define metrics for identifying abuse and abnormal behavior

[GeorgianaElena]

Context

Noting 5xx errors from user pods in general is a good indication that something is wrong.

For example, here are two images from the openscapes grafana that were particularly useful:

• Users over time

• CPU usage histogram over time

Ref: #328

Proposal

• define a few metrics that are particularly useful for identifying abuse and problematic abnormal behavior
• define some thresholds for these metrics

Updates and actions

No response

[GeorgianaElena]

Closing this, as I believe we are tracking current monitoring efforts in other, more actual issues.