feat(crons): Refactor to record_clock_tick_volume_metric

[evanpurkhiser]

This changes how we're using the volume history. Previously we were intending to use the volume history to make a decision for a specific tick, we cannot do that since we'll actually need to look at historic volume metrics to determine if we've entered an incident or if we just had an abnormality in mean deviation.

• The clock_dispatch no longer includes a volume_anomaly_result. I will remove this from the sentry-kafka-schema package in an upcoming PR (ref(crons): Remove unused volume_anomaly_result from clock_tick sentry-kafka-schemas#349).

• Instead of evaluating a tick decision during dispatch, we now record the metrics for the timestamp we just ticked past into redis. This is done during the processing of the clock tick in the clock_tick_consumer.

• The clock_tick_consumer no longer reads the volume_anomaly_result into a TickVolumeAnomolyResult. We'll still do something with this since in the future we'll be evaluating a tick result decision based on the tick metrics and will need to dispatch mark_unknown when entering an incident. But for now I've removed this logic.

• I've also updated the pct_deviation metric (which is the one recorded into the redis key) to not be an absolute value, since we want to know which direction we've deviated in, we do not want to produce an incident in the scenario that we've increased in volume.

• I've removed the safe_evaluate_tick_decision instead of creating a safe_record_clock_tick_volume_metric since we're now running this logic in a consumer which can backlog if we do have some kind of issue. This wrapper only existed since it was in a hot path that could fail in an unrecoverable way. We've also had this code running for a while now with no problems, so it's safe to not be overly cautious.

Part of GH-79328

[codecov]

Codecov Report

Attention: Patch coverage is 95.23810% with 1 line in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/sentry/monitors/system_incidents.py	93.75%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #80605      +/-   ##
==========================================
- Coverage   78.36%   78.36%   -0.01%     
==========================================
  Files        7207     7207              
  Lines      318744   318737       -7     
  Branches    43909    43908       -1     
==========================================
- Hits       249782   249770      -12     
- Misses      62600    62603       +3     
- Partials     6362     6364       +2     

[evanpurkhiser]

Here's where we adjust the pct mean deviation to allow negative values