Deprecate sessions

[chadwhitacre]

Problem

There's a thing called "sessions" and it's been deprecated in SaaS/ST. We need to deprecate it in SH as well.
https://www.notion.so/04282577b3fd4640b99a2e1e4549e00c 🔒

So what's going on (more or less) is that we've retired a Clickhouse table called sessions in favor of one called metrics. The old table was for Release Health (although it may also have been used for a couple mobile-related numbers):

Release Health in recent versions of Sentry is configurable to use either table. However, any bugfixes to Release Health will be done in metrics and not backported to sessions. Since Release Health is not actively being developed right now there's not urgent pressure to get this migration done for self-hosted users, but if/when future work happens on it we'll be out of time.

What needs to be done?

In order to provide as much of a smooth transition as possible for self-hosted users, we'll need to dual write to the metrics and sessions tables in Clickhouse for 90 days (max retention for snuba). We're going to have so many docker containers... 😓
Metrics table is already in self-hosted, but is unused. The dataset readiness state is Complete in snuba.

Stage 1: Dual Write

• Add ingest consumers ingest-metrics and ingest-generic-metrics and billing-metrics-consumer (apparently dynamic sampling will fail if this isn't added)
• Add kafka topics ingest-metrics and ingest-performance-metrics
• Add subscription consumer metrics-subscription-results and generic-metrics-subscription-results
• Add snuba consumers metrics-consumer, generic-metrics-distributions-consumer, generic-metrics-sets-consumer, generic-metrics-counters-consumer, and subscriptions-scheduler-executor-metrics.
• Enable feature flags for metrics, or just delete the feature flags entirely. Are there any other feature flags that are relevant here?
  • organizations:transaction-metrics-extraction
  • organizations:metrics-extraction
  • organizations:dashboards-rh-widget
  • organizations:mep-rollout-flag
  • organizations:dashboards-mep
  • SENTRY_RELEASE_HEALTH = "sentry.release_health.metrics.MetricsReleaseHealthBackend"
  • SENTRY_RELEASE_MONITOR = "sentry.release_health.release_monitor.metrics.MetricReleaseMonitorBackend"

All of this is now done here: #2355

We need to warn users in the release these new docker containers in about upcoming changes to use metrics infrastructure after 90 days of ingesting events.

Stage 2: Cutover

3-6 months after self-hosted releases are dual writing

• Remove all sessions docker containers (snuba-sessions-consumer, snuba-subscription-consumer-sessions)

[chadwhitacre]

I believe this refers to release health sessions.

[chadwhitacre]

Meeting scheduled for next week to understand this.

[hubertdeng123]

Sessions are configured here:

self-hosted/docker-compose.yml

Line 267 in 5c77236

snuba-sessions-consumer:

[chadwhitacre]

https://www.notion.so/sentry/Dataset-Readiness-States-da12e741fe024a7c8916c7b36b94ed8f

[hubertdeng123]

Why is the billing-metrics-consumer needed?

It generates outcomes for transactions that are sampled away and only metrics were stored.
It is related to billing, but if it isn't running outcomes (org stats) will be off too once accounts are using dynamic sampling.

[hubertdeng123]

Seeing stuff in ClickHouse and Postgres related to metrics 😮. Using this script to test:
https://github.com/getsentry/sentry/blob/master/bin/send_metrics.py

[williamdes]

Remove all sessions docker containers (snuba-sessions-consumer, snuba-subscription-consumer-sessions)

Does it mean users can remove the mentioned containers before they are actually removed ?
Ref: #2271

[mmospanenko]

Any ideas how to fix this?

it overloads CPU

[hubertdeng123]

You should be fine to remove that container from your docker compose file if it's causing you problems.
I have a PR up for this right now: #2514