feat: 2nd attempt to split save_event_transaction into 3 queues

[lynnagara]

Retry #78868, this time with separate config
for region and control silo

[codecov]

Codecov Report

Attention: Patch coverage is 70.00000% with 3 lines in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/sentry/conf/server.py	62.50%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #78889      +/-   ##
==========================================
- Coverage   78.27%   78.25%   -0.02%     
==========================================
  Files        7122     7125       +3     
  Lines      313583   313629      +46     
  Branches    51178    51189      +11     
==========================================
- Hits       245452   245438      -14     
- Misses      61712    61759      +47     
- Partials     6419     6432      +13     

[markstory]

Looks good to me. Using server.conf to minimize the number of places is one suggestion I had.

[markstory]

If you wanted to avoid this you could set settings.CELERY_SPLIT_QUEUE_TASK_ROUTES in server.py in the existing conditional blocks based on the silo mode.

[lynnagara]

good point, updated

[fpacifici]

Do we have a place in the code base where we can validate the config and prevent a pod from starting with an invalid config?

My mistake when I implemented the router was to conclude Celery would instantiate the router at Sentry initialization https://github.com/getsentry/sentry/blob/master/src/sentry/celery.py#L127-L129 .

Though it seems that happened lazily thus we would see failures only when we would try to send messages.
I think we should make celery fail earlier if there is a bad configuration.

Consider creating the queues during Celery initialization as a follow up.