Incident Report: Apr 06, 2024

[dyc3]

Timeline

• 11:30 EST: 502 Bad Gateway being served to clients being routed directly to the monolith
• 13:02 EST: Discovered, not alerted
• 13:06 EST: nginx restarted
• 13:06 EST: service restored

Cause

nginx failed to resolve hostname for the monolith, and did not try to resolve it again

Fix

restart nginx

Follow up items

• create alerts for accessing OTT through the balancer and directly to the monolith
• investigate nginx config options to make it more reliable (upstream failover options?)