test(metrics): Less flaky graceful shutdown #1203
Conversation
| @@ -572,18 +572,23 @@ def test_graceful_shutdown(mini_sentry, relay): | |||
| metrics_payload = f"bar:17|c" | |||
| future_timestamp = timestamp + 60 | |||
| relay.send_metrics(project_id, metrics_payload, future_timestamp) | |||
|
|
|||
| sleep(1) # Give Relay some time to fetch project config | |||
There was a problem hiding this comment.
My problem is that in that time, Relay may already have flushed out the metrics, making this test less meaningful as then there's nothing to do in shutdown. We may want to bump the flush interval (not sure which setting that is)
There was a problem hiding this comment.
The "bar" metric would only be flushed after initial_delay (that was 10 seconds, have now bumped it to 100 seconds).
There was a problem hiding this comment.
I'm starting to suspect that the actual problem here is in Relay rather than the tests and is the same cause as the dropped buckets we observe on shutdown. Relay immediately flushes the buckets when it receives the shutdown signal and drops the buckets if project states are not yet fetched (or outdated). Instead, it should try to fetch the states during the grace period, still.
|
Closing this in favor of #1205. |
test_graceful_shutdownfailed in ~50% of runs. With this change, it passes 100 subsequent runs, at least on local machine.#skip-changelog