dekaf: Track and report additional metrics, and improve connection stability/freshness #1655
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ability/freshness I observed some concerning behavior when switching `dekaf.estuary.dev` to advertise `dekaf.estuary-data.com`, namely all API requests to the upstream MSK broker were returning with error codes. A restart of Dekaf "fixed" the problem, and I ended up realizing that those connections were probably sitting open and idle for days. The theory is that even though we set TCP keepalive, the connections had gotten into a bad state. So I introduced a couple different mitigations: * All idle connections (that is, connections that were opened but have since been returned to the pool) get closed after a short period of time (60s in this case) * All idle connections that were opened over 30m ago get closed. The stateless usage pattern of connections should mean that this results in no connection lasting longer than roughly 30m. * A connection is "exercised" every time it's returned to the pool to check whether it's still open and able to communicate with the broker. New metrics: * `pool_size` (by broker): The number of connections open to this broker * `pool_available` (by broker): The number of open, free connections available for use * `pool_waiting` (by broker): The number of requests blocked waiting for a connection to become available * `pool_connection_avg_age` (by broker): the average age of a pool connection to this broker * `pool_connection_avg_idle` (by broker): the average time a connection to this broker has been idle * `api_call_time` (by api method): A histogram of the time it takes to serve a kafka protocol request
jshearer
force-pushed
the
jshearer/dekaf_metrics
branch
from
f6916df
to
f9d8b71
Compare
jshearer
requested review from
jgraettinger and
a team
and removed request for
jgraettinger
jgraettinger
approved these changes
Sep 27, 2024
| let reap_interval = Duration::from_secs(30); | ||
| let max_age = Duration::from_secs(60 * 30); | ||
| let max_idle = Duration::from_secs(60); | ||
| let reaper = tokio_util::task::AbortOnDropHandle::new(tokio::spawn({ |
There was a problem hiding this comment.
❤️ thanks for the explicit syntax
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I observed some concerning behavior when switching
dekaf.estuary.devto advertisedekaf.estuary-data.com, namely all API requests to the upstream MSK broker were returning with error codes. A restart of Dekaf "fixed" the problem, and I ended up realizing that those connections were probably sitting open and idle for days.The theory is that even though we set TCP keepalive, the connections had gotten into a bad state. So I introduced a couple different mitigations:
New metrics:
pool_size(by broker): The number of connections open to this brokerpool_available(by broker): The number of open, free connections available for usepool_waiting(by broker): The number of requests blocked waiting for a connection to become availablepool_connection_avg_age(by broker): the average age of a pool connection to this brokerpool_connection_avg_idle(by broker): the average time a connection to this broker has been idleapi_call_time(by api method): A histogram of the time it takes to serve a kafka protocol requestThis change is