Periodically resync proxies to agents #18050
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rosstimothy
force-pushed
the
tross/refresh_agent_proxies
branch
2 times, most recently
from
d7dbb72 to
3523747
Compare
fspmarshall
approved these changes
Nov 2, 2022
espadolini
approved these changes
Nov 3, 2022
The table below shows the size of a marshaled
|
Prior to #14262, resource watchers would periodically close their watcher, create a new one and refetch the current set of resources. It turns out that the reverse tunnel subsytem relied on this behavior to periodically broadcast the list of proxies to agents during steady state. Now that watchers are persistent and no longer perform a refetch, agents that are unable to connect to a proxy expire them after a period of time, and since they never receive the periodic refresh, they never attempt to connect to said proxy again. To remedy this, a new ticker is added to the `localsite` that grabs the current set of proxies from its proxy watcher and sends a discovery request to the agent. The frequency of the ticker is set to fire prior to the tracker would expire the proxy so that if a proxy exists in the cluster, then the agent will continually try to connect to it.
rosstimothy
force-pushed
the
tross/refresh_agent_proxies
branch
from
5bb90a5 to
66e48d3
Compare
rosstimothy
force-pushed
the
tross/refresh_agent_proxies
branch
from
b46bd52 to
66523d9
Compare
rosstimothy
force-pushed
the
tross/refresh_agent_proxies
branch
from
15e4581 to
fdde20d
Compare
|
@rosstimothy See the table below for backport results.
|
rosstimothy
added a commit
that referenced
this pull request
Nov 4, 2022
Prior to #14262, resource watchers would periodically close their watcher, create a new one and refetch the current set of resources. It turns out that the reverse tunnel subsystem relied on this behavior to periodically broadcast the list of proxies to agents during steady state. Now that watchers are persistent and no longer perform a refetch, agents that are unable to connect to a proxy expire them after a period of time, and since they never receive the periodic refresh, they never attempt to connect to said proxy again. To remedy this, a new ticker is added to the `localsite` that grabs the current set of proxies from its proxy watcher and sends a discovery request to the agent. The frequency of the ticker is set to fire prior to the tracker would expire the proxy so that if a proxy exists in the cluster, then the agent will continually try to connect to it.
rosstimothy
added a commit
that referenced
this pull request
Nov 4, 2022
Prior to #14262, resource watchers would periodically close their watcher, create a new one and refetch the current set of resources. It turns out that the reverse tunnel subsystem relied on this behavior to periodically broadcast the list of proxies to agents during steady state. Now that watchers are persistent and no longer perform a refetch, agents that are unable to connect to a proxy expire them after a period of time, and since they never receive the periodic refresh, they never attempt to connect to said proxy again. To remedy this, a new ticker is added to the `localsite` that grabs the current set of proxies from its proxy watcher and sends a discovery request to the agent. The frequency of the ticker is set to fire prior to the tracker would expire the proxy so that if a proxy exists in the cluster, then the agent will continually try to connect to it.
rosstimothy
added a commit
that referenced
this pull request
Nov 4, 2022
Prior to #14262, resource watchers would periodically close their watcher, create a new one and refetch the current set of resources. It turns out that the reverse tunnel subsystem relied on this behavior to periodically broadcast the list of proxies to agents during steady state. Now that watchers are persistent and no longer perform a refetch, agents that are unable to connect to a proxy expire them after a period of time, and since they never receive the periodic refresh, they never attempt to connect to said proxy again. To remedy this, a new ticker is added to the `localsite` that grabs the current set of proxies from its proxy watcher and sends a discovery request to the agent. The frequency of the ticker is set to fire prior to the tracker would expire the proxy so that if a proxy exists in the cluster, then the agent will continually try to connect to it.
rosstimothy
added a commit
that referenced
this pull request
Nov 7, 2022
* Periodically resync proxies to agents (#18050) Prior to #14262, resource watchers would periodically close their watcher, create a new one and refetch the current set of resources. It turns out that the reverse tunnel subsystem relied on this behavior to periodically broadcast the list of proxies to agents during steady state. Now that watchers are persistent and no longer perform a refetch, agents that are unable to connect to a proxy expire them after a period of time, and since they never receive the periodic refresh, they never attempt to connect to said proxy again. To remedy this, a new ticker is added to the `localsite` that grabs the current set of proxies from its proxy watcher and sends a discovery request to the agent. The frequency of the ticker is set to fire prior to the tracker would expire the proxy so that if a proxy exists in the cluster, then the agent will continually try to connect to it.
rosstimothy
added a commit
that referenced
this pull request
Nov 7, 2022
Prior to #14262, resource watchers would periodically close their watcher, create a new one and refetch the current set of resources. It turns out that the reverse tunnel subsystem relied on this behavior to periodically broadcast the list of proxies to agents during steady state. Now that watchers are persistent and no longer perform a refetch, agents that are unable to connect to a proxy expire them after a period of time, and since they never receive the periodic refresh, they never attempt to connect to said proxy again. To remedy this, a new ticker is added to the `localsite` that grabs the current set of proxies from its proxy watcher and sends a discovery request to the agent. The frequency of the ticker is set to fire prior to the tracker would expire the proxy so that if a proxy exists in the cluster, then the agent will continually try to connect to it.
rosstimothy
added a commit
that referenced
this pull request
Nov 7, 2022
Prior to #14262, resource watchers would periodically close their watcher, create a new one and refetch the current set of resources. It turns out that the reverse tunnel subsystem relied on this behavior to periodically broadcast the list of proxies to agents during steady state. Now that watchers are persistent and no longer perform a refetch, agents that are unable to connect to a proxy expire them after a period of time, and since they never receive the periodic refresh, they never attempt to connect to said proxy again. To remedy this, a new ticker is added to the `localsite` that grabs the current set of proxies from its proxy watcher and sends a discovery request to the agent. The frequency of the ticker is set to fire prior to the tracker would expire the proxy so that if a proxy exists in the cluster, then the agent will continually try to connect to it.
rosstimothy
added a commit
that referenced
this pull request
Nov 16, 2022
Moves `UpdateTrustedCluster` logging from debug to info so default logging level includes when admin operations are performed to establish or remove trust. Alters `remoteSite` such that it logs in the same manner as `localSite` Cherry-picks some of the availability changes made in #18050 to ensure that agents spawned for trusted clusters are more robust to connection issues.
rosstimothy
added a commit
that referenced
this pull request
Nov 18, 2022
* Improve site and trusted cluster logging and availability Moves `UpdateTrustedCluster` logging from debug to info so default logging level includes when admin operations are performed to establish or remove trust. Alters `remoteSite` such that it logs in the same manner as `localSite` Cherry-picks some of the availability changes made in #18050 to ensure that agents spawned for trusted clusters are more robust to connection issues. * Ensure metric `remote_cluster` reflects current state The metric wasn't properly updated when remote sites went offline of when remote cluster resources were removed. Any change to the remoteSite state or the remoteCluster resource are now accurately reflected in the metric. * Add tracking of outbound connections to remote clusters The metric `trust_clusters` existed and was exported, but was never used anywhere. Now when the `RemoteClusterTunnelManager` starts and stops agent pools it will create and delete a counter for the cluster. Within the `AgentPool` the metric is set to the number of connected proxies within `updateConnectedProxies`.
github-actions bot
pushed a commit
that referenced
this pull request
Nov 18, 2022
Moves `UpdateTrustedCluster` logging from debug to info so default logging level includes when admin operations are performed to establish or remove trust. Alters `remoteSite` such that it logs in the same manner as `localSite` Cherry-picks some of the availability changes made in #18050 to ensure that agents spawned for trusted clusters are more robust to connection issues.
rosstimothy
added a commit
that referenced
this pull request
Nov 18, 2022
Moves `UpdateTrustedCluster` logging from debug to info so default logging level includes when admin operations are performed to establish or remove trust. Alters `remoteSite` such that it logs in the same manner as `localSite` Cherry-picks some of the availability changes made in #18050 to ensure that agents spawned for trusted clusters are more robust to connection issues. * Ensure metric `remote_cluster` reflects current state The metric wasn't properly updated when remote sites went offline of when remote cluster resources were removed. Any change to the remoteSite state or the remoteCluster resource are now accurately reflected in the metric. * Add tracking of outbound connections to remote clusters The metric `trust_clusters` existed and was exported, but was never used anywhere. Now when the `RemoteClusterTunnelManager` starts and stops agent pools it will create and delete a counter for the cluster. Within the `AgentPool` the metric is set to the number of connected proxies within `updateConnectedProxies`.
rosstimothy
added a commit
that referenced
this pull request
Nov 18, 2022
* Improve site and trusted cluster logging and availability Moves `UpdateTrustedCluster` logging from debug to info so default logging level includes when admin operations are performed to establish or remove trust. Alters `remoteSite` such that it logs in the same manner as `localSite` Cherry-picks some of the availability changes made in #18050 to ensure that agents spawned for trusted clusters are more robust to connection issues. * Ensure metric `remote_cluster` reflects current state The metric wasn't properly updated when remote sites went offline of when remote cluster resources were removed. Any change to the remoteSite state or the remoteCluster resource are now accurately reflected in the metric. * Add tracking of outbound connections to remote clusters The metric `trust_clusters` existed and was exported, but was never used anywhere. Now when the `RemoteClusterTunnelManager` starts and stops agent pools it will create and delete a counter for the cluster. Within the `AgentPool` the metric is set to the number of connected proxies within `updateConnectedProxies`.
zmb3
pushed a commit
that referenced
this pull request
Nov 18, 2022
* Improve site and trusted cluster logging and availability Moves `UpdateTrustedCluster` logging from debug to info so default logging level includes when admin operations are performed to establish or remove trust. Alters `remoteSite` such that it logs in the same manner as `localSite` Cherry-picks some of the availability changes made in #18050 to ensure that agents spawned for trusted clusters are more robust to connection issues. * Ensure metric `remote_cluster` reflects current state The metric wasn't properly updated when remote sites went offline of when remote cluster resources were removed. Any change to the remoteSite state or the remoteCluster resource are now accurately reflected in the metric. * Add tracking of outbound connections to remote clusters The metric `trust_clusters` existed and was exported, but was never used anywhere. Now when the `RemoteClusterTunnelManager` starts and stops agent pools it will create and delete a counter for the cluster. Within the `AgentPool` the metric is set to the number of connected proxies within `updateConnectedProxies`. * update metrics docs * Update docs/pages/includes/metrics.mdx Co-authored-by: Alex Fornuto <alex.fornuto@goteleport.com> Co-authored-by: Alex Fornuto <alex.fornuto@goteleport.com>
zmb3
pushed a commit
that referenced
this pull request
Nov 18, 2022
* Improve site and trusted cluster logging and availability Moves `UpdateTrustedCluster` logging from debug to info so default logging level includes when admin operations are performed to establish or remove trust. Alters `remoteSite` such that it logs in the same manner as `localSite` Cherry-picks some of the availability changes made in #18050 to ensure that agents spawned for trusted clusters are more robust to connection issues. * Ensure metric `remote_cluster` reflects current state The metric wasn't properly updated when remote sites went offline of when remote cluster resources were removed. Any change to the remoteSite state or the remoteCluster resource are now accurately reflected in the metric. * Add tracking of outbound connections to remote clusters The metric `trust_clusters` existed and was exported, but was never used anywhere. Now when the `RemoteClusterTunnelManager` starts and stops agent pools it will create and delete a counter for the cluster. Within the `AgentPool` the metric is set to the number of connected proxies within `updateConnectedProxies`.
zmb3
pushed a commit
that referenced
this pull request
Nov 21, 2022
Moves `UpdateTrustedCluster` logging from debug to info so default logging level includes when admin operations are performed to establish or remove trust. Alters `remoteSite` such that it logs in the same manner as `localSite` Cherry-picks some of the availability changes made in #18050 to ensure that agents spawned for trusted clusters are more robust to connection issues. * Ensure metric `remote_cluster` reflects current state The metric wasn't properly updated when remote sites went offline of when remote cluster resources were removed. Any change to the remoteSite state or the remoteCluster resource are now accurately reflected in the metric. * Add tracking of outbound connections to remote clusters The metric `trust_clusters` existed and was exported, but was never used anywhere. Now when the `RemoteClusterTunnelManager` starts and stops agent pools it will create and delete a counter for the cluster. Within the `AgentPool` the metric is set to the number of connected proxies within `updateConnectedProxies`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Prior to #14262, resource watchers would periodically close their watcher, create a new one and refetch the current set of resources. It turns out that the reverse tunnel subsytem relied on this behavior to periodically broadcast the list of proxies to agents during steady state. Now that watchers are persistent and no longer perform a refetch, agents that are unable to connect to a proxy expire them after a period of time, and since they never receive the periodic refresh, they never attempt to connect to said proxy again.
To remedy this, a new ticker is added to the
localsitethat grabs the current set of proxies from its proxy watcher and sends a discovery request to the agent. The frequency of the ticker is set to fire prior to the tracker would expire the proxy so that if a proxy exists in the cluster, then the agent will continually try to connect to it.