Fan out events in async mode for async recordings. #4696
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
a-palchikov
reviewed
Nov 4, 2020
fspmarshall
approved these changes
Nov 4, 2020
webvictim
reviewed
Nov 4, 2020
klizhentas
force-pushed
the
sasha/nohang
branch
2 times, most recently
from
de28d77
to
0082311
Compare
|
@webvictim can you try this commit on your hanging server please? Please also set the environment variables to show connection state: |
a-palchikov
approved these changes
Nov 6, 2020
awly
reviewed
Nov 7, 2020
klizhentas
force-pushed
the
sasha/nohang
branch
from
0082311
to
c555bcc
Compare
Closed
|
@awly @a-palchikov @fspmarshall @russjones hey folks, please take another look. I have added some defensive logic to make sure SSH connections never hang due to audit at the expense of loosing a portion of the events. I've been struggling to find a balance between security and audits, but decided to get on the side use not being waken up with S1s over hanging sessions. To compensate this, I propose #4755 in 5.1 |
|
@webvictim can you give this branch (fresh bulid) another try? It should never hang your server |
a-palchikov
reviewed
Nov 9, 2020
|
@klizhentas I have just deployed your second update - I'll let you know. The previous branch that you asked me to try before seemed stable for a while, but then when I tried to log in again today after a couple of days idle it turned out it was hung again. Here's the |
awly
approved these changes
Nov 9, 2020
|
@webvictim these goroutine dumps you provide are super helpful - thanks for collecting them for me! |
This commit fixes #4695. Teleport in async recording mode sends all events to disk, and uploads them to the server later. It uploads some events synchronously to the audit log so they show up in the global event log right away. However if the auth server is slow, the fanout blocks the session. This commit makes the fanout of some events to be fast, but nonblocking and never fail so sessions will not hang unless the disk writes hang. It adds a backoff period and timeout after which some events will be lost, but session will continue without locking.
klizhentas
force-pushed
the
sasha/nohang
branch
from
c555bcc
to
a49eec9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This commit fixes #4695.
Teleport in async recording mode sends all events to disk,
and uploads them to the server later.
It uploads some events synchronously to the audit log so
they show up in the global event log right away.
However if the auth server is slow, the fanout blocks the session.
This commit makes the fanout of some events to be fast,
but nonblocking and never fail so sessions will not hang
unless the disk writes hang.
It also adds ability to debug GRPC connection state
when running in debug mode.
To start sending GRPC connection state logs,
set environment variables:
GRPC_GO_LOG_SEVERITY_LEVEL=info GRPC_GO_LOG_VERBOSITY_LEVEL=99 teleport start -d