-
Notifications
You must be signed in to change notification settings - Fork 80
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
UI causes browser to run the system out of memory #5414
Comments
Did you have anything running in your UI? |
That's a lot of no auth request handler errors happening extremely fast, looking at the timestamps. |
The JS API's design is intended to make it impossible for it to accidentally queue up many requests like this on its own - until an auth refresh succeeds or fails, the new one won't be passed to For what its worth, the thread dump doesn't show any activity - and specifically nothing that shows a grpc call is presently being handled to refresh a token. |
We've found a way that this could happen in the JS API, it requires a table to fail before the disconnect, and the disconnect has to be long enough for the auth token to expire. The client then gets into a bad state of trying to reconnect the subscription for the failed table in a microtask "loop", resulting rapid calls to the server ... which then each fail with auth issues. I'll move this to deephaven-core and handle it there. |
Here's a simple set of steps to reproduce locally, simulating a network issue:
Expected, table stays broken, but the rest of the page keeps working Note that this might not be quite the same as what @rbasralian originally found, but it does have many of the same characteristics - fixing this hopefully will prevent the constant reconnect loop originally seen (but not yet reproduced). |
DHC reconnects are able to restore server streams to an existing session, but most of the JS API was written to assume that a lost connection requires rebuilding objects on the server by replaying operations. This fix handles cases where a table failed and then a network error occurred, causing the table to be stuck unable to reconnect, since the table has failed. Two bugs prevented this from working, both cases where after some operation couldn't be scheduled, a microtask would immediately try again, leading effectively to an infinite loop in the browser. Table subscriptions are fixed by first checking if the table is running, so can be subscribed, and table refetch is fixed by using null for its fetcher, and during refetch if the fetcher is null either fail right away with the existing fail message, or succeed right away. This fix currently makes it possible for a failed table on a reconnected worker to not signal that it is still failed - this will be addressed in a follow-up. Fixes #5414
DHC reconnects are able to restore server streams to an existing session, but most of the JS API was written to assume that a lost connection requires rebuilding objects on the server by replaying operations. This fix handles cases where a table failed and then a network error occurred, causing the table to be stuck unable to reconnect, since the table has failed. Two bugs prevented this from working, both cases where after some operation couldn't be scheduled, a microtask would immediately try again, leading effectively to an infinite loop in the browser. Table subscriptions are fixed by first checking if the table is running, so can be subscribed, and table refetch is fixed by using null for its fetcher, and during refetch if the fetcher is null either fail right away with the existing fail message, or succeed right away. This fix currently makes it possible for a failed table on a reconnected worker to not signal that it is still failed - this will be addressed in a follow-up. Fixes #5414
Something about the UI apparently causes Safari to use almost-inconceivable amounts of memory (the DHC UI is the only thing I had open in Safari here). It seems related to either stopping/restarting the DHC container or letting my computer go to sleep and wake back up (or maybe both). Given enough time, it causes my Mac to pop up a "your system is out of memory" dialog.
Whatever causes this also causes the server to repeatedly log "No AuthenticationRequestHandler registered for type Bearer":
Here is a server-side thread dump from when this was occurring: thread_dump.txt
Version info (this is a locally-built image off of commit
61ae61b6
):I don't have browser logs from when this was occurring but will try to get them if it happens again.
The text was updated successfully, but these errors were encountered: