Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snapshot messages over 100MB throw in JS API #4563

Open
Tracked by #5927
mattrunyon opened this issue Sep 27, 2023 · 1 comment
Open
Tracked by #5927

Snapshot messages over 100MB throw in JS API #4563

mattrunyon opened this issue Sep 27, 2023 · 1 comment
Labels
2023_noMilestone bug Something isn't working triage
Milestone

Comments

@mattrunyon
Copy link
Contributor

Description

This happened when downsampling a plot with nulls. At a certain size, the message from the server exceeds 100MB and the JS API throws an error as it receives 2 different payloads, but doesn't think there could be a 2nd payload after the 1st snapshot payload.

Steps to reproduce

  1. Run this Python in the web IDE
from deephaven import empty_table
from deephaven.plot.figure import Figure

t = empty_table(4000000).update(["Timestamp=epochMillisToInstant(i)","x=(double)i", "y=i%12==11?null:Math.sin(i)"])
f = Figure().plot_xy(series_name="Hello", t=t, x="Timestamp", y="y").show()

Expected results

  1. JS API emits a downsample finished event. Plot shows up

Actual results

  1. Plot shows an indefinite loader. JS console has a failed class cast exception and failed assertion. JS API never emits downsample finished/failed event.

Additional details and attachments

In WorkerConnection, the assertion of nextDeltaUpdatesinappendAndMaybeFlushis what fails. It seemsstartAndMaybeFlushis called w/isSnapshot=trueand then another message arrives and callsappendAndMaybeFlushfrom thestream.onData`

The reason this snapshot gets so big is when downsampling w/ nulls, nulls must be preserved. I'll open another ticket to add some way to limit the downsample output size (downsampling 4M pts to 1M is still awful for the web UI to try to plot in addition to the data transferred).

Versions

  • Deephaven: 0.28.1
@niloc132
Copy link
Member

#188 will "fix" this, though if the web UI is subscribing in cases where they need bigger payloads, something else is probably wrong.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2023_noMilestone bug Something isn't working triage
Projects
None yet
Development

No branches or pull requests

3 participants