Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate performance when writing barrage metadata of large sparse tables #6180

Open
niloc132 opened this issue Oct 8, 2024 · 0 comments
Assignees
Labels
barrage barrage-wrkr2wrkr core Core development tasks feature request New feature or request
Milestone

Comments

@niloc132
Copy link
Member

niloc132 commented Oct 8, 2024

Discovered researching #6177.

Subscribing (viewport or full table) to a large sparse table requires serializing the full rowset and appending that to the barrage metadata message. Presently we write the "added" to the FlatBufferBuilder early, then add other rowsets and messages, which can result in a case where we resize (allocating a new ByteBuffer and copying existing data) from 1024 to "very big" early, and then resize several more times as we append more data to send.

In theory we could start off at a better size by measuring how big each vector will be (caveat: this won't work quite the same with viewports). Another option could be to write the added rows as late as possible, so that large sparse snapshots only pay a fraction of the cost.

Example code:

from deephaven import empty_table
input=empty_table(100_000_000).view("A=ii").where("A % 2 == 0")

Fetching this in the browser with a small viewport results in around 100-200ms in processing to copy bytebuffers:
screenshot923

With viewport changes to send minimal updates outside of the visible range, this will only impact full table subscriptions like charts, worker-to-worker.

@rcaudy rcaudy added core Core development tasks and removed triage labels Oct 9, 2024
@rcaudy rcaudy added this to the Backlog milestone Oct 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
barrage barrage-wrkr2wrkr core Core development tasks feature request New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants