Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

server: add memory accouting for Statements gRPC endpoint #69032

Closed
Tracked by #64743
Azhng opened this issue Aug 17, 2021 · 3 comments
Closed
Tracked by #64743

server: add memory accouting for Statements gRPC endpoint #69032

Azhng opened this issue Aug 17, 2021 · 3 comments
Labels
A-sql-observability Related to observability of the SQL layer C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) no-issue-activity X-stale

Comments

@Azhng
Copy link
Contributor

Azhng commented Aug 17, 2021

Currently, the Statement (GET /_status/statements) performs RPC fanout to the entire cluster and buffers all response in memory. This can create issue where the total amount of data it buffers will exceeds the currently available memory.

We should introduce memory accounting in the RPC handler to abort the handler if the memory limit has been exceeded

Jira issue: CRDB-9415

@Azhng Azhng added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) A-sql-observability Related to observability of the SQL layer T-sql-observability labels Aug 17, 2021
@jordanlewis
Copy link
Member

I think the proper solution for this should be to introduce pagination to the statements endpoint. Perhaps the easiest way to do this would be to migrate to the API v2 framework.

@Azhng
Copy link
Contributor Author

Azhng commented Aug 23, 2021

Statement endpoint is essentially a hash aggregation. Introducing pagination won't really solve the problem here because we still need to complete the aggregation before pagination.

Alternative is that we can perhaps implement a Sort -> Merge in the RPC endpoint, which potentially can allow us do use the pagination infrastructure. Though this might be a bit tricky to get right.

@github-actions
Copy link

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-sql-observability Related to observability of the SQL layer C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) no-issue-activity X-stale
Projects
None yet
Development

No branches or pull requests

2 participants