Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix memory leak on large responses #177

Closed
mostafa opened this issue Mar 3, 2023 · 1 comment
Closed

Fix memory leak on large responses #177

mostafa opened this issue Mar 3, 2023 · 1 comment
Assignees
Labels
bug Something isn't working triage Triage based on the content

Comments

@mostafa
Copy link
Member

mostafa commented Mar 3, 2023

A table with 158369 rows and the response size of 67735631 bytes causes subsequent queries to hang. This could be related to #175.

Observations

  1. The query, select * from table1;, causes the client to get stuck and doesn't return any meaningful results, while the same query with LIMIT will work, that is select * from table1 limit 158369;.
  2. If the limit is set to 7300 or less, the result is cached correctly and later read from the cache, but if it exceeds that amount, the cache plugin fails to receive the data, hence it doesn't caching anything and returns this errors. This is related to Reading large inputs from server causes plugins and Run to hang on onTrafficFromServer hook #175.
    2023-03-09T17:44:35+01:00 ERR Hook returned an error error="rpc error: code = ResourceExhausted desc = grpc: received message larger than max (4492364 vs. 4194304)" hookName=HOOK_NAME_ON_TRAFFIC_FROM_SERVER priority=1000
  3. I tried bytes.Buffer with io.CopyX and a few other methods, but none of them work as expected. They get stuck waiting for an EOF, but PostgreSQL wire protocol doesn't work with EOFs, because it send the message size in the message.
  4. I don't want to bind GatewayD to any specific protocol for reading/writing server messages, as it makes it hard to extend and it defeats the purpose of using and extending plugins. That's why I rely on plugins to decode/encode server messages and they need to import the decoder/encoder/parser from the SDK.
  5. Setting deadlines for receive and send temporarily helps fix the issue, but it introduces another, that I filed here: Fix bug in handling connection timeouts #199.
  6. The last try to use bytes.Buffer worked: Improve client performance #200.
@mostafa mostafa self-assigned this Mar 3, 2023
@mostafa mostafa converted this from a draft issue Mar 3, 2023
@mostafa mostafa added the enhancement New feature or request label Mar 3, 2023
@mostafa mostafa added this to the v0.5.x milestone Mar 3, 2023
@mostafa mostafa moved this from 🆕 New to 📋 Backlog in GatewayD Core Public Roadmap Mar 7, 2023
@mostafa mostafa moved this from 📋 Backlog to 🏗 In progress in GatewayD Core Public Roadmap Mar 9, 2023
@mostafa mostafa added bug Something isn't working and removed enhancement New feature or request labels Mar 9, 2023
@mostafa mostafa moved this from 🏗 In progress to 📋 Backlog in GatewayD Core Public Roadmap Mar 9, 2023
@mostafa mostafa removed this from the v0.5.x milestone Mar 9, 2023
@mostafa mostafa added the triage Triage based on the content label Mar 9, 2023
@mostafa
Copy link
Member Author

mostafa commented Oct 31, 2023

This is resolved in #343. Pick this up if the issue persists.

@mostafa mostafa closed this as completed Oct 31, 2023
@mostafa mostafa moved this from 📋 Backlog to 🎉 Done in GatewayD Core Public Roadmap Oct 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage Triage based on the content
Projects
Development

No branches or pull requests

1 participant