Skip to content

BaseCursor#close(..) can exhaust the system memory #560

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
briandamaged opened this issue Oct 25, 2022 · 0 comments · Fixed by #601
Closed

BaseCursor#close(..) can exhaust the system memory #560

briandamaged opened this issue Oct 25, 2022 · 0 comments · Fixed by #601

Comments

@briandamaged
Copy link

briandamaged commented Oct 25, 2022

Describe the bug

The BaseCursor#close(..) method invokes BaseCursor#nextset(..), which will invoke self.fetchall() under certain circumstances:

Unfortunately, this can cause Python to run out of memory when the cursor yields hundreds of millions of records.

To Reproduce

  1. Create a new table
  2. Populate it with hundreds of millions of records
  3. Create a cursor that will yield all of the records from this table. (Ex: SELECT * FROM HugeTable)
  4. Fetch a handful of records from the cursor. (ie: cursor.fetchmany(100) )
  5. Close the cursor

Environment

This bug is present regardless of environment; however, here's the environment that I was using:

MySQL Server

  • AWS RDS. (Presumably, the use Amazon Linux)
  • Server Version 8.0.23

MySQL Client

  • OS: Ubuntu Jammy
  • Python >= 3.7
  • mysqlclient 2.0.3 and mysqlclient 2.1.1

Additional Context

I can probably implement a fix for this once we confirm that it's a legitimate bug. At a high-level, I suspect the call to self.fetchall() probably needs to be replaced with a call to self.fetchmany(..). However, this might be complicated by structural details that I'm not aware of.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

1 participant