You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Unfortunately, this can cause Python to run out of memory when the cursor yields hundreds of millions of records.
To Reproduce
Create a new table
Populate it with hundreds of millions of records
Create a cursor that will yield all of the records from this table. (Ex: SELECT * FROM HugeTable)
Fetch a handful of records from the cursor. (ie: cursor.fetchmany(100) )
Close the cursor
Environment
This bug is present regardless of environment; however, here's the environment that I was using:
MySQL Server
AWS RDS. (Presumably, the use Amazon Linux)
Server Version 8.0.23
MySQL Client
OS: Ubuntu Jammy
Python >= 3.7
mysqlclient 2.0.3 and mysqlclient 2.1.1
Additional Context
I can probably implement a fix for this once we confirm that it's a legitimate bug. At a high-level, I suspect the call to self.fetchall() probably needs to be replaced with a call to self.fetchmany(..). However, this might be complicated by structural details that I'm not aware of.
The text was updated successfully, but these errors were encountered:
Describe the bug
The
BaseCursor#close(..)
method invokesBaseCursor#nextset(..)
, which will invokeself.fetchall()
under certain circumstances:mysqlclient/MySQLdb/cursors.py
Line 83 in 8f0cbac
mysqlclient/MySQLdb/cursors.py
Line 134 in 8f0cbac
Unfortunately, this can cause Python to run out of memory when the cursor yields hundreds of millions of records.
To Reproduce
SELECT * FROM HugeTable
)cursor.fetchmany(100)
)Environment
This bug is present regardless of environment; however, here's the environment that I was using:
MySQL Server
MySQL Client
Additional Context
I can probably implement a fix for this once we confirm that it's a legitimate bug. At a high-level, I suspect the call to
self.fetchall()
probably needs to be replaced with a call toself.fetchmany(..)
. However, this might be complicated by structural details that I'm not aware of.The text was updated successfully, but these errors were encountered: