concurrent.futures._base.CancelledError in prepared_stmt.fetch from random queries in sqlalchemy

[pankajkumar229]

I am getting a cancellederror and not sure how to debug it. I tried a number of suggestions outlined in sqlalchemy/sqlalchemy#6467 but cannot overcome the issue.

The stack for the error looks like this:
File "/product/labgrader/nbgrader/api.py", line 2093, in upsert_student_to_course
course_student_association = await self.find_association_by_ids(course_id, student_id)
File "/product/labgrader/nbgrader/api.py", line 1819, in find_association_by_ids
.where(Course_user_association_table.course_id == course_id))
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/ext/asyncio/session.py", line 217, in execute
**kw
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 136, in greenlet_spawn
result = context.switch(value)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1692, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1614, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 326, in _execute_on_connection
self, multiparams, params, execution_options
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1491, in _execute_clauseelement
cache_hit=cache_hit,
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1846, in execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 2030, in handle_dbapi_exception
util.raise(exc_info[1], with_traceback=exc_info[2])
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 207, in raise
raise exception
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1803, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 461, in execute
self._prepare_and_execute(operation, parameters)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 76, in await_only
return current.driver.switch(awaitable)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 129, in greenlet_spawn
value = await result
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 435, in _prepare_and_execute
self._handle_exception(error)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 370, in _handle_exception
self._adapt_connection._handle_exception(error)
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 665, in _handle_exception
raise error
File "/usr/local/lib/python3.7/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 423, in _prepare_and_execute
self._rows = await prepared_stmt.fetch(*parameters)
File "/usr/local/lib/python3.7/site-packages/asyncpg/prepared_stmt.py", line 176, in fetch
data = await self.__bind_execute(args, 0, timeout)
File "/usr/local/lib/python3.7/site-packages/asyncpg/prepared_stmt.py", line 242, in __bind_execute
lambda protocol: protocol.bind_execute(
File "/usr/local/lib/python3.7/site-packages/asyncpg/prepared_stmt.py", line 230, in __do_execute
return await executor(protocol)
File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
concurrent.futures._base.CancelledError

I have tried setting prepared_cache_size , prepared_statement_cache_size to 0 or using a different pool type but nothing seems to solve the error.

• asyncpg version: 0.27.0
• PostgreSQL version: POSTGRES_12_11.R20220710.01_08
• Do you use a PostgreSQL SaaS? If so, which? Can you reproduce :
  the issue with a local PostgreSQL install?: Google cloud postgres , we have not tried local
• Python version: 3.7
• Platform: linux
• Do you use pgbouncer?: No
• Did you install asyncpg with pip?: Yes
• If you built asyncpg locally, which version of Cython did you use?: We did not build locally

[pankajkumar229]

With some debugging, I could overcome this bug. The main problem was that I was running from within the tornado test framework which was calling timeout before waiting for the test method to complete. I moved all logic to the setup phase of the test and things started to work