PYTHON-4584 Add length option to Cursor.to_list for motor compat #1791
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This reverts commit 69e7257.
Jibola
requested changes
Aug 13, 2024
Comment on lines
403
to
409
| res: list[_DocumentType] = [] | ||
| remaining = length | ||
| while self.alive: | ||
| if not await self._next_batch(res): | ||
| if remaining != 0 and not await self._next_batch(res, remaining): | ||
| break | ||
| if length != -1: | ||
| remaining = length - len(res) |
Contributor
There was a problem hiding this comment.
In this case, I think someone can pass -2 and it be valid. Should we also throw a ValueError here as well?
Member
Author
There was a problem hiding this comment.
I'm going to switch to Optional[int], per Shane's recommendation.
ShaneHarvey
reviewed
Aug 13, 2024
test/test_cursor.py
Outdated
| c = self.db.test.aggregate([{"$changeStream": {}}], maxAwaitTimeMS=1) | ||
| self.addCleanup(c.close) | ||
| docs = c.to_list(2) | ||
| self.assertEqual(len(docs), 2) |
Member
There was a problem hiding this comment.
How long does this test take on a sharded cluster?
Member
Author
There was a problem hiding this comment.
It doesn't work on sharded, I'll make a separate test for CommandCursor
gridfs/asynchronous/grid_file.py
Outdated
| iter = self.__aiter__() | ||
| ret = [] | ||
| for _ in range(length): | ||
| ret.append(await iter.next()) |
Member
There was a problem hiding this comment.
Can we call self.next() instead of using __iter__?
| await self.close() | ||
|
|
||
| async def to_list(self) -> list[_DocumentType]: | ||
| async def to_list(self, length: int = -1) -> list[_DocumentType]: |
Member
There was a problem hiding this comment.
Why use -1? I was thinking we'd use None to mean no limit to match Motor. Is it for nicer typing (since there's no need for Optional)?
Member
Author
There was a problem hiding this comment.
I'll switch to Optional[int].
| self.addCleanup(coll.drop) | ||
| c = coll.find() | ||
| docs = c.to_list(3) | ||
| self.assertEqual(len(docs), 3) |
Member
There was a problem hiding this comment.
Could you add another test with a small batchSize? like:
c = coll.find(batch_size=2)
docs = c.to_list(3)
self.assertEqual(len(docs), 3)
docs = c.to_list(3)
self.assertEqual(len(docs), 2)| docs = c.to_list() | ||
| self.assertEqual([], docs) | ||
|
|
||
| def test_to_list_length(self): |
Member
There was a problem hiding this comment.
Could you create an identical test for CommandCursor?
Member
Author
There was a problem hiding this comment.
There is a test_command_cursor_to_list_length
Member
Author
There was a problem hiding this comment.
Updated to use aggregate
ShaneHarvey
requested changes
Aug 13, 2024
| """Get all or some available documents from the cursor.""" | ||
| if not len(self._data) and not self._killed: | ||
| await self._refresh() | ||
| print("hi", len(self._data)) # noqa: T201 |
Member
There was a problem hiding this comment.
Forgot to remove a print here.
ShaneHarvey
approved these changes
Aug 14, 2024
ShaneHarvey
requested changes
Aug 14, 2024
Member
ShaneHarvey
left a comment
ShaneHarvey
left a comment
There was a problem hiding this comment.
Some test issues:
FAILED test/asynchronous/test_cursor.py::TestCursor::test_to_list_length - TypeError: object of type 'coroutine' has no len()
FAILED test/asynchronous/test_session.py::TestSession::test_end_sessions - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <coroutine object AsyncCursor.to_list at 0x7f99a0b51ec0>
Traceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/warnings.py", line 506, in _warn_unawaited_coroutine
warn(msg, category=RuntimeWarning, stacklevel=2, source=coro)
RuntimeWarning: coroutine 'AsyncCursor.to_list' was never awaited
Coroutine created at (most recent call last)
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/unittest/async_case.py", line 158, in run
return super().run(result)
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/unittest/case.py", line 676, in run
self._callTestMethod(testMethod)
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/unittest/async_case.py", line 65, in _callTestMethod
self._callMaybeAsync(method)
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/unittest/async_case.py", line 88, in _callMaybeAsync
return self._asyncioTestLoop.run_until_complete(fut)
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/asyncio/base_events.py", line 603, in run_until_complete
self.run_forever()
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
self._run_once()
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/asyncio/base_events.py", line 1851, in _run_once
handle._run()
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/asyncio/events.py", line 81, in _run
self._context.run(self._callback, *self._args)
File "/opt/hostedtoolcache/Python/3.8.18/x64/lib/python3.8/unittest/async_case.py", line 102, in _asyncioLoopRunner
ret = await awaitable
File "/home/runner/work/mongo-python-driver/mongo-python-driver/test/asynchronous/test_cursor.py", line 1413, in test_to_list_length
docs = c.to_list(3)
= 2 failed, 2347 passed, 2083 skipped, 667 deselected, 26 xfailed, 38 warnings in 189.81s (0:03:09) =
Member
Author
|
Fixed the tests. |
ShaneHarvey
approved these changes
Aug 14, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.