bpo-31861: Provide aiter and anext builtins #23847

jab · 2020-12-18T21:12:17Z

This is the C implementation for bpo-31861 requested as an alternative to the Python implementation I provided in #8895.

For a more direct translation of this into Python (in case it makes reviewing easier), see jab@ce35092.

Patch by @justin39, @lordmauve, and me.

https://bugs.python.org/issue31861

...on top of latest master. Also drop now-removed `loop` kwarg from asyncio.sleep call. Ref: https://bugs.python.org/issue42392

the-knights-who-say-ni · 2020-12-18T21:12:20Z

Hello, and thanks for your contribution!

I'm a bot set up to make sure that the project can legally accept this contribution by verifying everyone involved has signed the PSF contributor agreement (CLA).

Recognized GitHub username

We couldn't find a bugs.python.org (b.p.o) account corresponding to the following GitHub usernames:

@justin39

This might be simply due to a missing "GitHub Name" entry in one's b.p.o account settings. This is necessary for legal reasons before we can look at this contribution. Please follow the steps outlined in the CPython devguide to rectify this issue.

You can check yourself to see if the CLA has been received.

Thanks again for the contribution, we look forward to reviewing it!

Objects/iterobject.c

Doc/library/functions.rst

jab · 2021-02-26T17:26:00Z

Shoot, noticed this PR now has conflicts. Will resolve and push a new revision ASAP.

…n39/aiter-c

justin39 · 2021-02-26T19:52:06Z

Resolved the conflicts - hopefully we can get this reviewed before a new conflict pops up!

* master: (129 commits) bpo-43452: Micro-optimizations to PyType_Lookup (pythonGH-24804) bpo-43517: Fix false positive in detection of circular imports (python#24895) bpo-43494: Make some minor changes to lnotab notes (pythonGH-24861) Mention that code.co_lnotab is deprecated in what's new for 3.10. (python#24902) bpo-43244: Remove symtable.h header file (pythonGH-24910) bpo-43466: Add --with-openssl-rpath configure option (pythonGH-24820) Fix a typo in c-analyzer (pythonGH-24468) bpo-41561: Add workaround for Ubuntu's custom security level (pythonGH-24915) bpo-43521: Allow ast.unparse with empty sets and NaN (pythonGH-24897) bpo-43244: Remove the PyAST_Validate() function (pythonGH-24911) bpo-43541: Fix PyEval_EvalCodeEx() regression (pythonGH-24918) bpo-43244: Fix test_peg_generators on Windows (pythonGH-24913) bpo-39342: Expose X509_V_FLAG_ALLOW_PROXY_CERTS in ssl module (pythonGH-18011) bpo-43244: Fix test_peg_generator for PyAST_Validate() (pythonGH-24912) bpo-42128: Add 'missing :' syntax error message to match statements (pythonGH-24733) bpo-43244: Add pycore_ast.h header file (pythonGH-24908) bpo-43244: Rename pycore_ast.h to pycore_ast_state.h (pythonGH-24907) Remove unnecessary imports in the grammar parser (pythonGH-24904) bpo-35883: Py_DecodeLocale() escapes invalid Unicode characters (pythonGH-24843) Add PEP 626 to what's new in 3.10. (python#24892) ...

jab · 2021-03-20T20:46:39Z

Merged in latest master and fixed some minor nits. All checks have passed against the latest revision.

UPDATE (ICYMI): There is now ongoing discussion in this recent python-dev thread about whether to merge this PR (favored by @1st1, and perhaps others who've ❤️'d and participated in this PR), or whether to instead resurrect #8895, my PR from 2018 that added operator.aiter and operator.anext (favored by @ramalho, @pbryan, @gvanrossum, @brettcannon, @terryjreedy). Looking forward to seeing what the core developers decide, and then hopefully being able to merge one of these two PRs before the upcoming 3.10 feature freeze on May 3. Thank you!

gvanrossum

The basic code is solid. Some questions and suggestions about how much needs to be public and documented.

Doc/glossary.rst

gvanrossum · 2021-03-22T17:51:11Z

Doc/library/functions.rst

+   Return an :term:`asynchronous iterator`. This is the async variant
+   of the :func:`iter` builtin, and behaves similarly.


Frankly, this doesn't tell me much. The description doesn't even state that aiter(x) is equivalent to x.__aiter__(), which to me is the key point. Certainly it shouldn't start by stating the type of what it returns; it should describe how the return value relates to the input. (Compare the entry for abs(x) above, "Return the absolute value of a number." This clearly references the input and what the function does to that value.)

Also, state explicitly that aiter(aiter(x)) is the same as aiter(x) (IOW that aiter(x) itself has an __aiter__() method that returns self.)

Incorporated in the latest revision, but please let me know if it needs further refinement. Thanks for the great feedback!

Doc/library/functions.rst

gvanrossum · 2021-03-22T18:42:54Z

Include/abstract.h

+/* Takes an AsyncIterable object and returns an AsyncIterator for it.
+   This is typically a new iterator but if the argument is an AsyncIterator,
+   this returns itself. */
+PyAPI_FUNC(PyObject *) PyObject_GetAiter(PyObject *);


Is it necessary to add these to the public C API? Just because PyObject_GetIter() is public I'm not sure that the Aiter variant needs to be. @vstinner tends to push back on adding new things to the C API. @1st1 what do you think?

I don't mind adding this function -- while somewhat trivial, it's something that projects like Cython (and potentially our own modules like _asynciomodule.c) have to reimplement.

gvanrossum · 2021-03-22T18:44:05Z

Include/iterobject.h

@@ -7,6 +7,8 @@ extern "C" {

 PyAPI_DATA(PyTypeObject) PySeqIter_Type;
 PyAPI_DATA(PyTypeObject) PyCallIter_Type;
+PyAPI_DATA(PyTypeObject) PyAsyncCallAwaitable_Type;


Maybe these types and the function below should remain CPython implementation details? Just because they're returned by builtins doesn't mean all the implementation types need to be in the C-level API.

+1. I'd make them internal.

Makes sense, thanks. Done in the latest revision. Another look?

Objects/abstract.c

gvanrossum

Almost there!

gvanrossum · 2021-03-22T21:21:05Z

Doc/library/functions.rst

+   Equivalent to calling ``x.__aiter__()``.

-   *async_iterable* must be an :term:`asynchronous iterable`,
-   and :func:`aiter` returns an asynchronous iterator for it.
+   ``aiter(aiter(x))`` is the same as ``aiter(x)``.
+   (``aiter(x)`` itself has an ``__aiter__()`` method that returns ``self``.)

-   Unlike the :func:`iter` builtin, :func:`aiter` has no 2-argument variant.
-   Often, this variant can be replaced with assignment expressions::
+   Formally, given an :term:`asynchronous iterable`,
+   return an :term:`asynchronous iterator`.


Much better! Here is how I would rearrange this:

Return an :term:`asynchronous iterator` for an :term:`asynchronous iterable`. Equivalent to calling ``x.__aiter__()``. ``aiter(x)`` itself has an ``__aiter__()`` method that returns ``x``, so ``aiter(aiter(x))`` is the same as ``aiter(x)``.

Incorporated in the latest revision, thanks!

(((
Come to think of it, iter also has the iter(iter(x)) == iter(x) property, which is not currently mentioned in the iter docs. The iter docs have a lot more work to do though, to cover the 1- and 2-arg variants (and they're already doing this very well).

And now that I'm looking at those again, I notice the only code example there is for the 2-arg variant:

from functools import partial with open('mydata.db', 'rb') as f: for block in iter(partial(f.read, 64), b''): process_block(block)

If this pattern is now obsoleted by assignment expressions[1], is it worth updating the iter docs to (1) mention the iter(iter(x)) == iter(x) property, and (2) remove the obsoleted example code? If so, happy to submit a separate PR for that.

[1] as in the following:

with open('mydata.db', 'rb') as f: while block := f.read(64): process_block(block)

)))

I don't want to personally get into the weeds about the iter() docs, sorry. Something for the docs WG perhaps.

I don't think it's worth updating the docs to promise iter(iter(x)) == iter(x) as it isn't a specific benefit to the user beyond logically doing the right thing.

I can see that.

@gvanrossum, do you think @brettcannon's rationale applies equally to the aiter docs too? I did add a test_aiter_idempotent() for this in 6ee8824 to go along with the promise that the aiter docs are now making, but can remove that test along with that part of the docs if that's better.

I think there's a different bar for updating the iter() docs than for the initial version of the aiter() docs. And they don't have to match precisely.

FWIW the main reason the idempotency property is important is because of the implicit [a]iter() call in a for-loop, since in

for i in x: ...

the for-loop calls iter(x), so that in

for i in iter(x): ...

the for-loop ends up calling iter(iter(x)). It's the same for async for and aiter().

gvanrossum

Thanks for your patience. This all looks good to me now! I'll merge after applying my own suggestion.

Doc/library/functions.rst

bedevere-bot · 2021-03-23T22:47:24Z

@gvanrossum: Please replace # with GH- in the commit message next time. Thanks!

gvanrossum · 2021-03-23T22:47:33Z

Congrats!

jab · 2021-03-23T22:53:03Z

Thanks so much, @gvanrossum! We’re honored to have been able to contribute!

bedevere-bot · 2021-03-23T22:54:58Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x RHEL8 3.x has failed when building commit f0a6fde.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/509/builds/900) and take a look at the build logs.
Check if the failure is related to this commit (f0a6fde) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/509/builds/900

Summary of the results of the build (if available):

== Tests result: ENV CHANGED ==

412 tests OK.

10 slowest tests:

test_concurrent_futures: 3 min 9 sec
test_multiprocessing_spawn: 1 min 36 sec
test_peg_generator: 1 min 26 sec
test_capi: 1 min 17 sec
test_gdb: 1 min 14 sec
test_unparse: 1 min 12 sec
test_multiprocessing_forkserver: 1 min 10 sec
test_asyncio: 1 min 5 sec
test_tokenize: 1 min 3 sec
test_multiprocessing_fork: 1 min

1 test altered the execution environment:
test_asyncio

14 tests skipped:
test_devpoll test_ioctl test_kqueue test_msilib test_nis
test_ossaudiodev test_startfile test_tix test_tk test_ttk_guionly
test_winconsoleio test_winreg test_winsound test_zipfile64

Total duration: 5 min 18 sec

Click to see traceback logs

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py", line 321, in __del__
    self.close()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py", line 316, in close
    self._ssl_protocol._start_shutdown()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py", line 590, in _start_shutdown
    self._abort()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/sslproto.py", line 731, in _abort
    self._transport.abort()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/selector_events.py", line 680, in abort
    self._force_close(None)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/selector_events.py", line 731, in _force_close
    self._loop.call_soon(self._call_connection_lost, exc)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/base_events.py", line 745, in call_soon
    self._check_closed()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-rhel8-z/build/Lib/asyncio/base_events.py", line 510, in _check_closed
    raise RuntimeError('Event loop is closed')
RuntimeError: Event loop is closed

gvanrossum · 2021-03-23T22:57:28Z

I believe test_asyncio is unstable or broken on this platform.

vstinner · 2021-04-07T22:59:17Z

I proposed to PR #25266 to rename PyAnextAwaitable_Type to _PyAnextAwaitable_Type, and to initialize the type at Python startup: can someone please have a look?

jab and others added 20 commits November 30, 2020 09:41

bpo-31861: Add operator.aiter and operator.anext

44b2a60

address comments from first review

142523b

Address new review comments + rebase

3ebce05

...on top of latest master. Also drop now-removed `loop` kwarg from asyncio.sleep call. Ref: https://bugs.python.org/issue42392

improve tests

2f8df99

Sketch out async iterator in C

ad12116

Implement aiter() and anext() using sync methods only

ce35092

Add basic aiter built-in [WIP]

f9dc183

Start aiter implementation

086bb79

Get test_asyncgen tests passing

29ef712

Fix awaitable iternext

331e80e

Add anext builtin

95879d8

Use stop iter functions for anext

5b64589

Use stop iter functions in aiter

6e145db

Note about implementing __reduce__ for aiter

2fd4be6

Refactor aiter and anext type names

430dd59

Add documentation for aiter()

4266e65

Update documentation for aiter and anext

ecbedc7

Clean up docs and formatting

5fa3812

Remove async iterator code from operator.py

0f9c814

Cleanup formatting

8160c82

the-knights-who-say-ni added the CLA not signed label Dec 18, 2020

bedevere-bot added the awaiting review label Dec 18, 2020

jab mentioned this pull request Dec 18, 2020

bpo-31861: Add operator.aiter and operator.anext #8895

Closed

eric-wieser reviewed Dec 18, 2020

View reviewed changes

Objects/iterobject.c Outdated Show resolved Hide resolved

jab added 3 commits December 18, 2020 17:04

Fix test_builtins_have_signatures + misc. cleanups

fa8b12a

Use PyErr_SetNone now that we can

a2242d9

whitespace

3ce5675

jab commented Dec 19, 2020

View reviewed changes

Doc/library/functions.rst Outdated Show resolved Hide resolved

cosmetic fixes

0038cde

Merge branch 'master' of https://github.com/python/cpython into justi…

e687d88

…n39/aiter-c

jab added 3 commits March 20, 2021 16:00

Delint tests + docs (fixes CI), alphabetize names.

259af97

Fix code style and comments.

dd7c02d

jab added 2 commits March 22, 2021 09:58

Remove 2-arg variant of aiter.

2fbdd5b

Fix use of clinic.

009118c

gvanrossum reviewed Mar 22, 2021

View reviewed changes

jab added 2 commits March 22, 2021 16:23

Address some feedback from Guido's review.

71fede3

No longer need to exclude aiter from test_builtins_have_signatures.

4a51ace

gvanrossum reviewed Mar 22, 2021

View reviewed changes

jab added 4 commits March 22, 2021 17:32

Improve aiter() docs.

108e4a3

Fix typo.

042f596

Add test_aiter_idempotent().

6ee8824

Remove public API added for anext.

6f50ef8

gvanrossum approved these changes Mar 23, 2021

View reviewed changes

Doc/library/functions.rst Outdated Show resolved Hide resolved

bedevere-bot added awaiting merge and removed awaiting core review labels Mar 23, 2021

Slightly improve wording

ef40fb7

gvanrossum merged commit f0a6fde into python:master Mar 23, 2021

bedevere-bot removed the awaiting merge label Mar 23, 2021

jab changed the title ~~bpo-31861: Add aiter and anext to builtins~~ bpo-31861: Provide aiter and anext builtins May 22, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bpo-31861: Provide aiter and anext builtins #23847

bpo-31861: Provide aiter and anext builtins #23847

jab commented Dec 18, 2020 •

edited

Loading

the-knights-who-say-ni commented Dec 18, 2020

jab commented Feb 26, 2021

justin39 commented Feb 26, 2021

jab commented Mar 20, 2021 •

edited

Loading

gvanrossum left a comment

gvanrossum Mar 22, 2021

jab Mar 22, 2021

gvanrossum Mar 22, 2021

1st1 Mar 23, 2021

gvanrossum Mar 22, 2021

1st1 Mar 23, 2021

jab Mar 23, 2021

gvanrossum left a comment

gvanrossum Mar 22, 2021

jab Mar 22, 2021

gvanrossum Mar 22, 2021

brettcannon Mar 23, 2021

jab Mar 23, 2021 •

edited

Loading

gvanrossum Mar 23, 2021

gvanrossum left a comment

bedevere-bot commented Mar 23, 2021

gvanrossum commented Mar 23, 2021

jab commented Mar 23, 2021

bedevere-bot commented Mar 23, 2021

gvanrossum commented Mar 23, 2021 via email

vstinner commented Apr 7, 2021

		Return an :term:`asynchronous iterator`. This is the async variant
		of the :func:`iter` builtin, and behaves similarly.

bpo-31861: Provide aiter and anext builtins #23847

bpo-31861: Provide aiter and anext builtins #23847

Conversation

jab commented Dec 18, 2020 • edited Loading

the-knights-who-say-ni commented Dec 18, 2020

Recognized GitHub username

jab commented Feb 26, 2021

justin39 commented Feb 26, 2021

jab commented Mar 20, 2021 • edited Loading

gvanrossum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvanrossum left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jab Mar 23, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gvanrossum left a comment

Choose a reason for hiding this comment

bedevere-bot commented Mar 23, 2021

gvanrossum commented Mar 23, 2021

jab commented Mar 23, 2021

bedevere-bot commented Mar 23, 2021

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

gvanrossum commented Mar 23, 2021 via email

vstinner commented Apr 7, 2021

jab commented Dec 18, 2020 •

edited

Loading

jab commented Mar 20, 2021 •

edited

Loading

jab Mar 23, 2021 •

edited

Loading