From aaf2b0def841c625fd2309b0ef83521b480c7744 Mon Sep 17 00:00:00 2001 From: Peter Bierma Date: Sat, 23 May 2026 22:30:06 -0400 Subject: [PATCH] Changes to PEP 828. --- peps/pep-0828.rst | 401 +++++++++++++++++++++------------------------- 1 file changed, 179 insertions(+), 222 deletions(-) diff --git a/peps/pep-0828.rst b/peps/pep-0828.rst index 95ec463d0ee..95ecaad857f 100644 --- a/peps/pep-0828.rst +++ b/peps/pep-0828.rst @@ -1,6 +1,7 @@ PEP: 828 Title: Supporting 'yield from' in asynchronous generators Author: Peter Bierma +PEP-Delegate: Yury Selivanov Discussions-To: https://discuss.python.org/t/106459 Status: Draft Type: Standards Track @@ -14,47 +15,19 @@ Abstract ======== This PEP introduces support for :keyword:`yield from ` in an -:ref:`asynchronous generator function `. - -For example, the following code is valid under this PEP: - -.. code-block:: python - - def generator(): - yield 1 - yield 2 - - async def main(): - yield from generator() - - -In addition, this PEP introduces a new ``async yield from`` construct to -delegate to an asynchronous generator: +:ref:`asynchronous generator function ` +through a new ``async yield from`` construct: .. code-block:: python async def agenerator(): yield 1 yield 2 - - async def main(): - async yield from agenerator() - - -In order to allow use of ``async yield from`` as an expression, this PEP -removes the existing limitation that asynchronous generators may not return -a non-``None`` value. For example, the following code is valid under this -proposal: - -.. code-block:: python - - async def agenerator(): - yield 1 - return 2 + return 3 async def main(): result = async yield from agenerator() - assert result == 2 + assert result == 3 Terminology @@ -62,9 +35,8 @@ Terminology This PEP refers to an ``async def`` function that contains a ``yield`` as an :term:`asynchronous generator`, sometimes suffixed with "function". - -In contrast, the object returned by an asynchronous generator is referred to -as an :term:`asynchronous generator iterator` in this PEP. +This is not to be confused with an :term:`asynchronous generator iterator`, +which is the object *returned* by an asynchronous generator. This PEP also uses the term "subgenerator" to refer to a generator, synchronous or asynchronous, that is used inside of a ``yield from`` or ``async yield from`` @@ -141,8 +113,7 @@ Compiler changes ^^^^^^^^^^^^^^^^ The compiler will no longer emit a :exc:`SyntaxError` for -:keyword:`yield from ` and :keyword:`return` statements inside -asynchronous generators. +:keyword:`return` statements inside asynchronous generators. Grammar changes @@ -160,117 +131,109 @@ The ``yield_expr`` and ``simple_stmt`` rules need to be updated for the new | &('yield' | 'async') yield_stmt -``yield from`` behavior in asynchronous generators --------------------------------------------------- +Changes to ``StopAsyncIteration`` +--------------------------------- -This PEP retains all existing ``yield from`` semantics; the only detail is -that asynchronous generators may now use it. +The :class:`StopAsyncIteration` exception will gain a new ``value`` attribute +to be used as the result of ``async yield from`` expressions. -Because the existing ``yield from`` behavior may only yield from a synchronous -subgenerator, this is true for asynchronous generators as well. - -For example: +This attribute can be supplied by passing a positional argument to +``StopAsyncIteration``. For example: -.. code-block:: python +.. code-block:: pycon - def generator(): - yield 1 - yield 2 - yield 3 + >>> exception = StopAsyncIteration(42) + >>> exception.value + 42 - async def main(): - yield from generator() - yield 4 -In the above code, ``main`` will yield ``1``, ``2``, ``3``, ``4``. -All subgenerator delegation semantics are retained. +If no argument is supplied, ``value`` will be ``None``. -``async yield from`` as a statement ------------------------------------ +``return`` statements inside asynchronous generators +---------------------------------------------------- -``async yield from`` is equivalent to ``yield from``, with the exception that: +In an asynchronous generator, the statement ``return expression`` is +roughly equivalent to ``raise StopAsyncIteration(expression)``. However, +the exception cannot be caught in the body of the asynchronous generator. -1. :meth:`~object.__aiter__` is called to retrieve the asynchronous - generator iterator. -2. :meth:`~agen.asend` is called to advance the asynchronous generator - iterator. -``async yield from`` is only allowed in an asynchronous generator function; -using it elsewhere will raise a :exc:`SyntaxError`. +``async yield from`` semantics +------------------------------ -In an asynchronous generator, ``async yield from`` is conceptually equivalent to: +The statement .. code-block:: python - async for item in agenerator(): - yield item - -``async yield from`` retains all the subgenerator delegation behavior present -in standard ``yield from`` expressions. This behavior is outlined in :pep:`380` -and :ref:`the documentation `. In short, values passed with -:meth:`~agen.asend` and exceptions supplied with :meth:`~agen.athrow` -are also passed to the target generator. - + RESULT = async yield from EXPR -``async yield from`` as an expression -------------------------------------- - -``async yield from`` may also be used as an expression. For reference, -the result of a ``yield from`` expression is the object returned by the -synchronous generator. ``async yield from`` does the same; the expression -value is the value returned by the executed asynchronous generator. - -However, Python currently prevents asynchronous generators from returning -any non-``None`` value. This limitation is removed by this PEP. - -When an asynchronous generator iterator is exhausted, it will raise a -:exc:`StopAsyncIteration` exception with a ``value`` attribute, similar -to the existing :exc:`StopIteration` behavior with synchronous generators. -To visualize: +is roughly equivalent to the following: .. code-block:: python - async def agenerator(): - yield 1 - return 2 - - async def main(): - gen = agenerator() - print(await gen.asend(None)) # 1 - try: - await gen.asend(None) - except StopAsyncIteration as result: - print(result.value) # 2 + aiterator = aiter(EXPR) + try: + item = await anext(aiterator) + except StopAsyncIteration as stop: + RESULT = stop.value + else: + while True: + try: + received = yield item + except GeneratorExit as gen_exit: + try: + aclose = aiterator.aclose + except AttributeError: + pass + else: + await aclose() + raise gen_exit + except BaseException as exception: + try: + athrow = aiterator.athrow + except AttributeError: + raise exception from None + else: + try: + item = await athrow(exception) + except StopAsyncIteration as stop: + RESULT = stop.value + break + else: + try: + if received is None: + item = await anext(aiterator) + else: + item = await aiterator.asend(received) + except StopAsyncIteration as stop: + RESULT = stop.value + break -The contents of the ``value`` attribute will be the result of the ``async -yield from`` expression. +Rationale +========= -For example: -.. code-block:: python +Relation to ``yield from`` +-------------------------- - async def agenerator(): - yield 1 - return 2 - - async def main(): - result = async yield from agenerator() - print(result) # 2 +This PEP aims to be very similar to the semantics of ``yield from``, with the +exception that asynchronous generator methods are used instead of synchronous +generator methods when delegating. This is a very intuitive design and furthers +symmetry with synchronous generators. -Rationale -========= +Choice of ``async yield from`` as the syntax +-------------------------------------------- -The distinction between ``yield from`` and ``async yield from`` in this proposal -is consistent with existing asynchronous syntax constructs in Python. -For example, there are two constructs for context managers: ``with`` and -``async with``. +This PEP uses ``async yield from`` as the syntax to ensure that the behavior +of the syntax is immediately clear to the user. -This PEP follows this pattern; ``yield from`` continues to be synchronous, even -in asynchronous generators, and ``async yield from`` is the asynchronous -variation. +However, it is acknowledged that this is somewhat verbose. There is not any +great solution to this problem; see :ref:`pep-828-rejected-ideas` for +discussion about proposed alternatives. In short, ``async yield from`` was +chosen as the best choice of syntax because, while verbose, it is very clear +and readable. Backwards Compatibility @@ -290,54 +253,34 @@ How to Teach This The details of this proposal will be located in Python's canonical documentation, as with all other language constructs. However, this PEP -intends to be very intuitive; users should be able to deduce the behavior of -``yield from`` in an asynchronous generator based on their own background -knowledge of ``yield from`` in synchronous generators. - - -Potential footguns ------------------- - - -Forgetting to ``await`` a future -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +intends to be very intuitive; users should be able to naturally reach +for ``async yield from`` given their own background knowledge about generators +in Python. This can be encouraged further by suggesting +``async yield from`` in the error message when a user attempts to use +``yield from`` in a synchronous generator. -In :mod:`asyncio`, a :ref:`future ` object is natively -iterable. This means that if one were trying to iterate over the result of a -future, forgetting to :keyword:`await` the future may accidentally await the -future itself, leading to a spurious error. -For example: +Reference Implementation +======================== -.. code-block:: python +A reference implementation of this PEP can be found at +`python/cpython#145716 `__. - import asyncio - async def steps(): - await asyncio.sleep(0.25) - await asyncio.sleep(0.25) - await asyncio.sleep(0.25) - return [1, 2, 3] +.. _pep-828-rejected-ideas: - async def agenerator(): - # Forgot to await! - yield from asyncio.ensure_future(steps()) +Rejected Ideas +============== - async def run(): - total = 0 - async for i in agenerator(): - # TypeError?! - total += i - print(total) +Using ``yield from`` to delegate to asynchronous generators +----------------------------------------------------------- -Attempting to use ``yield from`` on an asynchronous subgenerator -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Due to the verbosity of ``async yield from``, it was proposed to overload +the existing ``yield from`` syntax to perform asynchronous subgenerator +delegation when used inside of an asynchronous generator. -A common intuition among developers is that ``yield from`` inside an -asynchronous generator will also delegate to another asynchronous generator. -As such, many users were surprised to see that, in this proposal, the following -code is invalid: +For example: .. code-block:: python @@ -349,96 +292,104 @@ code is invalid: yield from asubgenerator() -As a solution, when ``yield from`` is given an object that is not iterable, -the implementation can detect if that object is asynchronously iterable. -If it is, ``async yield from`` can be suggested in the exception message. - -This is done in the reference implementation of this proposal; the example -above raises a :exc:`TypeError` that reads ``async_generator object is not -iterable. Did you mean 'async yield from'?`` - +This has the benefit of being more concise than ``async yield from``, but also +has a few downsides. -Reference Implementation -======================== +Most importantly, this makes the asynchronous context switches necessary for +delegation implicit, which has no precedent in Python; all syntax that may +execute an ``await`` is prefixed with ``async``. It has been argued that one +of the upsides of ``async``/``await`` over threads is the explicit switch +points, so hiding ``await``\ s behind a ``yield from`` hurts this benefit. -A reference implementation of this PEP can be found at -`python/cpython#145716 `__. +Second, many were uncomfortable with ``yield from`` being context-dependent. +It felt like a potential footgun for ``yield from`` to mean something +different based on the type of generator it was used in. In practice, this +may come up in a scenario where one wants to convert a synchronous generator +into an asynchronous generator. +For example, imagine a developer is writing a function for streaming data +to the caller: -Rejected Ideas -============== +.. code-block:: python + def stream_data(): + yield ... + yield from something_else() + yield ... -Using ``yield from`` to delegate to asynchronous generators ------------------------------------------------------------ +Now, imagine that the developer wants to add an ``await`` call somewhere in +this function; the ``yield from something_else()`` statement would suddenly +become invalid. With the current proposal, the existence of ``async yield from`` +(which would ideally be included in the error message) would make it much +clearer that ``something_else`` must also be asynchronous in order to use it +in the function. -It has been argued that many developers may intuitively believe that using a -plain ``yield from`` inside an asynchronous generator would also delegate to -an asynchronous subgenerator rather than a synchronous subgenerator. As such, -it was proposed to make ``yield from`` always delegate to an asynchronous -subgenerator. +Finally, this would preclude the introduction of support for synchronous +subgenerator delegation inside asynchronous generators (see +:ref:`pep-828-synchronous-delegation`), because the ``yield from`` +syntax would already be overloaded. However, the author of this proposal +does acknowledge that the issues with synchronous subdelegation may preclude +the introduction of this anyway -- it is not entirely clear whether the issues +are solvable given time. -For example: -.. code-block:: python +``async from``, ``await from``, and similar spellings +----------------------------------------------------- - async def asubgenerator(): - yield 1 - yield 2 +As an alternate solution to the verbosity of ``async yield from``, some have +suggested using spellings such as ``async from`` in order to cut down the +verbosity. Unfortunately, changes in the spelling will likely hurt the +readability of the syntax as a whole. - async def agenerator(): - yield from asubgenerator() +The benefit of ``async yield from`` is that it specifies each of the three +important parts without introducing new keywords. In particular: +1. ``async`` is necessary to imply an asynchronous context switch. +2. ``yield`` is necessary to indicate that the generator will be suspended. +3. ``from`` is necessary to differentiate between "standard" generator + suspension (a ``yield`` statement) and subgenerator delegation. -This was rejected, primarily because it felt very wrong for ``yield from x`` to -be valid or invalid depending on the type of generator it was used in. +Given these three constraints, it seems unlikely that a more concise spelling +exists. -In addition, there is no precedent for this kind of behavior in Python; inherently -synchronous constructs always have an asynchronous counterpart for use in -asynchronous functions, instead of implicitly switching protocols depending on -the type of function it is used in. For example, :keyword:`with` always means that the -:term:`synchronous context management protocol ` will -be invoked, even when used in an ``async def`` function. -Finally, this would leave a gap in asynchronous generators, because there would be -no mechanism for delegating to a synchronous subgenerator. Even if this is not a -common pattern today, this may become common in the future, in which case it would -be very difficult to change the meaning of ``yield from`` in an asynchronous -generator. +.. _pep-828-synchronous-delegation: +Allowing delegation to synchronous subgenerators +------------------------------------------------ -Letting ``yield from`` determine which protocol to use ------------------------------------------------------- +In an earlier revision of this proposal, the synchronous ``yield from`` +construct was allowed in an asynchronous generator, which would delegate to a +synchronous generator from an asynchronous one. This had a number of hidden +issues. -As a solution to the above rejected idea, it was proposed to allow ``yield from x`` -to invoke the synchronous or asynchronous generator protocol depending on the type -of ``x``. In turn, this would allow developers to delegate to both synchronous -and asynchronous subgenerators while continuing to use the familiar ``yield from`` -syntax. +In particular, the mixing of asynchronous frames with synchronous frames had a +layer of complexity unfit for Python. In the implementation, there would have +to be a hidden translation layer between synchronous generator methods and +asynchronous generator methods: :meth:`~agen.asend` to :meth:`~generator.send`, +:meth:`~agen.athrow` to :meth:`~generator.throw`, and :meth:`~agen.aclose` +to :meth:`~generator.close`. -For example: +For example, asynchronous exceptions could be injected into synchronous +generators: .. code-block:: python - async def asubgenerator(): - yield 1 - yield 2 + async def agen(): + async with asyncio.timeout(3): + # If the timeout fails, then an asyncio.TimeoutError would be raised + # in a *synchronous* generator! + yield from subgen() - async def agenerator(): - yield from asubgenerator() - yield from range(3, 5) +To quote Brandt Bucher (paraphrased): -Mechanically, this is possible, but the exact behavior will likely be counterintuitive -and ambigious. In particular: + At that point, why not just allow synchronous functions to await coroutines? -1. If an object implements both :meth:`~object.__iter__` and :meth:`~object.__aiter__`, - it's not clear which protocol Python should choose. -2. If the chosen protocol raises an exception, should the exception be propagated, or - should Python try to use the other protocol first? -Additionally, this approach is inherently slower, because of the additional overhead -of detecting which generator protocol to use. +In addition, there seemed to be much less demand for this feature compared to +support for asynchronous delegation, so solving these issues is less of a +priority for now. Acknowledgements @@ -450,11 +401,17 @@ changes alongside the support for non-``None`` return values inside asynchronous generators were largely based on Alex Dixon's design from `python/cpython#125401 `__. +Special thanks to Yury Selivanov for providing extensive feedback and also +collecting outside opinions about the design and implementation. + Change History ============== -TBD. +- 22-May-2026 + + - Removed support for delegating to a synchronous subgenerator (via + a plain ``yield from``). Copyright