asyncio Notes

Source annotated-py-asyncio

Reading Supplement:#

1. Basic Concepts:#

1.1 Coroutines:#

"A coroutine is a computer program component for non-preemptive multitasking, coroutines allow different entry points to pause or start executing the program at different positions".
Technically, "a coroutine is a function that can be paused and resumed".
If you think of it as "just like a generator", then you're right.

1.2 Event Loop:#

An event loop "is a programming architecture for waiting for a program to allocate events or messages".
Basically, an event loop is "when A happens, do B".
Perhaps the simplest example to explain this concept is the JavaScript event loop that exists in every browser.
When you click on something ("when A happens"), this click action is sent to the JavaScript event loop, which checks if there is a registered onclick callback to handle this click ("do B").
As long as there are registered callback functions, they will be executed along with the details of the click action.
The event loop is considered a loop because it continuously collects events and responds to them through a loop.

1.3 Python Event Loop:#

For Python, the asyncio library, which provides an event loop, has been added to the standard library.
asyncio focuses on solving problems in network services, where the event loop treats I/O from sockets that are ready to read and/or write as "when A happens" (via the selectors module).
In addition to GUI and I/O, event loops are also often used to execute code in other threads or subprocesses, and the event loop serves as a coordination mechanism (e.g., cooperative multitasking).
If you happen to understand Python's GIL, the event loop is useful in places where you need to release the GIL.
The event loop provides a looping mechanism that allows you to "when A happens, do B".
Basically, an event loop listens for what happens, and the event loop also cares about it and executes the corresponding code.
Python 3.4 and later have the ability to obtain event loop features through the standard library asyncio.

1.4 async, await:#

View async/await as the API for asynchronous programming.
Basically, async and await generate magical generators, which we call coroutines.
Additional support is needed, such as awaitable objects and converting ordinary generators into coroutines.
All of these together support concurrency, making Python better suited for asynchronous programming.
Compared to similar features in threads, this is a more wonderful and simpler method.

In Python 3.4, functions for asynchronous programming and marked as coroutines look like this:


    # This also works in Python 3.5.
    @asyncio.coroutine
    def py34_coro():
        yield from stuff()

Python 3.5 added the types.coroutine decorator, which can also mark generators as coroutines, just like asyncio.coroutine.
You can define a coroutine function using async def, although this function cannot contain any form of yield statement; only return and await can return values from the coroutine.

    async def py35_coro():
        await stuff()

You will find that not only async, but Python 3.5 also introduced the await expression (which can only be used in async def).
Although the use of await is similar to yield from, the objects that await can accept are different.
Of course, await can accept coroutines, as the concept of coroutines is the basis of all this.
However, when you use await, the object it accepts must be an awaitable object: it must define the await() method and this method must return an iterator that is not a coroutine.
Coroutines themselves are also considered awaitable objects (which is why collections.abc.Coroutine inherits from collections.abc.Awaitable).
This definition follows Python's tradition of converting most syntax structures into method calls at the lower level, just like a + b is actually a.add(b) or b.radd(a).

Why do async-based coroutines and generator-based coroutines differ in the corresponding suspension expressions?
The main reason is for the sake of optimizing Python performance, ensuring that you do not confuse different objects with the same API.
Since generators implement the coroutine API by default, it is very likely that you will mistakenly use a generator when you want to use a coroutine.
And because not all generators can be used in coroutine-based control flow, you need to avoid using generators incorrectly.

Coroutines can be defined using async def.
Another way to define coroutines is through the types.coroutine decorator
-- Technically, it adds the CO_ITERABLE_COROUTINE flag
-- Or it is a subclass of collections.abc.Coroutine.
You can only achieve coroutine suspension through generator-based definitions.

An awaitable object is either a coroutine or an object that defines the await() method
-- It is also collections.abc.Awaitable
-- And await() must return an iterator that is not a coroutine.

The await expression is basically the same as yield from, but it can only accept awaitable objects (ordinary iterators cannot).
An async-defined function contains a return statement
-- Including the default return None for all Python functions
-- And/or await expressions (yield expressions are not allowed).
The restrictions on async functions ensure that you do not mix generator-based coroutines with ordinary generators, as the expectations for these two types of generators are very different.

1.5 Comparison of Python Coroutines and Golang:#

From Python to Go and Back Again
- PPT
- Discussion on this PPT: Go is not much faster than PyPy, but the complexity and difficulty of debugging increase significantly
- The end promotes Rust.
Reference to asynchronous libraries:
- hyper
- curio
  - Views asyncio as a framework for asynchronous programming using the async/await API
  - David views async/await as an API for asynchronous programming and created the curio project to implement his own event loop.
  - Allows projects like curio to have different ways of operating at a lower level
  - (e.g., asyncio uses future objects as the API for communication with the event loop, while curio uses tuples)

2. Source Code Module:#

2.1 futures.py #

2.1.1 References:#

How the heck does async/await work in Python 3.5?
- Translated: How the heck does async/await work in Python 3.5?
- Translated: GitHub
- Translated: Blog
Python Coroutine: From yield/send to async/await
- Analysis of the future source code
concurrent.futures Source Code Reading Notes (Python)
- concurrent.futures is an asynchronous library
- concurrent.futures — Asynchronous computation

2.1.2 Generator VS Iterator:#

Improve Your Python: Yield and Generators Explained
- Translated: 提高你的 Python: 解释‘yield’和‘Generators（生成器）’
- The simplest generator outside of Python should be something called coroutines.
- A generator is used to generate a series of values.
- yield is like the return result of a generator function.
- Another thing that yield does is save the state of a generator function.
- A generator is a special type of iterator.
- Like an iterator, we can use next() to get the next value from the generator.
- By implicitly calling next(), we can ignore some values.
Relationship between Generators and Iterators
- A generator is a special type of iterator, and its implementation is simpler and more elegant. yield is the key to implementing the next() method of a generator.
Python Magic - Iterators
Generators