-
Notifications
You must be signed in to change notification settings - Fork 6
3.0 History
Eric Snow edited this page May 31, 2023
·
1 revision
The history of this project is presented here as a combination of segmented timelines and resources related to those segments.
This is the full timeline, segmented at specific key major events.
(expand)
(expand)
| date | who | core Python | Python community | other |
|---|---|---|---|---|
| 1960s(?) | Djikstra(?), others | threads introduced | ||
| 1960s | Hoare | CSP | ||
| 1991 (Feb) | Guido | Python first released | ||
| ^^^(?) | Guido | GIL added | ||
| 1997 (1.5) | Guido | subinterpreter support added | ||
| ??? | -- | multi-core becomes mainstream | ||
| 2003 (2.3) | Hammond | GILState API (PEP 311) | ||
| 2006 | Eric | (Eric's first software job (Python)) | ||
| 2007 | Dumpleton | ModWSGI starts using subinterpreters | ||
| 2007 (3.0) | MvL | PEP 3121 | ||
| ~2008 | Eric | (Eric starts getting involved) | ||
| 2009 (3.2) | MvL | PEP 384 | ||
| 2009 | Pitrou | "new" GIL | ||
| 2012 | Coghlan | PEP 432 | ||
| 2012 | Eric | (Eric becomes core dev) | ||
| 2013 (3.5) | Viktorin | PEP 489 | ||
| 2013 (3.5) | Guido, et al | async added to Python | ||
| 2013 | Trent Nelson's PyParallel | |||
(expand)
| date | who | core Python (multi-core-related) |
Python community (multi-core-related) |
other |
|---|---|---|---|---|
| 2014 (Oct) | Eric | motivating event | ||
| 2014-15 | Eric | research | ||
| 2015 (late) | Eric | decide to take multiple interpreters approach | ||
| 2014-16 | Eric | (PEP 468) | ||
| 2014-16 | Eric | (PEP 520) | ||
| 2014-16 | Eric | (C OrderedDict) | ||
| 2015 | Eric | many email threads about subinterpreters | ||
| 2015 | Eric | start prototyping ext. module | ||
| 2016 | Eric | burnt out | ||
I didn't start digging into the details of interpreter isolation until 2017, but by this point I did develop a good sense of what would be involved.
(expand)
Conclusion:
- per-interpreter GIL seems feasible
- thousands of small tasks, all within my skill level
- 95% of them worth doing anyway (for code health, pre-fork, embedding, ext. fini, etc.)
Constraints:
- little/no performance impact on single-interpreter usage
- no change in behavior for single-interpreter usage
- no C-API breakage (esp. limited API)
- must be better than the existing alternatives (e.g. multiprocessing)
- ...
Challenges:
- C global variables
- memory allocators
- objects exposed in C-API (builtin types/exceptions, singletons)
- objects & other mutable data shared by interpreters
- legacy extension modules
- managing shared resources (e.g. memory, file descriptors, dyn-loaded obj files)
- safe "communication" between interpreters
- overhead (startup, memory) of multiple interpreters
- C-API not designed around multiple interpreters
- most core devs not keeping multiple interpreters in mind
- not a lot of testing of subinterpreters
- long-standing, unresolved bugs
- daemon threads
- gilstate API
- ...
Plan (multiple phases):
- isolation & Python API a. expose subinterpreters via stdlib module b. improve interpreter isolation (enough) c. stop sharing the GIL between interpreters
- build on the foundation a. expand Python API (efficiency, sharing objects, etc.) b. optimization (interp startup) c. more isolation
Low-level Plan (Phase 1):
- resolve any remaining blockers
- add "interpreters" module (i.e. PEP 554)
- minimal multi-core solution a. stop sharing some global state (e.g. probably allocators) b. stop sharing the GIL c. no extension modules d. no fork or threading? (AKA subinterpreter "serial execution mode")
Secondary Work To Do (Phase 1):
- subinterpreter bugs
- improved interpreter isolation
- handling of process state (env vars, etc.)
- (mostly) eliminate process globals in core
(expand)
| date | who | core Python (multi-core-related) |
Python community (multi-core-related) |
other |
|---|---|---|---|---|
| 2016 | Viktorin, et al | PEP 573 | ||
| 2016 | Hastings | gilectomy | ||
| 2016 | Eric, et al | encouraging discussions at inaugural core sprint | ||
| 2017 | Eric | POC ext. module | ||
| 2017 | Eric | created PEP 554 | ||
| 2017 | Eric, core team | lots of discussions about multiple interpreters | ||
| 2017 | Eric | analyzing runtime state (notes) | ||
| 2017 | Eric | analyzing runtime init | PEP 432 (notes) | notes | ||
| 2017 | Eric | landed the PEP 432 impl | ||
| 2017 | Eric | consolidated runtime state (added _PyRuntimeState, and Includes/internal/) |
||
| 2017 | Eric, et al | good discussions at core sprint | ||
(expand)
(Dec. 2017)
| date | who | core Python (multi-core-related) |
Python community (multi-core-related) |
other |
|---|---|---|---|---|
| 2017 | Eric | (Eric has new job at Microsoft; get 20% time for OSS) | ||
| 2018 | Eric | merged PEP 554 implementation for internal testing use | ||
| 2018 | increasing usage of subinterpreters (JEP, OpenStack Ceph) |
|||
| 2018 | PyCon: lots of excitement about per-interpreter GIL | |||
| 2018 | Eric | moved GC and other state to PyInterpreterState
|
||
| 2018 (2H) | core devs | core governance | ||
| 2018 (Aug) | Eric | created multi-core-python project (copied notes) |
||
| 2018 | Eric | (baby #5) | ||
| 2019 | Victor | moving stuff out of public C-API | ||
| 2019 | Eric, et al | increasing collaboration (e.g. Facebook/Instagram) | ||
| 2019 | Eric | requested a decision on PEP 554 | ||
| 2019 | Eric | PyCon talk | ||
| 2019 | Eric | finally shifted gears to interp isolation | ||
| 2019 | Eric | started working on c-analyzer | ||
| 2019 | Eric, Potts | discussions w/ Davin Potts @ multiprocessing | ||
| 2019 | Eric, et al | core sprint | ||
| 2020 | Eric | added c-analyzer | ||
| 2020 | covid! | |||
| 2020 | core team | starting to isolate stdlib ext. modules | ||
| 2021 (Mar) | Guido, Eric | faster-cpython team @MS created | ||
| 2021 | Eric | finished consolidating globals | ||
| 2021 | Eric | started moving global state to interpreter | ||
| 2022 | Elizondo, Eric | PEP 683 | ||
| 2022 | Eric | PEP 684 | ||
| 2022 | Eric | still moving global state to interpreter | ||
| 2022 | Eric | trick for static builtin types | ||
| 2022 | Eric | per-interpreter allocators | ||
| 2023 | Elizondo | PEP 683 impl. landed | ||
(expand)
This is mostly about working on PEP 554 and on phase 2.
| date | who | core Python (multi-core-related) |
Python community (multi-core-related) |
other |
|---|---|---|---|---|
(expand)
Some ideas:
- add subinterpreter serial-execution mode
- add a high-level CSP module to the stdlib
- “frozen” objects
- read-only views
- "owned" objects (transfer ownership)
- named subinterpreters
- refcounts in own memory page
- faster/cleaner interpreter startup
- faster/cleaner subinterpreter startup (e.g. share some modules)
- more efficient sharing between interpreters (e.g. builtins)
- copy-on-write
- refactor C-API to take interpreter arg
- factor out pickle-independent parts of multiprocessing
- leaner subinterpreters?
- better object immutability (e.g. truer immutability, immutable mode)
- isolated object graphs
- "Local Interpreter Lock"
- issue #24991: Define instance mutability explicitly on type objects
- others in PEP 554
This is subsets of the overall timeline, grouped by category.
(expand)
(expand)
???? - new GIL implementation ???? - first real attempt to remove GIL ???? - gilectomy ???? - "no-gil" 2022 - PEP 703
(expand)
These are things that impacted the project's momentum and throughput.
...
(expand)
(expand)
(expand)
2007 - PEP 3121 2015 - PEP 489 accepted 2016 - PEP 573 2020 - PEP 630 ???? - porting stdlib to multi-phase init 2022 - PEP 687
(expand)
???? - support for multiple interpreters added (PyInterpreterState) ???? - GILState API added ... 2017 - added _PyRuntimeState (and Includes/internal/) 2018 - moved GC and other state to PyInterpreterState ... 2020 - added c-analyzer 2022 - PEP 683 2022 - PEP 684 2022 - finished globals consolidation 2023 - started working on per-interpreter GIL full-time 2023 - finished interpreter isolation
(expand)
... 2017 - opened PEP 554 2018 - added _xxsubinterpreters module (PEP 554 low-level implementation) ...
(expand)
runtime cleanup:
| date | Python version |
author | proposal | summary | notes |
|---|---|---|---|---|---|
| 2012 | Coghlan | PEP 432 | |||
| 2019 | Stinner | PEP 587 | (sort of) replaces PEP 432 | ||
| Stinner | opaque types | ||||
| Stinner | distinct internal API |
extension module isolation:
| date | Python version |
author | proposal | summary | notes |
|---|---|---|---|---|---|
| 2007 | 3.0 | v. Löwis | PEP 3121 | ext. module init/fini | |
| 2009 | 3.2 | v. Löwis | PEP 384 | stable ABI | |
| 2013 | 3.5 | Viktorin | PEP 489 | multi-phase init | |
| ??? | Viktorin | -- | stable ABI fixes | ||
| 2020 | 3.? | Viktorin | PEP 630 | ext. isolation howto | |
| 2022 | 3.12 | Aasland, Viktorin | PEP 687 | isolating stdlib ext. modules | |
interpreter isolation:
| date | Python version |
author | proposal | summary | notes |
|---|---|---|---|---|---|
| Elizondo,Snow | PEP 683 | immortal objects | |||
| PEP 684 | per-interpreter GIL | ||||
other related:
| date | Python version |
author | proposal | summary | notes |
|---|---|---|---|---|---|
| 2003 | 2.3 | PEP 311 | Hammond | GILState API | |
| 2008 | -- | PEP 583 | Yasskin | A Concurrency Memory Model for Python | |
| 2018 | 3.? | PEP 579 | Demeyer | Refactoring C functions and methods | |
| 2019 | 3.8 | PEP 590 | Shannon, Demeyer | Vectorcall |
(expand)
(expand)
(expand)
(expand)
| date | event | kind | presenter | title | notes |
|---|---|---|---|---|---|
| 2015 | Utah PUG | talk | Eric | Multiple Interpreters For Multi-Core Support |
summary | slides | notes |
| 2018 | Language Summit | talk | Eric | Multi-core via Subinterpreters | slides alt slides |
| 2018 (Dec) | Enterprise Python Summit | lightning | Eric | Multi-core Python | |
| with Subinterpreters | slides | ||||
| 2019 | MS brownbag | talk | Eric | to GIL or not to GIL: the Future of Multi-Core (C)Python |
(see PyCon talk) |
| 2019 | Language Summit | talk | Eric | ??? | |
| 2019 | PyCon US | talk | Eric | to GIL or not to GIL: the Future of Multi-Core (C)Python |
slides | video |
| 2019 | Core Sprint | talk | Eric, Joannah | slides | |
| 2020 | Language Summit | talk | Eric | A Retrospective | slides |
| 2021 | PyCon APAC | keynote | Eric | slides | video | |
| 2022 | Language Summit | talk | Eric | Reaching a Per-Interpreter GIL | slides |
| 2023 | PyCon US | talk | Eric | A Per-Interpreter GIL | slides | video |
| 2023 | PyCon US | lightning | Eric | Isolating Extension Modules | slides |