Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gh-91051: allow setting a callback hook on PyType_Modified #97875

Merged
merged 21 commits into from
Oct 21, 2022
Merged

gh-91051: allow setting a callback hook on PyType_Modified #97875

merged 21 commits into from
Oct 21, 2022

Conversation

carljm
Copy link
Member

@carljm carljm commented Oct 4, 2022 •
edited by bedevere-bot
Loading

@carljm
Copy link
Member Author

carljm commented Oct 5, 2022

PyPerformance results show neutral impact on geometric mean of benchmarks: https://gist.github.com/ericsnowcurrently/01b79f54360efa52d4c5bb536cc885e5

@carljm
Merge branch 'main' into typewatch
1af2a98 
* main: (53 commits)
  pythongh-94808: Coverage: Test that maximum indentation level is handled (python#95926)
  pythonGH-88050: fix race in closing subprocess pipe in asyncio  (python#97951)
  pythongh-93738: Disallow pre-v3 syntax in the C domain (python#97962)
  pythongh-95986: Fix the example using match keyword (python#95989)
  pythongh-97897: Prevent os.mkfifo and os.mknod segfaults with macOS 13 SDK (pythonGH-97944)
  pythongh-94808: Cover `PyUnicode_Count` in CAPI (python#96929)
  pythongh-94808: Cover `PyObject_PyBytes` case with custom `__bytes__` method (python#96610)
  pythongh-95691: Doc BufferedWriter and BufferedReader (python#95703)
  pythonGH-88968: Add notes about socket ownership transfers (python#97936)
  pythongh-96865: [Enum] fix Flag to use CONFORM boundary (pythonGH-97528)
  pythongh-65961: Raise `DeprecationWarning` when `__package__` differs from `__spec__.parent` (python#97879)
  docs(typing): add "see PEP 675" to LiteralString (python#97926)
  pythongh-97850: Remove all known instances of module_repr() (python#97876)
  I changed my surname early this year (python#96671)
  pythongh-93738: Documentation C syntax (:c:type:<C type> -> :c:expr:<C type>) (python#97768)
  pythongh-91539: improve performance of get_proxies_environment  (python#91566)
  build(deps): bump actions/stale from 5 to 6 (python#97701)
  pythonGH-95172 Make the same version `versionadded` oneline (python#95172)
  pythongh-88050: Fix asyncio subprocess to kill process cleanly when process is blocked (python#32073)
  pythongh-93738: Documentation C syntax (Function glob patterns -> literal markup) (python#97774)
  ...
@carljm
Copy link
Member Author

carljm commented Oct 6, 2022

For the record, discussed this in person with @markshannon and agreed to expand it so that you watch individual types instead of subscribing only to all changes to all types. This will make it more similar to dictionary watchers (#31787). So I'm working on that update.

@carljm
Merge branch 'main' into typewatch
415ed49 
* main:
  pythonGH-97002: Prevent `_PyInterpreterFrame`s from backing more than one `PyFrameObject` (pythonGH-97996)
  pythongh-97973: Return all necessary information from the tokenizer (pythonGH-97984)
  fixes pythongh-96078: os.sched_yield release the GIL while calling sched_yield(2). (pythongh-97965)
  pythongh-65961: Do not rely solely on `__cached__` (pythonGH-97990)
  pythongh-97850: Remove the open issues section from the import reference (python#97935)
  Docs: pin sphinx-lint (pythonGH-97992)
  pythongh-94590: add signatures to operator itemgetter, attrgetter, methodcaller (python#94591)
  Add Pynche's move to the What's new in 3.11 (python#97974)
  pythongh-97781: Apply changes from importlib_metadata 5. (pythonGH-97785)
  pythongh-86482: Document assignment expression need for ()s (python#23291)
  pythongh-97943: PyFunction_GetAnnotations should return a borrowed reference. (python#97949)
@carljm
Merge branch 'main' into typewatch
29e38df 
* main:
  bpo-35540 dataclasses.asdict now supports defaultdict fields (pythongh-32056)
  pythonGH-91052: Add C API for watching dictionaries (pythonGH-31787)
  bpo-38693: Use f-strings instead of str.format() within importlib (python#17058)
@carljm
Merge branch 'main' into typewatch
0de5aff 
* main:
  pythongh-86298: Ensure that __loader__ and __spec__.loader agree in warnings.warn_explicit() (pythonGH-97803)
  pythongh-82874: Convert remaining importlib format uses to f-str. (python#98005)
  Docs: Fix backtick errors found by sphinx-lint (python#97998)
  pythongh-97850: Remove deprecated functions from `importlib.utils` (python#97898)
  Remove extra spaces in custom openSSL documentation. (python#93568)
  pythonGH-90985: Revert  "Deprecate passing a message into cancel()" (python#97999)
@carljm
Merge branch 'main' into typewatch
27a4980 
* main: (38 commits)
  pythongh-92886: make test_ast pass with -O (assertions off) (pythonGH-98058)
  pythongh-92886: make test_coroutines pass with -O (assertions off) (pythonGH-98060)
  pythongh-57179: Add note on symlinks for os.walk (python#94799)
  pythongh-94808: Fix regex on exotic platforms (python#98036)
  pythongh-90085: Remove vestigial -t and -c timeit options (python#94941)
  pythonGH-83901: Improve Signature.bind error message for missing keyword-only params (python#95347)
  pythongh-61105: Add default param, note on using cookiejar subclass (python#95427)
  pythongh-96288: Add a sentence to `os.mkdir`'s docstring. (python#96271)
  pythongh-96073: fix backticks in NEWS entry (pythonGH-98056)
  pythongh-92886: [clinic.py] raise exception on invalid input instead of assertion (pythonGH-98051)
  pythongh-97997: Add col_offset field to tokenizer and use that for AST nodes (python#98000)
  pythonGH-88968: Reject socket that is already used as a transport (python#98010)
  pythongh-96346: Use double caching for re._compile() (python#96347)
  pythongh-91708: Revert params note in urllib.parse.urlparse table (python#96699)
  pythongh-96265: Fix some formatting in faq/design.rst (python#96924)
  pythongh-73196: Add namespace/scope clarification for inheritance section (python#92840)
  pythongh-97646: Change `.js` and `.mjs` files mimetype to conform to RFC 9239 (python#97934)
  pythongh-97923: Always run Ubuntu SSL tests with others in CI (python#97940)
  pythongh-97956: Mention `generate_global_objects.py` in `AC How-To` (python#97957)
  pythongh-96959: Update HTTP links which are redirected to HTTPS (python#98039)
  ...
@carljm
Merge branch 'main' into typewatch
d67d6fd 
* main:
  pythongh-68686: Retire eptag ptag scripts (python#98064)
  pythongh-97922: Run the GC only on eval breaker (python#97920)
  GitHub Workflows security hardening (python#96492)
  Add `@ezio-melotti` as codeowner for `.github/`. (python#98079)
  pythongh-97913 Docs: Add walrus operator to the index (python#97921)
  [doc] Fix broken links to C extensions accelerating stdlib modules (python#96914)
  pythongh-97822: Fix http.server documentation reference to test() function (python#98027)
  pythongh-91052: Add PyDict_Unwatch for unwatching a dictionary (python#98055)
  pythonGH-98023: Change default child watcher to PidfdChildWatcher on supported systems (python#98024)
  pythonGH-94182: Run the PidfdChildWatcher on the running loop (python#94184)
@carljm
Multiple watchers and per-type watching
cb71ae9
@carljm
Update NEWS blurb
ab1606c
@carljm
Copy link
Member Author

carljm commented Oct 8, 2022

@markshannon I think this is ready for your review now; I've implemented the changes we discussed.

You may want to take a look at how I chose to document the behavior of callback "aggregation" (the "only call back if valid version tag" stuff); I aimed to accurately describe the behavior without getting into how it is implemented internally, and without promising that it won't change.

Also note that I am calling assign_version_tag in PyType_Watch. I don't think this is strictly required for correctness, as presumably in real usage anyone who is calling PyType_Watch is probably also doing a lookup on the type around the same time which will populate the version tag. But I think it makes the behavior more intuitive and less potentially confusing, since it means you'll always get called back on the next modification to the type after you watch it. Let me know if you think we should do something different here.

(I also removed the method cache struct from the signature of assign_version_tag -- this seems fine since it was unused there, and this is a private static function.)

@carljm
Fix doc build issues
26ddee2
@carljm
Fix typeobject size test
4783e07
@carljm
Merge branch 'main' into typewatch
71d48c1 
* main:
  Minor edits to the Descriptor HowTo Guide (pythonGH-24901)
  Fix link to Lifecycle of a Pull Request in CONTRIBUTING (python#98102)
  pythonGH-94597: deprecate `SafeChildWatcher`, `FastChildWatcher` and `MultiLoopChildWatcher` child watchers  (python#98089)
  Auto-cancel old builds when new commit pushed to branch (python#98009)
  pythongh-95011: Migrate syslog module to Argument Clinic (pythonGH-95012)
@carljm
Fix issue number in whatsnew (and add for dict watchers too)
21f42ee
itamaro
itamaro reviewed Oct 9, 2022
View reviewed changes
Doc/whatsnew/3.12.rst Show resolved Hide resolved
Doc/c-api/type.rst Show resolved Hide resolved
Lib/test/test_capi.py Show resolved Hide resolved
Lib/test/test_capi.py Show resolved Hide resolved
@carljm
Merge branch 'main' into typewatch
6e53514 
* main: (38 commits)
  pythongh-98251: Allow venv to pass along PYTHON* variables to pip and ensurepip when they do not impact path resolution (pythonGH-98259)
  Bpo-41246: IOCP Proactor avoid callback code duplication (python#21399)
  bpo-46364: Use sockets for stdin of asyncio only on AIX (python#30596)
  pythongh-98178: syslog() is not thread-safe on macOS (python#98213)
  Mark all targets in `Doc/Makefile` as `PHONY` (pythonGH-98189)
  pythongh-97982: Factorize PyUnicode_Count() and unicode_count() code (python#98025)
  pythongh-96265: Formatting changes for faq/general (python#98129)
  tutorial: remove "with single quotes" (python#98204)
  pythongh-97669: Remove Tools/scripts/startuptime.py (python#98214)
  signalmodule.c uses _PyErr_WriteUnraisableMsg() (python#98217)
  pythongh-97669: Fix test_tools reference leak (python#98216)
  pythongh-97669: Create Tools/patchcheck/ directory (python#98186)
  pythongh-65046: Link to logging cookbook from asyncio docs (python#98207)
  Formatting fixes in contextlib docs (python#98111)
  pythongh-95276: Add callable entry to the glossary (python#95738)
  pythongh-96130: Rephrase use of "typecheck" verb for clarity (python#98144)
  Fix some incorrect indentation around the main switch (python#98177)
  pythongh-98172: Fix formatting in `except*` docs (python#98173)
  pythongh-97982: Remove asciilib_count() (python#98164)
  pythongh-95756: Free and NULL-out code caches when needed (pythonGH-98181)
  ...
@carljm
Type watcher review comments
2b42bfe
erlend-aasland
erlend-aasland reviewed Oct 17, 2022
View reviewed changes
Copy link
Contributor

@erlend-aasland erlend-aasland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some comments regarding test coverage.

Objects/typeobject.c Show resolved Hide resolved
Objects/typeobject.c Outdated Show resolved Hide resolved
@carljm
Merge branch 'main' into typewatch
cdbe93b 
* main: (31 commits)
  pythongh-95913: Move subinterpreter exper removal to 3.11 WhatsNew (pythonGH-98345)
  pythongh-95914: Add What's New item describing PEP 670 changes (python#98315)
  Remove unused arrange_output_buffer function from zlibmodule.c. (pythonGH-98358)
  pythongh-98174: Handle EPROTOTYPE under macOS in test_sendfile_fallback_close_peer_in_the_middle_of_receiving (python#98316)
  pythonGH-98327: Reduce scope of catch_warnings() in _make_subprocess_transport (python#98333)
  pythongh-93691: Compiler's code-gen passes location around instead of holding it on the global compiler state (pythonGH-98001)
  pythongh-97669: Create Tools/build/ directory (python#97963)
  pythongh-95534: Improve gzip reading speed by 10% (python#97664)
  pythongh-95913: Forward-port int/str security change to 3.11 What's New in main (python#98344)
  pythonGH-91415: Mention alphabetical sort ordering in the Sorting HOWTO (pythonGH-98336)
  pythongh-97930: Merge with importlib_resources 5.9 (pythonGH-97929)
  pythongh-85525: Remove extra row in doc (python#98337)
  pythongh-85299: Add note warning about entry point guard for asyncio example (python#93457)
  pythongh-97527: IDLE - fix buggy macosx patch (python#98313)
  pythongh-98307: Add docstring and documentation for SysLogHandler.createSocket (pythonGH-98319)
  pythongh-94808: Cover `PyFunction_GetCode`, `PyFunction_GetGlobals`, `PyFunction_GetModule` (python#98158)
  pythonGH-94597: Deprecate child watcher getters and setters (python#98215)
  pythongh-98254: Include stdlib module names in error messages for NameErrors (python#98255)
  Improve speed. Reduce auxiliary memory to 16.6% of the main array. (pythonGH-98294)
  [doc] Update logging cookbook with an example of custom handling of levels. (pythonGH-98290)
  ...
@carljm
Review comments
4329743
@carljm
Copy link
Member Author

carljm commented Oct 17, 2022

Thanks @erlend-aasland, updated the test coverage according to your comments.

@markshannon I think this is ready, pretty straightforward along the same lines as dict watchers, would be great to just get it in and stop having to resolve conflicts.

@erlend-aasland
Copy link
Contributor

Thanks @erlend-aasland, updated the test coverage according to your comments.

Thanks! Looks good to me.

markshannon
markshannon reviewed Oct 20, 2022
View reviewed changes
Objects/typeobject.c Outdated Show resolved Hide resolved
markshannon
markshannon approved these changes Oct 20, 2022
View reviewed changes
Copy link
Member

@markshannon markshannon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One suggestion to speedup a loop (which you may ignore if you think performance really doesn't matter here)

@carljm
Merge branch 'main' into typewatch
8bba5da 
* main: (40 commits)
  pythongh-98461: Fix source location in comprehensions bytecode (pythonGH-98464)
  pythongh-98421: Clean Up PyObject_Print (pythonGH-98422)
  pythongh-98360: multiprocessing now spawns children on Windows with correct argv[0] in virtual environments (pythonGH-98462)
  CODEOWNERS: Become a typing code owner (python#98480)
  [doc] Improve logging cookbook example. (pythonGH-98481)
  Add more tkinter.Canvas tests (pythonGH-98475)
  pythongh-95023: Added os.setns and os.unshare functions (python#95046)
  pythonGH-98363: Presize the list for batched() (pythonGH-98419)
  pythongh-98374: Suppress ImportError for invalid query for help() command. (pythongh-98450)
  typing tests: `_overload_dummy` raises `NotImplementedError`, not `RuntimeError` (python#98351)
  pythongh-98354: Add unicode check for 'name' attribute in _imp_create_builtin (pythonGH-98412)
  pythongh-98257: Make _PyEval_SetTrace() reentrant (python#98258)
  pythongh-98414: py.exe launcher does not use defaults for -V:company/ option (pythonGH-98460)
  pythongh-98417: Store int_max_str_digits on the Interpreter State (pythonGH-98418)
  Doc: Remove title text from internal links (python#98409)
  [doc] Refresh the venv introduction documentation, and correct the statement about VIRTUAL_ENV (pythonGH-98350)
  Docs: Bump sphinx-lint and fix unbalanced inline literal markup (python#98441)
  pythongh-92886: Replace assertion statements in `handlers.BaseHandler` to support running with optimizations (`-O`) (pythonGH-93231)
  pythongh-92886: Fix tests that fail when running with optimizations (`-O`) in `_test_multiprocessing.py` (pythonGH-93233)
  pythongh-92886: Fix tests that fail when running with optimizations (`-O`) in `test_py_compile.py` (pythonGH-93235)
  ...
@carljm
Short circuit watched bits check if no more bits are set
22d190b
@carljm
Copy link
Member Author

carljm commented Oct 20, 2022

@markshannon good suggestion, updated!

@markshannon
Copy link
Member

Thanks @carljm for implementing this.

@markshannon markshannon merged commit 82ccbf6 into python:main Oct 21, 2022
@carljm carljm deleted the typewatch branch October 21, 2022 15:35
@erlend-aasland erlend-aasland mentioned this pull request Nov 10, 2022
Open
@mattip mattip mentioned this pull request Nov 25, 2022
Closed
scoder added a commit to cython/cython that referenced this pull request Nov 29, 2022
@scoder
Add new "tp_watched" PyTypeObject field in Py3.12.
6add7fe 
See python/cpython#97875
facebook-github-bot pushed a commit to facebookincubator/cinder that referenced this pull request Aug 11, 2023
@facebook-github-bot
backport 3.12 type watchers
5900729 
Summary:
Backport python/cpython#97875 plus some later updates.

The only change in the backport is that instead of adding `tp_watched` field to the `PyTypeObject` struct, we instead steal 8 bits of `tp_flags`. This is because some extension modules contain statically-defined `PyTypeObject` structs, and we don't want to have to recompile all extensions. If we don't, we'd get junk data from `type->tp_watched` for such statically defined type structs.

There are no free bits in the lower 32 bits of `tp_flags`, but all the upper 32 bits are unused, since CPython supports 32-bit architectures. Since Cinder doesn't, we know those bits are present and free for our use, so we steal the lower 8 of them.

Reviewed By: itamaro

Differential Revision: D47202165

fbshipit-source-id: 370ac09e2bc7502742480cbbd0fb09804102f783
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@itamaro itamaro itamaro left review comments

@erlend-aasland erlend-aasland erlend-aasland approved these changes

@markshannon markshannon markshannon approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@carljm @erlend-aasland @markshannon @itamaro @bedevere-bot