Skip to content

gh-111681: Fix doctests in typing.rst and remove unused imports #111682

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Nov 3, 2023

Conversation

sobolevn
Copy link
Member

@sobolevn sobolevn commented Nov 3, 2023

Without globs it would fail:

» ./python.exe -m test test_typing
Using random seed: 1924694082
0:00:00 load avg: 1.95 Run 1 test sequentially
0:00:00 load avg: 1.95 [1/1] test_typing
test test_typing failed -- Traceback (most recent call last):
  File "/Users/sobolev/Desktop/cpython2/Lib/doctest.py", line 2263, in runTest
    raise self.failureException(self.format_failure(new.getvalue()))
AssertionError: Failed doctest test for typing.rst
  File "/Users/sobolev/Desktop/cpython2/Doc/library/typing.rst", line 0

----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython2/Doc/library/typing.rst", line 648, in typing.rst
Failed example:
    Z[int, [dict, float]]
Expected:
    __main__.Z[int, [dict, float]]
Got:
    Z[int, [dict, float]]
----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython2/Doc/library/typing.rst", line 669, in typing.rst
Failed example:
    X[int, str]
Expected:
    __main__.X[[int, str]]
Got:
    X[[int, str]]
----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython2/Doc/library/typing.rst", line 671, in typing.rst
Failed example:
    X[[int, str]]
Expected:
    __main__.X[[int, str]]
Got:
    X[[int, str]]
----------------------------------------------------------------------
File "/Users/sobolev/Desktop/cpython2/Doc/library/typing.rst", line 1996, in typing.rst
Failed example:
    Alias.__module__
Expected:
    '__main__'
Got nothing

📚 Documentation preview 📚: https://cpython-previews--111682.org.readthedocs.build/

@AlexWaygood
Copy link
Member

AlexWaygood commented Nov 3, 2023

I think we already run these doctests via Sphinx in CI. If there are certain doctests that aren't being run via that CI job, then I think we should investigate why that is, rather than running the entire doctest suite twice in CI

@AlexWaygood
Copy link
Member

AlexWaygood commented Nov 3, 2023

I think we already run these doctests via Sphinx in CI.

See https://github.com/python/cpython/actions/runs/6741804289/job/18326927659 for a recent run, in which 55/55 typing doctests were reported to have passed (but maybe there's some that the job missed)

@sobolevn
Copy link
Member Author

sobolevn commented Nov 3, 2023

@AlexWaygood as you can see there's something wrong with Doctest CI job: https://github.com/python/cpython/actions/runs/6743114958/job/18330437061?pr=111682 It passes on a clearly broken example. But, my new job fails as it should: https://github.com/python/cpython/actions/runs/6743114958/job/18330434893?pr=111682

I know this because I've already fixed multiple doctests for other modules.

@AlexWaygood
Copy link
Member

@AlexWaygood as you can see there's something wrong with Doctest CI job: https://github.com/python/cpython/actions/runs/6743114958/job/18330437061?pr=111682 It passes on a clearly broken example.

In which case, I think we should either try to fix the broken CI job, or, if it's useless, remove it from CI. @AA-Turner, any idea why the doctest CI job doesn't seem to be doing what it should be here? :)

Comment on lines 651 to 652
>>> 1 + 2
4
Copy link
Member

@AlexWaygood AlexWaygood Nov 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this isn't actually doctested, I don't think, since you don't have the .. doctest:: directive above this block. (I've just pushed to your branch to add it.)

Copy link
Member Author

@sobolevn sobolevn Nov 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is a doctest, clearly:

» ./python.exe -m doctest Doc/library/typing.rst
**********************************************************************
File "Doc/library/typing.rst", line 651, in typing.rst
Failed example:
    1 + 2
Expected:
    4
Got:
    3
**********************************************************************
1 items had failures:
   1 of  48 in typing.rst
***Test Failed*** 1 failures.

It works the same way with and without .. doctest::.
The problem might that Sphinx does not recognize doctests without this directive, but this is clearly a Sphinx's own issue.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that was what I meant, sorry for not being clear: that it wouldn't be picked up by sphinx's doctest job :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might need to use doctest_test_doctest_blocks from https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might need to use doctest_test_doctest_blocks from https://www.sphinx-doc.org/en/master/usage/extensions/doctest.html

We can try enabling that, sure, but that should be a separate issue/PR. This PR is about fixing broken doctests in typing.rst, not changing the way we determine in CI which doctests should be run. The majority of these that need to be tested already are tested, using the .. doctest:: directive.

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like the current CI job is not broken at all. The "clearly broken" example you added correctly failed in CI with the Sphinx doctest job after the .. doctest:: marker was added: https://github.com/python/cpython/actions/runs/6743594059/job/18331803827?pr=111682

@@ -1954,7 +1959,7 @@ without the dedicated syntax, as documented below.

.. doctest::

>>> from typing import ParamSpec
>>> from typing import ParamSpec, get_origin
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't mind the change you're making here. But the reason why this doesn't currently fail when Sphinx runs our doctests in CI is because of this, which is run before every doctest block:

.. testsetup:: *
import typing
from dataclasses import dataclass
from typing import *

@bedevere-app
Copy link

bedevere-app bot commented Nov 3, 2023

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

@sobolevn
Copy link
Member Author

sobolevn commented Nov 3, 2023

It seems like the current CI job is not broken at all

It might not be broken, it just allows a lot of broken doctests in.
I think that regular doctests must always work, not just some sub-selection of them.

Because what happens in reality is that a lot of code blocks / example do not have .. doctest directive and they become outdated, maintainers do not notice it, users get upset.

@AlexWaygood
Copy link
Member

AlexWaygood commented Nov 3, 2023

Because what happens in reality is that a lot of code blocks / example do not have .. doctest directive and they become outdated, maintainers do not notice it, users get upset.

I agree this situation is suboptimal. But that makes me think that we should do one of the following two things:

  1. Get rid of the current doctest job in CI, try to make sure all doctests in all the docs are run via unittest
  2. Explore using doctest_test_doctest_blocks, as you suggested in gh-111681: Fix doctests in typing.rst and remove unused imports #111682 (comment), so that all doctest blocks in all the docs are tested by Sphinx by default.

It makes no sense to me to do what this PR currently proposes: run most of the doctests in typing.rst twice in CI.

@sobolevn
Copy link
Member Author

sobolevn commented Nov 3, 2023

Yes, I agree. I will convert it to a very minimalistic one: just remove unused imports and fix doctests a bit, without adding new tests to the suite.

@sobolevn sobolevn changed the title gh-111681: Fix doctests in typing.rst and run it in CI gh-111681: Fix doctests in typing.rst and remove unused imports Nov 3, 2023
Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@AlexWaygood AlexWaygood added needs backport to 3.11 only security fixes needs backport to 3.12 only security fixes labels Nov 3, 2023
@AlexWaygood AlexWaygood enabled auto-merge (squash) November 3, 2023 11:49
@AlexWaygood AlexWaygood merged commit ccc8caa into python:main Nov 3, 2023
@miss-islington-app
Copy link

Thanks @sobolevn for the PR, and @AlexWaygood for merging it 🌮🎉.. I'm working now to backport this PR to: 3.11, 3.12.
🐍🍒⛏🤖

@miss-islington-app
Copy link

Sorry, @sobolevn and @AlexWaygood, I could not cleanly backport this to 3.12 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker ccc8caa8587103c4ccf617ba106cef63344504dd 3.12

@miss-islington-app
Copy link

Sorry, @sobolevn and @AlexWaygood, I could not cleanly backport this to 3.11 due to a conflict.
Please backport using cherry_picker on command line.

cherry_picker ccc8caa8587103c4ccf617ba106cef63344504dd 3.11

@AlexWaygood
Copy link
Member

@sobolevn, can I leave the backports with you? :)

@sobolevn
Copy link
Member Author

sobolevn commented Nov 3, 2023

Will do, I will also create a new issue about our CI + doctest.

@AlexWaygood AlexWaygood assigned sobolevn and unassigned AlexWaygood Nov 3, 2023
@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot AMD64 Windows11 Bigmem 3.x has failed when building commit ccc8caa.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1079/builds/2658) and take a look at the build logs.
  4. Check if the failure is related to this commit (ccc8caa) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1079/builds/2658

Summary of the results of the build (if available):

==

Click to see traceback logs
remote: Enumerating objects: 8, done.        
remote: Counting objects:  12% (1/8)        
remote: Counting objects:  25% (2/8)        
remote: Counting objects:  37% (3/8)        
remote: Counting objects:  50% (4/8)        
remote: Counting objects:  62% (5/8)        
remote: Counting objects:  75% (6/8)        
remote: Counting objects:  87% (7/8)        
remote: Counting objects: 100% (8/8)        
remote: Counting objects: 100% (8/8), done.        
remote: Compressing objects:  12% (1/8)        
remote: Compressing objects:  25% (2/8)        
remote: Compressing objects:  37% (3/8)        
remote: Compressing objects:  50% (4/8)        
remote: Compressing objects:  62% (5/8)        
remote: Compressing objects:  75% (6/8)        
remote: Compressing objects:  87% (7/8)        
remote: Compressing objects: 100% (8/8)        
remote: Compressing objects: 100% (8/8), done.        
remote: Total 8 (delta 0), reused 1 (delta 0), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to 'ccc8caa8587103c4ccf617ba106cef63344504dd'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at ccc8caa858 gh-111681: minor fixes to typing doctests; remove unused imports in `test_typing` (#111682)
Switched to and reset branch 'main'

Could Not Find R:\buildarea\3.x.ambv-bb-win11.bigmem\build\Lib\*.pyc
The system cannot find the file specified.
Could Not Find R:\buildarea\3.x.ambv-bb-win11.bigmem\build\PCbuild\python*.zip

Could Not Find R:\buildarea\3.x.ambv-bb-win11.bigmem\build\PCbuild\python*.zip

@bedevere-bot
Copy link

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot x86-64 macOS 3.x has failed when building commit ccc8caa.

What do you need to do:

  1. Don't panic.
  2. Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
  3. Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/366/builds/5809) and take a look at the build logs.
  4. Check if the failure is related to this commit (ccc8caa) or if it is a false positive.
  5. If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/366/builds/5809

Summary of the results of the build (if available):

==

Click to see traceback logs
Note: switching to 'ccc8caa8587103c4ccf617ba106cef63344504dd'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at ccc8caa858 gh-111681: minor fixes to typing doctests; remove unused imports in `test_typing` (#111682)
Switched to and reset branch 'main'

configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: libpython3.13d.a(dynamic_annotations.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: libpython3.13d.a(pymath.o) has no symbols
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups
ld: warning: -undefined dynamic_lookup may not work with chained fixups

make: *** [buildbottest] Error 3

AlexWaygood added a commit to AlexWaygood/cpython that referenced this pull request Nov 9, 2023
…s in `test_typing` (python#111682)

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
@bedevere-app
Copy link

bedevere-app bot commented Nov 13, 2023

GH-112035 is a backport of this pull request to the 3.12 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.12 only security fixes label Nov 13, 2023
@bedevere-app
Copy link

bedevere-app bot commented Nov 13, 2023

GH-112037 is a backport of this pull request to the 3.11 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.11 only security fixes label Nov 13, 2023
AlexWaygood added a commit to AlexWaygood/cpython that referenced this pull request Nov 13, 2023
AlexWaygood added a commit that referenced this pull request Nov 13, 2023
…ts in `test_typing` (#111682) (#112035)

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Nikita Sobolev <mail@sobolevn.me>
aisk pushed a commit to aisk/cpython that referenced this pull request Feb 11, 2024
…s in `test_typing` (python#111682)

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Glyphack pushed a commit to Glyphack/cpython that referenced this pull request Sep 2, 2024
…s in `test_typing` (python#111682)

Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
skip news tests Tests in the Lib/test dir
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants