Add a workflow to check for examples in docs #459

purva-thakre · 2024-02-02T22:43:08Z

Description

Fixes #289

Todos

Notable points that this PR has either accomplished or will accomplish.

TODO 1

Questions

Question1

Status

Ready to go

purva-thakre · 2024-02-02T22:43:50Z

To do: If I locally run make linkcheck or make doctest will add files c632cef

We don't want to add these files.

codecov · 2024-02-02T22:47:21Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (e0e89f9) 98.1% compared to head (3be51d8) 98.1%.

Additional details and impacted files

@@          Coverage Diff           @@
##           master    #459   +/-   ##
======================================
  Coverage    98.1%   98.1%           
======================================
  Files         160     160           
  Lines        3107    3108    +1     
  Branches      749     749           
======================================
+ Hits         3048    3049    +1     
  Misses         37      37           
  Partials       22      22

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

purva-thakre · 2024-02-03T14:55:27Z

Doctest is not correctly identifying the examples in the docstrings.

No tests are run in the added job.

Locally, I can see multiple failures due to float comparisons.

Step 1: We want this workflow to fail.

purva-thakre · 2024-02-07T02:28:53Z

Doctest is not correctly identifying the examples in the docstrings.

This should be fixed with 9813786. Apparently, make html has to be used before make doctest.

All the examples are correctly identified now. The issue that I am running into is that the Github Actions environment can't find any of the installed packages.

https://stackoverflow.com/a/75711927/10241324

purva-thakre · 2024-02-08T03:13:11Z

Fixed the import issue. Yay!

If poetry is used to install a package, have to make sure poetry run is used before the command we want to run. This is because poetry installs things in a virtual environment.

purva-thakre · 2024-02-08T03:19:33Z

@vprusso heads up! This PR is going to have a huge diff. I can't run make doctest for 1 folder. Here's a link to all that's failing: https://github.com/vprusso/toqito/actions/runs/7824351650/job/21346779560?pr=459#step:5:2040

Do you want me to ping you every time I finish up with 1 module?

It will be a bit of a tedious process. With my 2 commits below, you can see the number of failing tests have been reduced compared to the screenshot in my other comment.

I do not know why the number of tests also decreased unless doctest counts a syntax failure as a failing test as well. fa09a66 had 2 such failing lines.

purva-thakre · 2024-02-08T03:52:09Z

This comment has been fixed with a workaround in aaf0f9d

There are other options linked in this comment if the chosen workaround is not ideal.

~~To do: figure out a way to declare a global approximation for all float outputs.~~

use regex? https://documenter.juliadocs.org/stable/man/doctests/#Filtering-Doctests
https://stackoverflow.com/a/2428747/10241324

Or use available doctest options: https://github.com/scientific-python/pytest-doctestplus?tab=readme-ov-file#floating-point-comparison

https://docs.python.org/3/library/doctest.html#option-flags

toqito/nonlocal_games/xor_game.py

purva-thakre · 2024-02-08T14:16:10Z

toqito/states/max_mixed.py

-    >>> max_mixed(2, is_sparse=True)
-        <2x2 sparse matrix of type '<class 'numpy.float64'>'
+    >>> max_mixed(2, is_sparse=True) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
+    <2x2 sparse matrix of type '<class 'numpy.float64'>'


I had to use # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE here because I couldn't figure out what the difference between the expected output vs. what doctest got was even after I made sure the indentation for both was the same.

https://docs.python.org/3/library/doctest.html#option-flags

Hmm. I see. Wonder if this will not be an issue when/if we switch from matrix to np.array objects universally.

Wonder if this will not be an issue when/if we switch from matrix to np.array objects universally.

Don't know.

Although doctest was being very particular about the whitespace. Maybe 1 is supposed to be a tab while the other is not. ruff did not like it when I used tab to create the whitespace.

toqito/channel_metrics/diamond_norm.py

toqito/states/trine.py

toqito/rand/random_circulant_gram.py

purva-thakre · 2024-02-09T01:23:56Z

@vprusso This PR is ready for a review. Feel free to take your time.

Except for some of the outstanding comments in this PR, there were some last-minute failures that I chose to skip for now. 6d8ddf1 , 5add9a0

I also use SKIP, ELLIPSIS, NORMALIZE_WHITESPACE for the examples that fail no matter what I try. The expected output vs. provided output were virtually identical. 🤷🏾‍♀️ One of my comments links the option flags allowed by doctest.

vprusso

Few minor comments, but overall, looks good!

toqito/channel_metrics/diamond_norm.py

toqito/channels/reduction.py

toqito/matrices/cyclic_permutation.py

toqito/matrices/gen_gell_mann.py

vprusso · 2024-02-09T23:10:41Z

toqito/nonlocal_games/nonlocal_game.py

@@ -308,8 +308,8 @@ def quantum_value_lower_bound(
        ...                    pred_mat[a_alice, b_bob, x_alice, y_bob] = 1
        >>>
        >>> chsh = NonlocalGame(prob_mat, pred_mat)
-        >>> chsh.quantum_value_lower_bound()
-        0.8535533840915605
+        >>> chsh.quantum_value_lower_bound()   # doctest: +SKIP


Should we have the skip here or just truncate the value?

The very first thing I tried was truncating the value. Unfortunately, this function was calculating two different values: 0.75 or 0.85 whenever make doctest was run multiple times.

Easier option was to make doctest skip this.

toqito/rand/random_density_matrix.py

vprusso · 2024-02-09T23:20:10Z

toqito/state_props/is_unextendible_product_basis.py

@@ -39,7 +39,7 @@ def is_unextendible_product_basis(vecs: list[np.ndarray], dims: list[int]) -> tu
    >>>
    >>> non_upb_tiles = np.array([tile(i) for i in range(4)])
    >>> dims = np.array([3, 3])
-    >>> is_unextendible_product_basis(non_upb_tiles, dims)
+    >>> is_unextendible_product_basis(non_upb_tiles, dims)  # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE


Hmm, are these comments needed here (i.e. the doctest stuff)

Yes, otherwise the test would fail even when I made sure I copied & pasted the expected output. Plus, in this particular example, the output also contains float values which already causes issues with doctest.

That was my issue with these scenarios as well. How to make it clear to a user that they don't have to use the doctest stuff?

vprusso · 2024-02-09T23:21:06Z

toqito/states/bb84.py

@@ -21,12 +21,12 @@ def bb84() -> np.ndarray:

    >>> from toqito.states import bb84
    >>> x = bb84()
-    >>> print(f"|0> = {x[0][0].T}, \n |1> = {x[0][1].T}")
-    >>> print(f"|+> = {x[1][0].T}, \n |-> = {x[1][1].T}")
+    >>> print(f"|0> = {x[0][0].T}, \n |1> = {x[0][1].T}")  # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE


Hmm, same here. I'm wondering if we just need to change something to remove this ignore we're doing.

We are not ignoring this output. ELLIPSIS will make sure doctest is satisfied with a substring match and NORMALIZE_WHITESPACE will make sure doctest doesn't fail the test when the whitespace in the expected and calculated outputs is not identical.

vprusso · 2024-02-09T23:22:46Z

toqito/states/max_mixed.py

-    >>> max_mixed(2, is_sparse=True)
-        <2x2 sparse matrix of type '<class 'numpy.float64'>'
+    >>> max_mixed(2, is_sparse=True) # doctest: +ELLIPSIS, +NORMALIZE_WHITESPACE
+    <2x2 sparse matrix of type '<class 'numpy.float64'>'


Hmm. I see. Wonder if this will not be an issue when/if we switch from matrix to np.array objects universally.

toqito/states/trine.py

purva-thakre · 2024-02-11T15:52:55Z

Why are the following tests failing at random? Locally, if I see these 3 failures, I will run make doctest again and again until the workflow passes.

rebasing this branch does not help either.

The job did pass for

python 3.11 but not for 3.10,
reran the workflow for failed jobs, passed for 3.10 but failed for 3.12
reran the workflow for failed jobs, finally passed for 3.12.

purva-thakre · 2024-02-11T16:58:29Z

toqito/state_opt/state_distinguishability.py

-     [-0. -0.j  0. +0.j -0. -0.j -0. -0.j]
-     [ 0. -0.j -0. +0.j  0. +0.j  0. -0.j]
-     [ 0.5+0.j -0. +0.j  0. +0.j  0.5+0.j]]
+    >>> np.around(measurements[0], decimals=5)  # doctest: +SKIP


Had to add this SKIP here because the py312 job kept failing due to below:

purva-thakre added a commit that referenced this pull request Feb 3, 2024

Fix #459 (comment) by adding docs/docs/_build/ to gitignore

b7cbafb

purva-thakre force-pushed the doctest_examples branch from 7a1ccdf to edaaffe Compare February 7, 2024 02:10

purva-thakre force-pushed the doctest_examples branch from 5f28965 to 9230ed8 Compare February 8, 2024 02:50

purva-thakre added a commit that referenced this pull request Feb 8, 2024

Fix #459 (comment)

aaf0f9d

purva-thakre commented Feb 8, 2024

View reviewed changes

toqito/nonlocal_games/xor_game.py Show resolved Hide resolved

purva-thakre commented Feb 8, 2024

View reviewed changes

toqito/channel_metrics/diamond_norm.py Show resolved Hide resolved

purva-thakre commented Feb 8, 2024

View reviewed changes

toqito/states/trine.py Outdated Show resolved Hide resolved

purva-thakre commented Feb 8, 2024

View reviewed changes

toqito/rand/random_circulant_gram.py Show resolved Hide resolved

purva-thakre added a commit that referenced this pull request Feb 9, 2024

Fix #459 (comment) by adding docs/docs/_build/ to gitignore

73e64ab

purva-thakre added a commit that referenced this pull request Feb 9, 2024

Fix #459 (comment)

2f79571

purva-thakre force-pushed the doctest_examples branch from b3cf766 to 69f1ab0 Compare February 9, 2024 00:56

purva-thakre marked this pull request as ready for review February 9, 2024 01:04

vprusso requested changes Feb 9, 2024

View reviewed changes

purva-thakre mentioned this pull request Feb 10, 2024

Refactor functions in toqito to output np.array #468

Closed

12 tasks

purva-thakre added 7 commits February 11, 2024 07:14

add doctest job

9c2d1b1

make doctest or linkcheck adds files

f593d85

add folders to gitignore

e0d22ca

add to gitignore

2ff1bda

remove docs inside docs

7c1e898

add the whole output to gitignore

f5e3a8b

Fix #459 (comment) by adding docs/docs/_build/ to gitignore

63045fd

purva-thakre added 13 commits February 11, 2024 07:14

autoapi/matrix_props/majorizes/index

e3fdb4e

autoapi/states/bb84/index

13fa385

autoapi/perms/perm_sign/index

253e582

autoapi/matrix_ops/vectors_to_gram_matrix/index

1a07e49

autoapi/state_opt/symmetric_extension_hierarchy/index

56ef3ff

autoapi/nonlocal_games/nonlocal_game/index

c099ea2

autoapi/channel_ops/partial_channel/index

cd5d284

has_same_dims

fd35428

state_exclusion

d84c9ec

state_dist

6bf621f

skip state_distinguishability

9d837e7

skip nonlocal games test

091e37e

toqito/states/trine.py

c832892

purva-thakre force-pushed the doctest_examples branch from cdac790 to c832892 Compare February 11, 2024 14:46

purva-thakre added 2 commits February 11, 2024 09:23

add admonition for '%.2f' %

34199ae

note about SKIP in rand

703ad8c

change default verbosity fid_of_Sep

534046e

purva-thakre force-pushed the doctest_examples branch from 6348be9 to 534046e Compare February 11, 2024 16:55

purva-thakre commented Feb 11, 2024

View reviewed changes

purva-thakre added 2 commits February 11, 2024 11:00

add note in state_dist

627c88c

admonition for +ELLIPSIS, +NORMALIZE_WHITESPACE

3be51d8

purva-thakre force-pushed the doctest_examples branch from c1aef00 to 3be51d8 Compare February 11, 2024 17:21

purva-thakre requested a review from vprusso February 11, 2024 17:29

purva-thakre mentioned this pull request Feb 12, 2024

Fixing PPT and other misc. #469

Merged

vprusso approved these changes Feb 13, 2024

View reviewed changes

purva-thakre merged commit bfeae4f into master Feb 13, 2024
18 checks passed

purva-thakre deleted the doctest_examples branch February 13, 2024 23:21

purva-thakre mentioned this pull request Feb 14, 2024

Failing examples in the doctest workflow #473

Closed

purva-thakre mentioned this pull request Apr 6, 2024

Do we need warnings for examples in docstrings? #541

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a workflow to check for examples in docs #459

Add a workflow to check for examples in docs #459

purva-thakre commented Feb 2, 2024

purva-thakre commented Feb 2, 2024 •

edited

Loading

codecov bot commented Feb 2, 2024 •

edited

Loading

purva-thakre commented Feb 3, 2024

purva-thakre commented Feb 7, 2024 •

edited

Loading

purva-thakre commented Feb 8, 2024 •

edited

Loading

purva-thakre commented Feb 8, 2024 •

edited

Loading

purva-thakre commented Feb 8, 2024 •

edited

Loading

purva-thakre Feb 8, 2024 •

edited

Loading

vprusso Feb 9, 2024

purva-thakre Feb 10, 2024 •

edited

Loading

purva-thakre commented Feb 9, 2024 •

edited

Loading

vprusso left a comment

vprusso Feb 9, 2024

purva-thakre Feb 10, 2024 •

edited

Loading

vprusso Feb 9, 2024

purva-thakre Feb 10, 2024 •

edited

Loading

vprusso Feb 9, 2024

purva-thakre Feb 10, 2024

vprusso Feb 9, 2024

purva-thakre commented Feb 11, 2024 •

edited

Loading

purva-thakre Feb 11, 2024

Add a workflow to check for examples in docs #459

Add a workflow to check for examples in docs #459

Conversation

purva-thakre commented Feb 2, 2024

Description

Todos

Questions

Status

purva-thakre commented Feb 2, 2024 • edited Loading

codecov bot commented Feb 2, 2024 • edited Loading

Codecov Report

purva-thakre commented Feb 3, 2024

purva-thakre commented Feb 7, 2024 • edited Loading

purva-thakre commented Feb 8, 2024 • edited Loading

purva-thakre commented Feb 8, 2024 • edited Loading

purva-thakre commented Feb 8, 2024 • edited Loading

purva-thakre Feb 8, 2024 • edited Loading

Choose a reason for hiding this comment

vprusso Feb 9, 2024

Choose a reason for hiding this comment

purva-thakre Feb 10, 2024 • edited Loading

Choose a reason for hiding this comment

purva-thakre commented Feb 9, 2024 • edited Loading

vprusso left a comment

Choose a reason for hiding this comment

vprusso Feb 9, 2024

Choose a reason for hiding this comment

purva-thakre Feb 10, 2024 • edited Loading

Choose a reason for hiding this comment

vprusso Feb 9, 2024

Choose a reason for hiding this comment

purva-thakre Feb 10, 2024 • edited Loading

Choose a reason for hiding this comment

vprusso Feb 9, 2024

Choose a reason for hiding this comment

purva-thakre Feb 10, 2024

Choose a reason for hiding this comment

vprusso Feb 9, 2024

Choose a reason for hiding this comment

purva-thakre commented Feb 11, 2024 • edited Loading

purva-thakre Feb 11, 2024

Choose a reason for hiding this comment

purva-thakre commented Feb 2, 2024 •

edited

Loading

codecov bot commented Feb 2, 2024 •

edited

Loading

purva-thakre commented Feb 7, 2024 •

edited

Loading

purva-thakre commented Feb 8, 2024 •

edited

Loading

purva-thakre commented Feb 8, 2024 •

edited

Loading

purva-thakre commented Feb 8, 2024 •

edited

Loading

purva-thakre Feb 8, 2024 •

edited

Loading

purva-thakre Feb 10, 2024 •

edited

Loading

purva-thakre commented Feb 9, 2024 •

edited

Loading

purva-thakre Feb 10, 2024 •

edited

Loading

purva-thakre Feb 10, 2024 •

edited

Loading

purva-thakre commented Feb 11, 2024 •

edited

Loading