Test the safe parts of notebooks that run on hardware in every CI run #1173
Labels
Comments
github-merge-queue bot
pushed a commit
that referenced
this issue
May 2, 2024
This uses the approach discussed in #1174 to test notebooks that submit jobs to IBM Quantum. The script behaviour has changed a bit: * We now have a list of `notebooks_no_mock`, these could include notebooks with large circuits or that demonstrate features not available on the fake backend. Future work could partially test these using the approach in [#1173](#1173). * By default, it runs all notebooks except `notebooks_no_mock`, with the patched package. * `--submit-jobs` will run **all** notebooks, without the patched package. * `--only-unmockable` will run only `notebooks_no_mock`, without the patched package. The following notebooks do **not** work with this patch, but some of them might not have worked anyway. * `tutorials/variational-quantum-eigensolver/notebook.ipynb` * `tutorials/submitting-transpiled-circuits/notebook.ipynb` * `tutorials/build-repitition-codes/notebook.ipynb` --------- Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
frankharkins
added a commit
to frankharkins/documentation
that referenced
this issue
Jul 22, 2024
This uses the approach discussed in Qiskit#1174 to test notebooks that submit jobs to IBM Quantum. The script behaviour has changed a bit: * We now have a list of `notebooks_no_mock`, these could include notebooks with large circuits or that demonstrate features not available on the fake backend. Future work could partially test these using the approach in [Qiskit#1173](Qiskit#1173). * By default, it runs all notebooks except `notebooks_no_mock`, with the patched package. * `--submit-jobs` will run **all** notebooks, without the patched package. * `--only-unmockable` will run only `notebooks_no_mock`, without the patched package. The following notebooks do **not** work with this patch, but some of them might not have worked anyway. * `tutorials/variational-quantum-eigensolver/notebook.ipynb` * `tutorials/submitting-transpiled-circuits/notebook.ipynb` * `tutorials/build-repitition-codes/notebook.ipynb` --------- Co-authored-by: Eric Arellano <14852634+Eric-Arellano@users.noreply.github.com>
|
@frankharkins think we should close this as not worth the complexity? In hello world, we would only be able to test about 25% of the notebook with this strategy, which isn't that big of a win. Removing tutorials changed the value, imo. This is what would be impacted: documentation/scripts/config/notebook-testing.toml Lines 48 to 54 in 632cb2c I lean towards closing this as not worth it. The cron job does a good job as is. |
|
Yeah I agree |
|
an idea is to have the notebooks modified such that our test_* backends will be used for pipeline testing wherever possible |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We cannot easily test notebooks that run on hardware in CI for every PR because a) there is often a queue and b) it costs actual resources. So, we currently skip those notebooks in PR runs:
documentation/scripts/nb-tester/test-notebook.py
Lines 33 to 43 in 8642507
We then run those notebooks every two weeks in a cron job with GH Actions.
However, we can actually run part of the notebooks safely! We can run every code cell up until the point when we call
Estimator.run()andSampler.run(), or do some other slow thing with Qiskit Runtime. All the code up to that point is safe to run.We can add a Jupyter notebook tag like
submits-jobto the code cells that use hardware. Then teachnb-testerto only run up to that code block, but stop there. We wouldn't run any subsequent code blocks since they may depend on the tagged code block.The text was updated successfully, but these errors were encountered: