Add download option button to tutorials in docs

[nathanshammah]

Issue Description

Add download option button to tutorials in docs, including .ipynb option. However, we would like to have the source code as it is, in terms of .md files, without adding .ipynb files in mitiq/docs/source/examples/, as Jupyter notebooks are hard to track in version control on Github. So, ideally, somehow the Jupyter notebooks should be created somehow (at documentation build or with another script), but still appear in the version of the docs that is hosted on readthedocs. It's a tricky issue :).

Proposed Solution

Find solution in other sphinx documentations.

Additional References

See the top right button in this QuTiP notebook: https://nbviewer.org/urls/qutip.org/qutip-tutorials/tutorials-v4/time-evolution/003_qubit-dynamics.ipynb

[nathanshammah]

I'd like to work on this issue. The myST docs themselves show these options on the top right part of the page:

Note that, nicely, it changes the output format to just md and pdf if there is no code,

It seems to me that this part of the conf.py file may configure it

# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_title = ""
html_theme = "sphinx_book_theme"
html_logo = "_static/logo-wide.svg"
html_favicon = "_static/logo-square.svg"
html_theme_options = {
    "github_url": "https://github.com/executablebooks/myst-nb",
    "repository_url": "https://github.com/executablebooks/myst-nb",
    "repository_branch": "master",
    "home_page_in_toc": True,
    "path_to_docs": "docs",
    "show_navbar_depth": 1,
    "use_edit_page_button": True,
    "use_repository_button": True,
    "use_download_button": True,
    "launch_buttons": {
        "binderhub_url": "https://mybinder.org",
        "notebook_interface": "classic",
    },
}

[nathanshammah]

Bummer... it seems that this configuration is not supported by the pydata theme. I found a hack around this by surfing projects using the theme, in the NetworkX docs, where a button on the top of examples allows one to export the .pynb file.

[devilkiller-ag]

Hi, I am interested on working this issue but I need to understand this point more clearly: "So, ideally, somehow the Jupyter notebooks should be created somehow (at documentation build or with another script), but still appear in the version of the docs that is hosted on readthedocs. It's a tricky issue :).". Can you explain this point?

[andreamari]

Hi @devilkiller-ag, we currently use .md files for the source code of the documentation. They are better than .ipynb files for the source code, and so we would like to keep using .md files in the source code.

So, the optimal solution would be to convert the .md files to .ipynb notebooks only during the building process of the documentation.
E.g. by acting in some way in the following configuration files:
https://github.com/unitaryfund/mitiq/blob/master/.readthedocs.yml
https://github.com/unitaryfund/mitiq/tree/master/.github/workflows
https://github.com/unitaryfund/mitiq/blob/master/docs/source/conf.py
https://github.com/unitaryfund/mitiq/blob/master/docs/make.bat

We don't have a clear solution in mind. In fact, the most difficult part of this issue is exploring and proposing a solution, if it exists :-)

[devilkiller-ag]

Hi, Can we use Jupytext to convert our .md to .ipynb?
We can do something like this:

ipynb_file = os.path.splitext(md_file)[0] + '.ipynb'
jupytext.writef(jupytext.read(md_file), ipynb_file, fmt='notebook')

We can create a script that utilizes jupytext to convert the .md files to .ipynb notebooks. And call it in sphinx conf.py file before build.

[andre-a-alves]

I don't want to admit how much time I spent on this without non-hacky success, but here is what I learned. I hope it can be useful to others.

Themes

As Nate mentioned, the download button is not an option in the pydata theme that Mitiq currently uses. However, the button is a part of the sphinx_book_theme. The theme is built on top of the pydata theme, and it can by used by implementing the following two steps together:

1. In dev_requirements.txt, replace pydata-sphinx-theme==0.11.0 with sphinx-book-theme==1.0.1
2. In conf.py, change line 298 from html_theme = "pydata_sphinx_theme" to html_theme = "sphinx_book_theme"

After making those changes, I recommend the following html_theme_options on line 314:

html_theme_options = {
    "announcement": '<a href="https://unitaryhack.dev/">unitaryHACK</a> is \
    coming <b>May 26-Jun 13</b>! Get rewarded for contributing to open source \
    quantum software!',
    "repository_url": "https://github.com/unitaryfund/mitiq",
    "repository_branch": "master",
    "use_source_button": False,
    "use_repository_button": True,
    "use_issues_button": True,
    "use_download_button": True,
    "use_fullscreen_button": False,
    "path_to_docs": "docs/source",
}

Those options do a few things, including adding the download button. Unfortunately, the download only includes the .md file and not the .ipynb file. If you start out with a function to convert the .md files to .ipynb files, then you get errors in Sphinx, which make it difficult to clean up those generated .ipynb files automatically, since make will not run a cleanup script after an exit code 1, and you are left with a bunch of files to cleanup by hand. This is also where things started to feel very hacked together.

Using the sphinx_book_theme, MyST-NB includes a download button that has both .ipynb and .md files, but I cannot for the life of me figure out how they did it after reading through their conf.py file. Jupyter Book also falls into the same boat, though their configuration is even harder to understand since Sphinx is only part of their setup and they actually use Jupyter Book to build their site. I considered reaching out to their maintainers to ask how they do it, but it's a long weekend in a few countries, and I don't want to still be doing this next week. If someone else gets in touch with them, more power to you.

Converting Notebooks

If you convert notebooks from .md to .ipynb, keep in mind that links between books still reference the .md version. Therefore, the conversion makes the links point to non-existent files, which raises more Sphinx errors.

Workflow

It's hard to know how changes are being implemented without rerunning the books from a clean state, but that takes forever. Therefore, the best luck I've found for a fast pipeline is to delete all but one or two books while iterating, and then rollback the deletions once you think you have a solution worth testing full-on. This makes make clean html not take very long at all.

Next Steps

My next two approaches that I am considering trying are:

1. Build the paired notebooks and see if anything changes when both paired notebooks are present.
2. Create a python script that first copies the entire source directory into something like _source so that notebooks can be converted, links can be changed, and Sphinx can run error free, all while taking into account that Makefile and make.bat need to mirror each other, so a Makefile solution would be more work to maintain.
3. Some combination of the above.

Closing Thoughts

I don't really care about a bounty, and now I just want this solved. I hope someone else benefits from what I learned and gets this done in an elegant way.

Happy hacking

[devilkiller-ag]

Thanks @andre-a-alves, I am going to give this a try.

[andre-a-alves]

@devilkiller-ag Take a look at my branch

It copies the source directory to _source, and then it converts .md files into .ipynb files, but only if they are Jupyter Notebook files. It also only converts the files if the .md files have been modified. These decisions were based on:

1. Links in the .md files are to other .md files, so if the .md files are not present, all links break.
2. If we convert every notebook every time, then the cache for previously run notebooks doesn't work, and building docs takes forever.

I still see the following issues that need to be dealt with:

1. The following Sphinx warning shows up for every new .ipynb: WARNING: multiple files found for the document
2. There are 5 "targets not found"
   a. vqe-pyquil-demo.ipynb:230002: WARNING: 'myst' reference target not found: ../guide/zne-3-options.md#extrapolation-methods-factory-objects
   b. glossary.ipynb:10018: WARNING: 'myst' reference target not found: ./zne-3-options.md#unitary-folding
   c. glossary.ipynb:10018: WARNING: 'myst' reference target not found: ./zne-3-options.md#folding-gates-by-fidelity
   d. rem-3-options.ipynb:10005: WARNING: 'myst' reference target not found: rem-5-theory.md#what-is-a-confusion-matrix
   e. zne-4-low-level.ipynb:220002: WARNING: 'myst' reference target not found: zne-3-options.md#extrapolation-methods-factory-objects
3. I did not update make.bat, which needs to be updated so this will run on Windows.

What I don't like is that it does not give the option to download either .md or .ipynb, and it also can't launch MyBinder, because MyBinder has the .md files and the docs have the .ipynb files.

I am sure there has to be a way to get both files into the download button like MyST-NB does, but I am at a loss as to what that is.

All that being said, I'm going to enjoy the next couple of days of a three-day weekend. Feel free to take my branch and work on the remaining issues if it helps

[devilkiller-ag]

Thanks @andre-a-alves I will look into your branch and understand the issue.

[nathanshammah]

Excellent writeup @andre-a-alves and hopefully this is helpful for a solution.

[aryanguptaaa]

Hi @nathanshammah I would also like to work on this issue. @devilkiller-ag can we team up ?

[devilkiller-ag]

@aryanguptaaa Yeah, that would be great 👍🏻

[devilkiller-ag]

@nathanshammah Can we team up?

[andreamari]

Yes, you can! There was a similar question on Discord and so I copy and paste the answer:

If you do work on a team, ensure you make it clear when opening a PR. You can do this by either ensuring all authors have commits on the branch, or adding a description to the PR with all the authors github handles.

Moreover, of course, you would get a single bounty which must be shared with the team.

[devilkiller-ag]

Thanks you @andreamari , I will add @aryanguptaaa as a collaborator in the forked repo I am working on and mention it in the draft PR I have made. I will also contact @aryanguptaaa to share the bounty when we get that.

[aryanguptaaa]

@devilkiller-ag let's talk and coordinate on discord ?

[aryanguptaaa]

Thank you @andreamari

[devilkiller-ag]

@devilkiller-ag let's talk and coordinate on discord ?

Here is my Discord ID: AshmitJaiSaritaGupta#5291