Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sagemath_doc_html: Use jupyter-sphinx for 3D graphics #33507

Closed
mkoeppe opened this issue Mar 15, 2022 · 105 comments
Closed

sagemath_doc_html: Use jupyter-sphinx for 3D graphics #33507

mkoeppe opened this issue Mar 15, 2022 · 105 comments

Comments

@mkoeppe
Copy link
Member

mkoeppe commented Mar 15, 2022

We use jupyter-sphinx

https://jupyter-sphinx.readthedocs.io/en/latest/

to facilitate live sage documentation based on jupyter and Thebe.

For documentation authors, new directives are available:

.. ONLY:: pdf 
.. JUPYTER-EXECUTE::  

in addition to

.. ONLY:: html

See the branch for examples.

Related tickets: #31027

Depends on #33529

CC: @kwankyu @jhpalmieri @nthiery @egourgoulhon @dimpase

Component: documentation

Author: Kwankyu Lee, Matthias Koeppe

Branch: 72c8378

Reviewer: Matthias Koeppe, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/33507

@mkoeppe mkoeppe added this to the sage-9.7 milestone Mar 15, 2022
@mkoeppe

This comment has been minimized.

@kwankyu
Copy link
Collaborator

kwankyu commented Mar 16, 2022

comment:2

It uses nbsphinx: https://nbsphinx.readthedocs.io/en/0.8.8/

nbsphinx allows including jupyter notebooks in toctrees. It seems that we cannot include a single cell of a jupyter notebook as a part of our documentation, which I guess is what we want.

@kwankyu
Copy link
Collaborator

kwankyu commented Mar 16, 2022

Branch: public/33507

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 16, 2022

Commit: 1b7e46b

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 16, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

1b7e46bExperiments with jupyter-spinx

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 16, 2022

Changed commit from 1b7e46b to 58f1338

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 16, 2022

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

58f1338Experiments with jupyter-sphinx

@kwankyu
Copy link
Collaborator

kwankyu commented Mar 16, 2022

comment:6

This tool

https://jupyter-sphinx.readthedocs.io/en/latest/

works well.

I uploaded a branch to experiment with the tool. First sage -pip install jupyter-sphinx.

@mkoeppe
Copy link
Member Author

mkoeppe commented Mar 16, 2022

comment:7

This package would be easy to add, we already have all dependencies - https://github.com/jupyter/jupyter-sphinx/blob/master/setup.py#L30

@mkoeppe mkoeppe changed the title sagemath_doc_html: Use pythreejs for 3D graphics sagemath_doc_html: Use jupyter-sphinx for 3D graphics Mar 16, 2022
@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 16, 2022

Changed commit from 58f1338 to 8f8cd2b

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 16, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

8f8cd2bActivate Thebelab

@kwankyu
Copy link
Collaborator

kwankyu commented Mar 16, 2022

comment:10

We need a sagemath binder repo for Thebe to work.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 17, 2022

Changed commit from 8f8cd2b to 84cec60

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 17, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

84cec60For pdf documentation

@kwankyu
Copy link
Collaborator

kwankyu commented Mar 17, 2022

comment:12

The last commit provides an example to provide live images to htmls and static images to pdfs.

@kwankyu
Copy link
Collaborator

kwankyu commented Mar 17, 2022

comment:13

One annoying thing is that .. only:: latex works but .. only:: pdf doesn't.

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 17, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

5f01f21Forgotten images

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 17, 2022

Changed commit from 84cec60 to 5f01f21

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 17, 2022

Branch pushed to git repo; I updated commit sha1. New commits:

73c8c37build/pkgs/jupyter_sphinx: New
bbc276ebuild/pkgs/sagemath_doc_html/dependencies: Add jupyter_sphinx
6133f9fconfigure.ac (--disable-doc): Disable jupyter_sphinx too

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 17, 2022

Changed commit from 5f01f21 to 6133f9f

@mkoeppe
Copy link
Member Author

mkoeppe commented Mar 17, 2022

comment:16

Here's the package

@nthiery
Copy link
Contributor

nthiery commented Mar 17, 2022

comment:17

Replying to @kwankyu:

We need a sagemath binder repo for Thebe to work.

I fixed 'Sage binder env' about a month ago:

https://github.com/sagemath/sage-binder-env

It now uses conda to install Sage which is more maintained than
Sage's docker image.

@kwankyu
Copy link
Collaborator

kwankyu commented Mar 17, 2022

comment:18

Replying to @mkoeppe:

Here's the package

Thank you!

@kwankyu
Copy link
Collaborator

kwankyu commented Mar 17, 2022

comment:19

Replying to @nthiery:

Replying to @kwankyu:

We need a sagemath binder repo for Thebe to work.

I fixed 'Sage binder env' about a month ago:

https://github.com/sagemath/sage-binder-env

It now uses conda to install Sage which is more maintained than
Sage's docker image.

I have no experience with Thebe. Jupyter-sphinx supports Thebe. But running an interact example cell through jupyter-sphinx only shows Waiting for kernel.... Is this supposed to take long? Would the interact work well once the kernel is up?

@sagetrac-git
Copy link
Mannequin

sagetrac-git mannequin commented Mar 18, 2022

Changed commit from 6133f9f to 7641a72

@mkoeppe
Copy link
Member Author

mkoeppe commented Mar 22, 2022

comment:74

Sorry, fixed now

@dimpase
Copy link
Member

dimpase commented Mar 22, 2022

comment:75

I am still getting this docbuild error on macOS, even after git clean -fdx.

[sagemath_doc_html-none] [repl     ] executing interact
[sagemath_doc_html-none] [repl     ] /Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jsonschema/validators.py:188: DeprecationWarning: Passing a schema to Validator.iter_errors is deprecated and will be removed in a future release. Call validator.evolve(schema=new_schema).iter_errors(...) instead.
[sagemath_doc_html-none] [repl     ]   warnings.warn(
[sagemath_doc_html-none] [repl     ] Extension error:
[sagemath_doc_html-none] [repl     ] Notebook execution failed (exception: [Errno 2] No such file or directory: '/Users/dima/sage/local/bin/sage')
...
[sagemath_doc_html-none] [plot3d   ] executing plot3d
[sagemath_doc_html-none] [plot3d   ] /Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jsonschema/validators.py:188: DeprecationWarning: Passing a schema to Validator.iter_errors is deprecated and will be removed in a future release. Call validator.evolve(schema=new_schema).iter_errors(...) instead.
[sagemath_doc_html-none] [plot3d   ]   warnings.warn(
[sagemath_doc_html-none] [plot3d   ] Extension error:
[sagemath_doc_html-none] [plot3d   ] Notebook execution failed (exception: [Errno 2] No such file or directory: '/Users/dima/sage/local/bin/sage')
...
sagemath_doc_html-none] [reference] ... done (500 todos, 2546 index, 1665 citations, 2139 modules)
[sagemath_doc_html-none] [reference] WARNING: Unable to fetch /Users/dima/sage/local/share/doc/sage/doctrees/en/reference/plot3d/environment.pickle
[sagemath_doc_html-none] [reference] WARNING: Unable to fetch /Users/dima/sage/local/share/doc/sage/doctrees/en/reference/repl/environment.pickle
[sagemath_doc_html-none] [reference] preparing documents... skipping loading of indexes... done
[sagemath_doc_html-none] [reference] /Users/dima/sage/src/doc/en/reference/index.rst:17: WARNING: unknown document: repl/index
[sagemath_doc_html-none] [reference] /Users/dima/sage/src/doc/en/reference/index.rst:24: WARNING: unknown document: plot3d/index
[sagemath_doc_html-none] [reference] The inventory files are in local/share/doc/sage/inventory/en/reference.
[sagemath_doc_html-none] Error building the documentation.
[sagemath_doc_html-none] Traceback (most recent call last):
[sagemath_doc_html-none]   File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
[sagemath_doc_html-none]     return _run_code(code, main_globals, None,
[sagemath_doc_html-none]   File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
[sagemath_doc_html-none]     exec(code, run_globals)
[sagemath_doc_html-none]   File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage_docbuild/__main__.py", line 2, in <module>
[sagemath_doc_html-none]     main()
[sagemath_doc_html-none]   File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 1751, in main
[sagemath_doc_html-none]     builder()
[sagemath_doc_html-none]   File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage_docbuild/__init__.py", line 141, in f
[sagemath_doc_html-none]     runsphinx()
[sagemath_doc_html-none]   File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage_docbuild/sphinxbuild.py", line 326, in runsphinx
[sagemath_doc_html-none]     sys.stderr.raise_errors()
[sagemath_doc_html-none]   File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/sage_docbuild/sphinxbuild.py", line 262, in raise_errors
[sagemath_doc_html-none]     raise OSError(self._error)
[sagemath_doc_html-none] OSError: WARNING: Unable to fetch /Users/dima/sage/local/share/doc/sage/doctrees/en/reference/plot3d/environment.pickle 
[sagemath_doc_html-none] 
[sagemath_doc_html-none]     Note: incremental documentation builds sometimes cause spurious
[sagemath_doc_html-none]     error messages. To be certain that these are real errors, run
[sagemath_doc_html-none]     "make doc-clean doc-uninstall" first and try again.
[sagemath_doc_html-none] make[6]: *** [doc-inventory--reference_top] Error 1
[sagemath_doc_html-none] make[5]: *** [doc-inventory-reference] Error 2
make[4]: *** [sagemath_doc_html-SAGE_DOCS-no-deps] Error 2

something in these two modules wants to call '/Users/dima/sage/local/bin/sage' -
which is absent, naturally, and probably this leads to absent environment.pickle in plot3d.

I presume the message "Notebook execution failed" comes from jupyter-sphinx:
https://github.com/jupyter/jupyter-sphinx/blob/master/jupyter_sphinx/execute.py

@dimpase
Copy link
Member

dimpase commented Mar 22, 2022

Changed reviewer from Matthias Koeppe, ... to Matthias Koeppe, Dima Pasechnik

@dimpase
Copy link
Member

dimpase commented Mar 22, 2022

comment:76

after

ln -s /Users/dima/sage/sage /Users/dima/sage/local/bin/sage

the docbuild completes, but it's certainly a problem to be fixed that needs this workaround.

@kwankyu
Copy link
Collaborator

kwankyu commented Mar 23, 2022

comment:77

Replying to @dimpase:

I am still getting this docbuild error on macOS, even after git clean -fdx.

[sagemath_doc_html-none] [repl     ] executing interact
[sagemath_doc_html-none] [repl     ] /Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jsonschema/validators.py:188: DeprecationWarning: Passing a schema to Validator.iter_errors is deprecated and will be removed in a future release. Call validator.evolve(schema=new_schema).iter_errors(...) instead.
[sagemath_doc_html-none] [repl     ]   warnings.warn(
[sagemath_doc_html-none] [repl     ] Extension error:
[sagemath_doc_html-none] [repl     ] Notebook execution failed (exception: [Errno 2] No such file or directory: '/Users/dima/sage/local/bin/sage')

...

I presume the message "Notebook execution failed" comes from jupyter-sphinx:
https://github.com/jupyter/jupyter-sphinx/blob/master/jupyter_sphinx/execute.py

I am on macOS too. Do you have no problem in running sagemath kernel in jupyter from your sage installation?

@dimpase
Copy link
Member

dimpase commented Mar 23, 2022

comment:78

If I remove the symlink as in comment:76 I get a jupyter kernel error

Traceback (most recent call last):
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/tornado/web.py", line 1704, in _execute
    result = await result
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
    future.result()
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/tornado/gen.py", line 769, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/notebook/services/sessions/handlers.py", line 74, in post
    model = yield maybe_future(
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/tornado/gen.py", line 769, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/notebook/services/sessions/sessionmanager.py", line 98, in create_session
    kernel_id = yield self.start_kernel_for_session(session_id, path, name, type, kernel_name)
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/tornado/gen.py", line 769, in run
    yielded = self.gen.throw(*exc_info)  # type: ignore
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/notebook/services/sessions/sessionmanager.py", line 110, in start_kernel_for_session
    kernel_id = yield maybe_future(
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/notebook/services/kernels/kernelmanager.py", line 176, in start_kernel
    kernel_id = await maybe_future(self.pinned_superclass.start_kernel(self, **kwargs))
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/utils.py", line 26, in wrapped
    raise e
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/utils.py", line 23, in wrapped
    return loop.run_until_complete(future)
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/nest_asyncio.py", line 81, in run_until_complete
    return f.result()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/multikernelmanager.py", line 187, in _async_start_kernel
    starter = ensure_async(km.start_kernel(**kwargs))
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/utils.py", line 26, in wrapped
    raise e
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/utils.py", line 23, in wrapped
    return loop.run_until_complete(future)
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/nest_asyncio.py", line 81, in run_until_complete
    return f.result()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/manager.py", line 362, in _async_start_kernel
    raise e
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/manager.py", line 351, in _async_start_kernel
    await ensure_async(self._launch_kernel(kernel_cmd, **kw))
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/utils.py", line 26, in wrapped
    raise e
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/utils.py", line 23, in wrapped
    return loop.run_until_complete(future)
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/nest_asyncio.py", line 81, in run_until_complete
    return f.result()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/manager.py", line 269, in _async_launch_kernel
    connection_info = await self.provisioner.launch_kernel(kernel_cmd, **kw)
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/provisioning/local_provisioner.py", line 179, in launch_kernel
    self.process = launch_kernel(cmd, **scrubbed_kwargs)
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/launcher.py", line 169, in launch_kernel
    raise ex
  File "/Users/dima/sage/local/var/lib/sage/venv-python3.9/lib/python3.9/site-packages/jupyter_client/launcher.py", line 157, in launch_kernel
    proc = Popen(cmd, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/Users/dima/sage/local/bin/sage'

With the link it works. It turns out I have outdated things in ~/Library/Jupyter, in particular sagemath/kernel.json with the obsolete path to sage (version 9.4.beta3).

Sorry for noise, it's OK.

@mkoeppe
Copy link
Member Author

mkoeppe commented Mar 23, 2022

comment:79

Probably the docbuild should try to isolate from such user settings.

@kwankyu
Copy link
Collaborator

kwankyu commented Mar 23, 2022

comment:80

Replying to @mkoeppe:

Probably the docbuild should try to isolate from such user settings.

I don't understand the precise mechanism. But it seems that jupyter-sphinx just runs a new jupyter notebook with sagemath kernel. Then there is nothing to do with the docbuild.

@kwankyu
Copy link
Collaborator

kwankyu commented Mar 23, 2022

comment:81

Replying to @dimpase:

Thank you for review.

@mkoeppe
Copy link
Member Author

mkoeppe commented Mar 24, 2022

comment:82

Replying to @kwankyu:

Replying to @mkoeppe:

Probably the docbuild should try to isolate from such user settings.

I don't understand the precise mechanism. But it seems that jupyter-sphinx just runs a new jupyter notebook with sagemath kernel. Then there is nothing to do with the docbuild.

But its happening during the docbuild according to the output that Dima shared

@kwankyu
Copy link
Collaborator

kwankyu commented Mar 24, 2022

comment:83

Replying to @mkoeppe:

But its happening during the docbuild according to the output that Dima shared

The source of the problem may be in our jupyter kernel configuration in repl.ipython_kernel.kernel.

@dimpase
Copy link
Member

dimpase commented Mar 24, 2022

comment:84

Replying to @mkoeppe:

Replying to @kwankyu:

Replying to @mkoeppe:

Probably the docbuild should try to isolate from such user settings.

I don't understand the precise mechanism. But it seems that jupyter-sphinx just runs a new jupyter notebook with sagemath kernel. Then there is nothing to do with the docbuild.

But its happening during the docbuild according to the output that Dima shared

I could reproduce this issue just by running ./sage -n. So this is indeed a "pure" notebook problem.

@vbraun
Copy link
Member

vbraun commented Apr 2, 2022

Changed branch from public/33507 to 72c8378

@mkoeppe
Copy link
Member Author

mkoeppe commented Apr 5, 2022

Changed commit from 72c8378 to none

@mkoeppe
Copy link
Member Author

mkoeppe commented Apr 5, 2022

comment:86

The first bug report on this may have arrived: https://groups.google.com/g/sage-release/c/-0luW4DPEPs/m/1bST2lbcCAAJ

@dimpase
Copy link
Member

dimpase commented Apr 5, 2022

comment:87

Replying to @mkoeppe:

The first bug report on this may have arrived: https://groups.google.com/g/sage-release/c/-0luW4DPEPs/m/1bST2lbcCAAJ

You mean

[sagemath_doc_html-none] [plot3d   ] Notebook execution failed (exception: [Errno 2] No such file or directory: '/home/furutaka/work/sage/sage-9.0-git-bld/local/bin/sage')

I suppose.

This looks identical to comment:75 - a stale Jupyter kernel somewhere.

@mkoeppe
Copy link
Member Author

mkoeppe commented Apr 6, 2022

comment:88

Yes. I have opened #33650 for this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants