Jupyter Tutorials (isl-org#1640)

* example jupyter tutorial for point cloud

* strip notebook output and add option point_show_normal to draw_geometries

* added nbsphinx to docs; replaced most of Basic tutorials with ipynb

* replaced many advanced tutorials with ipynb; added script to stip
jupyter output

* added filter for ipynb files

* add script for enabling filter

* added note about cleaning ipynb before commit to documentation

* nbconvert is required

* improved open3d logger such that it can redirect output to Python s stdout; also added context manager for verbosity level

* add nbsphinx to travis

* fix compilation of log example

* draw_geometries in tutorial ipynb with custom views

* mesh_show_wireframe and mesh_show_back_face parameters in draw_geometries; moved some examples to ipynb files

* apply-style

* limit text output in html files generated from notebooks

* added more ipynb tutorials

* addressing reviewing comments

* build juypter docs before running sphinx

* enable xvfb for travis

* try xvfb -a

* install pandoc in travis

* increase jupyter timeout

* no timeout for notebooks

* travis_wait for make_docs.py

* reduce the number of iterations on travis

* travis build docs without openmp

Co-authored-by: Benjamin Ummenhofer <benjaminum@gmail.com>
Co-authored-by: Yixing Lao <yixing.lao@gmail.com>

.gitattributes

@@ -0,0 +1 @@
+*.ipynb     filter=jupyter_clear_output

.gitfilters

@@ -0,0 +1,4 @@
+[filter "jupyter_clear_output"]
+    clean = "jupyter nbconvert --stdin --stdout --log-level=ERROR --to notebook --ClearOutputPreprocessor.enabled=True"
+    smudge = cat
+    required = true

.gitignore

@@ -39,6 +39,7 @@ compile_commands.json
 
 # Example resources
 examples/Python/ReconstructionSystem/dataset/
+examples/TestData/fountain_small
 examples/TestData/Armadillo.ply
 examples/TestData/Bunny.ply
 examples/TestData/eagle.ply
@@ -61,3 +62,12 @@ examples/Python/Basic/copy_of_lena_color.jpg
 examples/Python/Basic/copy_of_crate.mtl
 examples/Python/Basic/copy_of_crate.png
 examples/Python/Basic/copy_of_crate.obj
+
+# doc
+docs/TestData
+docs/tutorial/**/*.ipynb
+docs/tutorial/**/*.py
+docs/tutorial/**/*.jpg
+docs/tutorial/**/*.ply
+docs/tutorial/**/*.pcd
+docs/tutorial/**/*.json

.travis.yml

@@ -35,7 +35,7 @@ matrix:
         - pip install -U yapf==0.28.0
         - make check-style
 
-    # Build docs only
+    # Build headless and docs 
     - os: linux
       dist: bionic
       sudo: true
@@ -48,20 +48,23 @@ matrix:
           packages:
             - cmake
             - doxygen
+            - texlive
+            - texlive-latex-extra
+            - pandoc
             - libosmesa6-dev
       before_install: ./util/scripts/setup-linux.sh
       script:
         - mkdir build
         - cd build
         - which python
         - python -V
-        - pip install -U -q sphinx sphinx-rtd-theme
-        - cmake -DPYTHON_EXECUTABLE=`which python` -DENABLE_HEADLESS_RENDERING=ON -DBUILD_GLEW=ON -DBUILD_GLFW=ON ..
+        - pip install -U -q sphinx sphinx-rtd-theme nbsphinx Pillow
+        - cmake -DPYTHON_EXECUTABLE=`which python` -DENABLE_HEADLESS_RENDERING=ON -DBUILD_GLEW=ON -DBUILD_GLFW=ON -DWITH_OPENMP=OFF ..
         - make install-pip-package -j$NPROC
         - make -j$NPROC
         - bin/GLInfo
         - python -c "from open3d import *; import open3d; print(open3d)"
-        - ../util/scripts/make-documentation.sh
+        - travis_wait 40 ../util/scripts/make-documentation.sh
 
 
     # SHARED=ON, BUILD_DEPENDENCY_FROM_SOURCE=OFF

docs/_static/theme_overrides.css

@@ -12,3 +12,9 @@
        overflow: visible !important;
     }
  }
+
+/* Limit visible text output to about 30 lines and show scrollbar */
+.nboutput .output_area div > pre {
+        overflow-y: scroll !important;
+        max-height: 30em;
+}

docs/builddocs.rst

@@ -29,6 +29,9 @@ Clone and build Open3D from source. Visit :ref:`compilation` for details.
     conda install sphinx sphinx-autobuild
 
 
+You further need to install `nbsphinx`. See `here <http://nbsphinx.readthedocs.io>`_ for installation instructions.
+
+
 3. Install Doxygen
 ``````````````````
 

docs/conf.py

@@ -24,10 +24,12 @@
 import os
 import re
 import subprocess
+from pathlib import Path
+import shutil
 
 
 def get_git_short_hash():
-    rc = subprocess.check_output(['git', 'rev-parse', '--short', 'HEAD'])
+    rc = subprocess.check_output(["git", "rev-parse", "--short", "HEAD"])
     rc = rc.decode("utf-8").strip()
     return rc
 
@@ -39,7 +41,8 @@ def get_git_short_hash():
 sys.path.insert(
     0,
     os.path.join(current_file_dir, "..", "build", "lib", "python_package",
-                 "open3d"))
+                 "open3d"),
+)
 
 # -- General configuration ------------------------------------------------
 
@@ -51,26 +54,33 @@ def get_git_short_hash():
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    'sphinx.ext.mathjax', 'sphinx.ext.autodoc', 'sphinx.ext.autosummary',
-    'sphinx.ext.napoleon'
+    "sphinx.ext.mathjax",
+    "sphinx.ext.autodoc",
+    "sphinx.ext.autosummary",
+    "sphinx.ext.napoleon",
+    "nbsphinx",
 ]
 
+# Allow for more time for notebook cell evaluation
+nbsphinx_timeout = 6000
+# nbsphinx_allow_errors = True
+
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
 # source_suffix = ['.rst', '.md']
-source_suffix = '.rst'
+source_suffix = ".rst"
 
 # The master toctree document.
-master_doc = 'index'
+master_doc = "index"
 
 # General information about the project.
-project = u'Open3D'
-copyright = u'2018 - 2019, www.open3d.org'
-author = u'www.open3d.org'
+project = u"Open3D"
+copyright = u"2018 - 2020, www.open3d.org"
+author = u"www.open3d.org"
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the
@@ -93,10 +103,10 @@ def get_git_short_hash():
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This patterns also effect to html_static_path and html_extra_path
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store']
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "**.ipynb_checkpoints"]
 
 # The name of the Pygments (syntax highlighting) style to use.
-pygments_style = 'sphinx'
+pygments_style = "sphinx"
 
 # If true, `todo` and `todoList` produce output, else they produce nothing.
 todo_include_todos = False
@@ -127,13 +137,13 @@ def get_git_short_hash():
 
 # '_static' contains the theme overwrite
 static_path = os.path.join(theme_path, "sphinx_rtd_theme", "static")
-html_static_path = [static_path, '_static']
+html_static_path = [static_path, "_static"]
 
 # Force table wrap: https://rackerlabs.github.io/docs-rackspace/tools/rtd-tables.html
 html_context = {
-    'css_files': [
-        '_static/theme_overrides.css',  # override wide tables in RTD theme
-    ],
+    "css_files": [
+        "_static/theme_overrides.css"  # override wide tables in RTD theme
+    ]
 }
 
 # added by Jaesik to hide "View page source"
@@ -142,23 +152,20 @@ def get_git_short_hash():
 # -- Options for HTMLHelp output ------------------------------------------
 
 # Output file base name for HTML help builder.
-htmlhelp_basename = 'Open3Ddoc'
+htmlhelp_basename = "Open3Ddoc"
 
 # -- Options for LaTeX output ---------------------------------------------
 
 latex_elements = {
     # The paper size ('letterpaper' or 'a4paper').
     #
     # 'papersize': 'letterpaper',
-
     # The font size ('10pt', '11pt' or '12pt').
     #
     # 'pointsize': '10pt',
-
     # Additional stuff for the LaTeX preamble.
     #
     # 'preamble': '',
-
     # Latex figure (float) alignment
     #
     # 'figure_align': 'htbp',
@@ -167,37 +174,45 @@ def get_git_short_hash():
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
-latex_documents = [
-    (master_doc, 'Open3D.tex', u'Open3D Documentation', u'Qianyi Zhou',
-     'manual'),
-]
+latex_documents = [(
+    master_doc,
+    "Open3D.tex",
+    u"Open3D Documentation",
+    u"Qianyi Zhou",
+    "manual",
+)]
 
 # -- Options for manual page output ---------------------------------------
 
 # One entry per manual page. List of tuples
 # (source start file, name, description, authors, manual section).
-man_pages = [(master_doc, 'open3d', u'Open3D Documentation', [author], 1)]
+man_pages = [(master_doc, "open3d", u"Open3D Documentation", [author], 1)]
 
 # -- Options for Texinfo output -------------------------------------------
 
 # Grouping the document tree into Texinfo files. List of tuples
 # (source start file, target name, title, author,
 #  dir menu entry, description, category)
-texinfo_documents = [
-    (master_doc, 'Open3D', u'Open3D Documentation', author, 'Open3D',
-     'One line description of project.', 'Miscellaneous'),
-]
+texinfo_documents = [(
+    master_doc,
+    "Open3D",
+    u"Open3D Documentation",
+    author,
+    "Open3D",
+    "One line description of project.",
+    "Miscellaneous",
+)]
 
 # Version 0: Added by Jaesik to list Python members using the source order
 # Version 1: Changed to 'groupwise': __init__ first, then methods, then
 #            properties. Within each, sorted alphabetically.
-autodoc_member_order = 'groupwise'
+autodoc_member_order = "groupwise"
 
 
 def is_enum_class(func, func_name):
 
     def import_from_str(class_name):
-        components = class_name.split('.')
+        components = class_name.split(".")
         mod = __import__(components[0])
         for comp in components[1:]:
             mod = getattr(mod, comp)

docs/contribute/contribution_recipes.rst

@@ -268,6 +268,15 @@ Case 4: When adding a Python tutorial
 
 * Update the `index.rst` file to include your new tutorial
 
+.. note:: 
+   When you commit a ipynb notebook file make sure to remove the output cells
+   to keep the commit sizes small.
+   You can use the script ``examples/Python/jupyter_strip_output.sh`` for
+   stripping the output cells of all tutorials.
+   Alternatively, you can enable a pre-commit filter by running the script 
+   ``utils/scripts/git_enable_ipynb_filter.sh`` once. In any case you need to 
+   have ``jupyter`` in your ``PATH`` and ``nbconvert`` installed.
+
 Dos
 -----------------------
 

docs/getting_started.rst

@@ -74,29 +74,7 @@ Running Open3D tutorials
 ========================
 
 A complete set of Python tutorials and testing data will also be copied to
-demonstrate the usage of Open3D Python interface.
-For example, tutorial ``Basic/rgbd_redwood.py`` can be run with:
-
-.. code-block:: bash
-
-    cd lib/Tutorial/Basic
-    python rgbd_redwood.py
-
-It reads a pair of RGB-D images and visualizes them.
-
-.. image:: _static/Basic/rgbd_images/redwood_rgbd.png
-    :width: 400px
-
-It then transforms the RGB-D image into a point cloud, then renders the point
-cloud using Open3D visualizer.
-
-.. image:: _static/Basic/rgbd_images/redwood_pcd.png
-    :width: 400px
-
-The Python code is quite straightforward, and the detailed explanation can be
-found in :ref:`rgbd_redwood`.
-You may further experiment with different tutorials or check out the complete
-tutorial list in the :ref:`tutorial_index` page.
+demonstrate the usage of Open3D Python interface. See ``examples/Python`` for all Python examples.
 
 .. note:: Open3D's Python tutorial utilizes some external packages: ``numpy``,
     ``matplotlib``, ``opencv-python``. OpenCV is only used for reconstruction