Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generating Python API docs #2038

Open
wants to merge 29 commits into
base: main
Choose a base branch
from

Conversation

StefanHabel
Copy link
Contributor

@StefanHabel StefanHabel commented Sep 30, 2024

This PR adds support for generating Python API documentation in HTML format using Sphinx from the MaterialX Python modules that are built in the lib/ directory.

A new CMake build option named MATERIALX_BUILD_PYTHON_DOCS allows developers to turn generating Python API documentation on.

When MATERIALX_BUILD_PYTHON_DOCS is set to ON, MATERIALX_BUILD_PYTHON is set to ON as well, ensuring we have Python modules for which to build the Python API docs.

The core functionality of generating Python API documentation lives in a new directory named documents/PythonAPI/. It is controlled with a new CMakeLists.txt file in that directory, which defines a new target named MaterialXDocsPython, similar to the existing target MaterialXDocs that generates API documentation for the MaterialX C++ API.

Revised docstrings for modules were recently added in the following separate PR:

Revised docstrings for classes are added in the following separate PRs:

Revised docstrings for methods and functions are to be added in subsequent PRs separately.

Documentation in markdown format from the existing DeveloperGuide is integrated into the new Python API documentation by way of symlinking the four main .md files into the documents/PythonAPI/sources/ directory from which Sphinx generates the resulting HTML documentation.

To build the docs from scratch on macOS, I've used the following build script, naming it build.sh in the MaterialX checkout directory:

#!/usr/bin/env tcsh

echo build.sh: Updating Git submodules...
git submodule update --init --recursive

# Create and activate a virtual environment for installing Python dependencies
python3 -m venv /tmp/venv
source /tmp/venv/bin/activate.csh

echo build.sh: Installing dependencies...
python3 -m ensurepip
python3 -m pip install myst_parser  # https://pypi.org/project/myst-parser/

echo build.sh: Making build directory and changing into it...
mkdir build
cd build

echo build.sh: Configuring...
cmake .. \
    --fresh \
    -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk \
    -DMATERIALX_BUILD_PYTHON=ON \
    -DMATERIALX_BUILD_VIEWER=ON \
    -DMATERIALX_BUILD_GRAPH_EDITOR=ON \
    -DMATERIALX_BUILD_DOCS=ON \
    -DMATERIALX_BUILD_PYTHON_DOCS=ON \
    -DMATERIALX_BUILD_TESTS=ON \
&& \
echo build.sh: Building... \
&& \
cmake --build . -j 8 \
&& \
echo build.sh: Building target MaterialXDocs... \
&& \
cmake --build . --target MaterialXDocs \
&& \
echo build.sh: Building target MaterialXDocsPython... \
&& \
cmake --build . --target MaterialXDocsPython \
&& \
afplay /System/Library/Sounds/Blow.aiff

# Deactivate the virtual environment
deactivate

The build output currently ends with the following messages:

The parsed MaterialX Python API consists of:
    * 11 modules
    * 48 functions
    * 141 classes
    * 67 attributes
    * 1178 methods
    * 6 exception types

WARNING: 48 functions look like they do not have docstrings yet.
WARNING: 1020 methods look like they do not have docstrings yet.
WARNING: 32 functions look like their parameters have not all been named using `py::arg()`.
WARNING: 500 methods look like their parameters have not all been named using `py::arg()`.
build succeeded, 4 warnings.

The HTML pages are in ..
[100%] Built target MaterialXDocsPython

Screenshot showing the new Python API documentation main page:

Screenshot 2024-09-29 at 17 56 22

Screenshot showing the modified Viewer.md contents from #2037 in the context of the Python API documentation:

Screenshot 2024-09-30 at 16 04 14

Screenshot showing the (currently) incomplete PyMaterialXCore docs (to be amended in subsequent PRs):

Screenshot 2024-10-06 at 13 11 00

Split from #1567.

Update #342.

StefanHabel added a commit to StefanHabel/MaterialX that referenced this pull request Sep 30, 2024
This PR adds a docstring in markdown format to the `PyMaterialXCore` module.

The docstring is defined in a macro named `PyMaterialXCore_DOCSTRING` that
uses the new `PYMATERIALX_DOCSTRING` macro to allow us to place the first
and last lines in lines by themselves, rather than starting the docstring
contents immediately after the `R"docstring(` marker.

The `PyMaterialXCore_DOCSTRING` macro is placed in a separate header file
named `__doc__.md.h` which lives side-by-side with the `PyModule.cpp` file
in which it is included.

Split from AcademySoftwareFoundation#1567.

Depends on AcademySoftwareFoundation#2038.

Update AcademySoftwareFoundation#342.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
StefanHabel added a commit to StefanHabel/MaterialX that referenced this pull request Sep 30, 2024
This PR adds a docstring in markdown format to the `PyMaterialXFormat` module.

The docstring is defined in a macro named `PyMaterialXFormat_DOCSTRING`
that uses the new `PYMATERIALX_DOCSTRING` macro that is introduced in AcademySoftwareFoundation#2038
to allow us to place the first and last lines in lines by themselves,
rather than starting the docstring contents immediately after the
`R"docstring(` marker.

The `PyMaterialXFormat_DOCSTRING` macro is placed in a separate header file
named `__doc__.md.h` which lives side-by-side with the `PyModule.cpp` file
in which it is included.

Note that the docstrings for individual classes, methods, and functions are
to be added in separate PRs.

Note that this PR also renames `readFromXmlFileBase()` to `readFromXmlFile()`
allowing us to remove the alias from `python/MaterialX/main.py`.

Split from AcademySoftwareFoundation#1567.

Depends on AcademySoftwareFoundation#2038.

Update AcademySoftwareFoundation#342.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
StefanHabel added a commit to StefanHabel/MaterialX that referenced this pull request Sep 30, 2024
This PR adds docstrings in markdown format to the following modules:
- `PyMaterialXGenGlsl`
- `PyMaterialXGenMdl`
- `PyMaterialXGenMsl`
- `PyMaterialXGenOsl`
- `PyMaterialXGenShader`

The docstrings are defined in macros named `PyMaterialXGen*_DOCSTRING` that
use the new `PYMATERIALX_DOCSTRING` macro that is introduced in AcademySoftwareFoundation#2038 to
allow us to place the first and last lines in lines by themselves, rather
than starting the docstring contents immediately after the `R"docstring(`
marker.

The `PyMaterialXGen*_DOCSTRING` macros are placed in separate header files
named `__doc__.md.h` which live side-by-side with their corresponding
`PyModule.cpp` file in which it is included.

Note that the docstrings for individual classes, methods, and functions are
to be added in separate PRs.

Split from AcademySoftwareFoundation#1567.

Depends on AcademySoftwareFoundation#2038.

Update AcademySoftwareFoundation#342.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
StefanHabel added a commit to StefanHabel/MaterialX that referenced this pull request Sep 30, 2024
This PR adds docstrings in markdown format to the following modules:
- `PyMaterialXRender`
- `PyMaterialXRenderGlsl`
- `PyMaterialXRenderMsl`
- `PyMaterialXRenderOsl`

The docstrings are defined in macros named `PyMaterialXRender*_DOCSTRING`
that use the new `PYMATERIALX_DOCSTRING` macro that is introduced in AcademySoftwareFoundation#2038
to allow us to place the first and last lines in lines by themselves,
rather than starting the docstring contents immediately after the
`R"docstring(` marker.

The `PyMaterialXRender*_DOCSTRING` macros are placed in separate header
files named `__doc__.md.h` which live side-by-side with their corresponding
`PyModule.cpp` files in which they are included.

Note that the docstrings for individual classes, methods, and functions are
to be added in separate PRs.

Split from AcademySoftwareFoundation#1567.

Depends on AcademySoftwareFoundation#2038.

Update AcademySoftwareFoundation#342.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
@StefanHabel
Copy link
Contributor Author

Demo of generated MaterialX Python API Documentation now available here: https://stefanhabel.github.io

Source repo for the live demo available here: https://github.com/StefanHabel/StefanHabel.github.io

This PR adds support for generating Python API documentation in HTML format
using Sphinx from the MaterialX Python that are built in the `lib/`
directory.

A new CMake build option named `MATERIALX_BUILD_PYTHON_DOCS` allows
developers to turn generating Python API documentation on.

When `MATERIALX_BUILD_PYTHON_DOCS` is set to `ON`, `MATERIALX_BUILD_PYTHON`
is set to `ON` as well, ensuring we have Python modules for which to build
the Python API docs.

The core functionality of generating Python API documentation lives in a
new directory named `documents/PythonAPI/`. It is controlled with a new
`CMakeLists.txt` file in that directory, which defines a new target named
`MaterialXDocsPython`, similar to the existing target `MaterialXDocs` that
generates API documentation for the MaterialX C++ API.

To facilitate the curation and addition of docstrings in the implementation
files within `source/PyMaterialX/`, this PR adds a new helper macro named
`PYMATERIALX_DOCSTRING` that allows developers of Python modules to define
docstrings using the following pattern:
```cpp
PYMATERIALX_DOCSTRING(R"docstring(
...markdown text here...
)docstring");
```

Revised docstrings for modules and classes are to be added in subsequent PRs
separately.

Documentation in markdown format from the existing `DeveloperGuide` is
integrated into the new Python API documentation by way of symlinking the
four main `.md` files into the `documents/PythonAPI/sources/` directory
from which Sphinx generates the resulting HTML documentation.

Warnings that are issued when generating the documentation via Sphinx are
to be addressed in a separate PR for the markdown files:
AcademySoftwareFoundation#2037

To build the docs from scratch on macOS, I've used the following build
script, naming it `build.sh` in the `MaterialX` checkout directory:
```bash

echo build.sh: Updating Git submodules...
git submodule update --init --recursive

python3 -m venv /tmp/venv
source /tmp/venv/bin/activate.csh

echo build.sh: Installing dependencies...
python3 -m pip install myst_parser  # https://pypi.org/project/myst-parser/

echo build.sh: Making build directory and changing into it...
mkdir build
cd build

echo build.sh: Configuring...
cmake .. \
    --fresh \
    -DCMAKE_OSX_SYSROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX15.0.sdk \
    -DMATERIALX_BUILD_PYTHON=ON \
    -DMATERIALX_BUILD_VIEWER=ON \
    -DMATERIALX_BUILD_GRAPH_EDITOR=ON \
    -DMATERIALX_BUILD_DOCS=ON \
    -DMATERIALX_BUILD_PYTHON_DOCS=ON \
    -DMATERIALX_BUILD_TESTS=ON \
&& \
echo build.sh: Building... \
&& \
cmake --build . -j 8 \
&& \
echo build.sh: Building target MaterialXDocs... \
&& \
cmake --build . --target MaterialXDocs \
&& \
echo build.sh: Building target MaterialXDocsPython... \
&& \
cmake --build . --target MaterialXDocsPython \
&& \
afplay /System/Library/Sounds/Blow.aiff

deactivate
```
The build output currently ends with the following messages:
```python
The parsed MaterialX Python API consists of:
    * 11 modules
    * 48 functions
    * 139 classes
    * 1175 methods
    * 6 exception types

WARNING: 48 functions look like they do not have docstrings yet.
WARNING: 1019 methods look like they do not have docstrings yet.
WARNING: 32 functions look like their parameters have not all been named using `py::arg()`.
WARNING: 499 methods look like their parameters have not all been named using `py::arg()`.
build succeeded, 168 warnings.

The HTML pages are in ..
[100%] Built target MaterialXDocsPython
```

Split from AcademySoftwareFoundation#1567.

Update AcademySoftwareFoundation#342.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
StefanHabel added a commit to StefanHabel/MaterialX that referenced this pull request Oct 3, 2024
This PR adds a docstring in markdown format to the `PyMaterialXCore` module.

The docstring is defined in a macro named `PyMaterialXCore_DOCSTRING` that
uses the new `PYMATERIALX_DOCSTRING` macro to allow us to place the first
and last lines in lines by themselves, rather than starting the docstring
contents immediately after the `R"docstring(` marker.

The `PyMaterialXCore_DOCSTRING` macro is placed in a separate header file
named `__doc__.md.h` which lives side-by-side with the `PyModule.cpp` file
in which it is included.

Split from AcademySoftwareFoundation#1567.

Depends on AcademySoftwareFoundation#2038.

Update AcademySoftwareFoundation#342.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
We may not need the proposed `PYMATERIALX_DOCSTRING` macro.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
@StefanHabel
Copy link
Contributor Author

I've rebased the branch of this PR on top of main, and have added another commit to revert the changes in PyMaterialX.h: d76eb5a

This follows a discussion on #2039 in which the proposed new PYMATERIALX_DOCSTRING was first used.

The changes in this PR now stand as +1,174 −0.

@jstone-lucasfilm jstone-lucasfilm changed the title Generating Python API docs. Generating Python API docs Oct 3, 2024
jstone-lucasfilm and others added 5 commits October 3, 2024 08:46
Signed-off-by: Jonathan Stone <jstone@lucasfilm.com>
…menting class properties that don't store attribute name strings.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
… documentation pages.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
@StefanHabel
Copy link
Contributor Author

I've added a couple of commits to replace the previous Alphabetical Index on a module's documentation page, which did not provide links to module functions, with alphabetical listings of Classes, Exception Types, and Module Functions, each linking to either a separate documentation page or to a section on the same page:

Before After
Screenshot 2024-09-29 at 17 57 32 Screenshot 2024-10-06 at 13 11 00

The only warnings that are now being displayed by Sphinx at the end of building the Python API documentation are those that we add ourselves to highlight missing documentation:

WARNING: 48 functions look like they do not have docstrings yet.
WARNING: 1019 methods look like they do not have docstrings yet.
WARNING: 32 functions look like their parameters have not all been named using `py::arg()`.
WARNING: 499 methods look like their parameters have not all been named using `py::arg()`.
-build succeeded, 149 warnings.
+build succeeded, 4 warnings.

I've updated the generated HTML pages in the live demo here: https://stefanhabel.github.io/

Example of a class with attribute names and other properties:
https://stefanhabel.github.io/generated/PyMaterialXCore.Backdrop.html#PyMaterialXCore.Backdrop

The summary of the MaterialX Python API is now stated as follows:

The parsed MaterialX Python API consists of:
    * 11 modules
    * 48 functions
    * 139 classes
    * 59 attributes
    * 1175 methods
    * 6 exception types

We can add the missing documentation in subsequent separate PRs dedicated to individual modules and classes.

@jstone-lucasfilm
Copy link
Member

@StefanHabel I still need to test this locally using Sphinx, but overall this proposal is looking really good to me!

Docstrings of attributes are automatically included through the `autoattribute` declaration itself.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
documents/PythonAPI/custom.css Outdated Show resolved Hide resolved
CMakeLists.txt Outdated Show resolved Hide resolved
StefanHabel and others added 9 commits October 19, 2024 17:09
Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
…DOCS`.

Also using `DEPRECATION` warning for `MATERIALX_BUILD_IOS` option.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
Keeping the old name around, but making it issue a warning when building the MaterialXDocs target.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
Signed-off-by: Jonathan Stone <jstone@lucasfilm.com>
Signed-off-by: Jonathan Stone <jstone@lucasfilm.com>
Signed-off-by: Jonathan Stone <jstone@lucasfilm.com>
For developer clarity, I think it makes the most sense to remove the legacy MATERIALX_BUILD_DOCS option rather than deprecating it.

Signed-off-by: Jonathan Stone <jstone@lucasfilm.com>
Following the pattern of removing the MATERIALX_BUILD_DOCS option, I think it makes the most sense to remove the MaterialXDocs target rather than deprecating it.

Signed-off-by: Jonathan Stone <jstone@lucasfilm.com>
@jstone-lucasfilm
Copy link
Member

@StefanHabel Today I had a chance to try out this latest work with a local installation of Sphinx, but I encountered the following error when building the new MaterialXDocsPython project:

Could not import extension myst_parser (exception: No module named 'myst_parser')

Should we call out the need to install myst_parser in addition to sphinx, or will most developers assume both are installed together?

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
@StefanHabel
Copy link
Contributor Author

@StefanHabel Today I had a chance to try out this latest work with a local installation of Sphinx, but I encountered the following error when building the new MaterialXDocsPython project:

Could not import extension myst_parser (exception: No module named 'myst_parser')

Should we call out the need to install myst_parser in addition to sphinx, or will most developers assume both are installed together?

Thanks for giving this a go!

Yeah, probably good to highlight as a dependency in the same way as highlighting that Sphinx is required.

I've taken a stab at this in cb47402.

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
@StefanHabel
Copy link
Contributor Author

I've added another commit (f649cf1) to make the build fail early if myst-parser is not available, e.g.:

[...]
-- Found Sphinx: /tmp/venv/bin/sphinx-build
WARNING: Package(s) not found: myst-parser
CMake Error at documents/PythonAPI/CMakeLists.txt:11 (message):
  The `myst-parser` Python package does not appear to be available for
  `/tmp/venv/bin/python3.12`.

  Consider installing it via `pip install myst-parser`.

  PyPI: https://pypi.org/project/myst-parser/


-- Configuring incomplete, errors occurred!

@jstone-lucasfilm
Copy link
Member

@StefanHabel Now that we're wrapping up work for the upcoming v1.39.2 release, I wanted to check in to see what remaining work is needed to merge this proposal. Here is the Python API page that I get with the latest code in the PR, making no modifications to my display settings in Chrome:

PythonAPI_FrontPage

I see a few sections where the color scheme doesn't read clearly in light mode, and wanted to see whether this could potentially be resolved by leveraging the same color scheme as our C++ API documentation.

What are your thoughts?

@StefanHabel
Copy link
Contributor Author

@StefanHabel Now that we're wrapping up work for the upcoming v1.39.2 release, I wanted to check in to see what remaining work is needed to merge this proposal. Here is the Python API page that I get with the latest code in the PR, making no modifications to my display settings in Chrome:

PythonAPI_FrontPage

I see a few sections where the color scheme doesn't read clearly in light mode, and wanted to see whether this could potentially be resolved by leveraging the same color scheme as our C++ API documentation.

What are your thoughts?

Thanks for flagging this @jstone-lucasfilm!

I can reproduce the issue by switching to the Light appearance in macOS System Settings.

This likely needs a tweak or two to our custom.css in the branch of this PR, which is applied on top of the standard alabaster.css and the doxygen-awesome.css that the C++ API Documentation uses.

I'll have a look at applying tweaks tomorrow (currently in Germany for the holidays)

Signed-off-by: Stefan Habel <19556655+StefanHabel@users.noreply.github.com>
@StefanHabel
Copy link
Contributor Author

I've applied some tweaks in 9ffd9e4

Would you mind trying again @jstone-lucasfilm?

Screen recording of macOS switching from the Dark appearance to the Light appearance:

Screen.Recording.2024-12-17.at.17.52.38.mov

@jstone-lucasfilm
Copy link
Member

Thanks for the updates, @StefanHabel, and I'll give this a try!

@jstone-lucasfilm
Copy link
Member

The text colors look good in latest, thanks @StefanHabel!

Given the complexity of this PR, I'm thinking that we should aim to merge it near the beginning of development on MaterialX v1.39.3, followed by the dependent PRs for the various MaterialX modules.

@StefanHabel
Copy link
Contributor Author

OK cool, sounds good, thanks @jstone-lucasfilm!

I'll stand by to update the linked PRs when the time comes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants