Hang when processing example in docstring

[Andrew-Sheridan]

Problem Description

pdoc hangs when processing certain examples in docstrings. This took me ages to pin down :)

Steps to reproduce the behavior:

Add this content to some file, foo.py:

def foo():
    """Foo the foo.

    Examples:
        >>> class Foo(object):
        ...     pass
        >>> foo = Foo()
        >>> paths = foo.foo(dir_="src")
        >>> [str(p) for p in paths][:2]
        ['foo', 'bar']

    """

Run pdoc w/ google format

pdoc --docformat google --output-directory test-docs -- foo.py

• hangs :(

Counter example:
Add this content to some file, bar.py:

def bar():
    """Bar the bar.

    Examples:
        >>> class Bar(object):
        ...     pass
        >>> bar = Bar()
        >>> paths = bar.bar(dir_="src")

    """

pdoc --docformat google --output-directory test-docs -- foo.py

• works as expected :)

System Information

Paste the output of "pdoc --version" here.

❯ pdoc --version
pdoc: 11.0.0
Python: 3.7.9
Platform: Darwin-21.4.0-x86_64-i386-64bit

Thanks!

[Andrew-Sheridan]

Ok, so I tried this again in a fresh python3.9 virtual env

python3.9 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install pdoc

and it doesnt hang when processing foo.py. It takes 45 seconds, but it completes

Processing bar.py takes ~1 second.

The examples are really fiction, the real code and docstrings are ~1000 lines.
Processing that never seems to finish - developers have been cancelling the execution after 10 minutes or so. I'll start one before dinner and report back.

Edit: I killed the process after 1 hour 38 minutes.

If it helps the traceback is below

Click to expand!

❯ pdoc --docformat google --output-directory test-docs -- src/massmutual/de/utility_mixins
^CTraceback (most recent call last):
  File "/Users/mm10354/Projects/utility-mixins/vnev/bin/pdoc", line 8, in <module>
    sys.exit(cli())
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/__main__.py", line 192, in cli
    pdoc.pdoc(
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/__init__.py", line 456, in pdoc
    out = render.html_module(module, all_modules)
  File "/Users/mm10354/.Python_Environments/python3.9/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/render.py", line 96, in html_module
    return env.get_template("module.html.jinja2").render(
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/jinja2/environment.py", line 1289, in render
    return self.environment.concat(self.root_render_func(ctx))  # type: ignore
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/templates/default/module.html.jinja2", line 1018, in root
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/templates/default/frame.html.jinja2", line 43, in root
    </body>
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/templates/default/frame.html.jinja2", line 172, in block_body
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/templates/default/module.html.jinja2", line 1248, in block_content
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/templates/default/module.html.jinja2", line 1353, in block_module_contents
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/templates/default/module.html.jinja2", line 556, in macro
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/jinja2/runtime.py", line 298, in call
    return __obj(*args, **kwargs)
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/jinja2/runtime.py", line 763, in __call__
    return self._invoke(arguments, autoescape)
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/jinja2/runtime.py", line 777, in _invoke
    rv = self._func(*arguments)
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/templates/default/module.html.jinja2", line 588, in macro
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/render_helpers.py", line 74, in to_html
    return pdoc.markdown2.markdown(docstring, extras=markdown_extensions)  # type: ignore
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/markdown2/__init__.py", line 184, in markdown
    return Markdown(html4tags=html4tags, tab_width=tab_width,
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/markdown2/__init__.py", line 383, in convert
    text = self._run_block_gamut(text)
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/markdown2/__init__.py", line 1025, in _run_block_gamut
    text = self._do_block_quotes(text)
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/markdown2/__init__.py", line 2139, in _do_block_quotes
    return self._block_quote_re.sub(self._block_quote_sub, text)
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/markdown2/__init__.py", line 2122, in _block_quote_sub
    bq = self._run_block_gamut(bq)          # recurse
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/markdown2/__init__.py", line 1017, in _run_block_gamut
    text = self._prepare_pyshell_blocks(text)
  File "/Users/mm10354/Projects/utility-mixins/vnev/lib/python3.9/site-packages/pdoc/markdown2/__init__.py", line 1063, in _prepare_pyshell_blocks
    return _pyshell_block_re.sub(self._pyshell_block_sub, text)
KeyboardInterrupt

[mhils]

Ugs - thank for the fantastic detective work, looks like we have a regex with exponential search behavior in markdown2. I'll take a look. :)

Edit: Yup...

Running `markdown2.markdown()` on the following snippet took 49.3 seconds:
====================
Foo the foo.


###### Examples
> >>> class Foo(object):
> ...     pass
> >>> foo = Foo()
> >>> paths = foo.foo(dir_="src")
> >>> [str(p) for p in paths][:2]
> ['foo', 'bar']

====================

[mhils]

That indeed was the case! Fixed in 06beb3f, I'll file a PR upstream momentarily.
Fun fact: This bug existed for 14 years! 😅

[andrewasheridan]

14 years! Amazing :D

Thank you for your efforts