Markdown Export

[mhils]

hey any chance we can create a ticket for the markdown output support? I saw there is one that was closed but the markdown branch is over a hundred commits behind and I would love to track it properly :)

there's quite a bit wrong with simply renaming html files to md files right now. I am using mkdocs but, for example, the search feature of mkdocs will pick up on the html and show a lot of nonsense in the results because of it. I'm also struggling with links not working because of differences in how mkdocs treats files as directories in the URL but pdoc links to html files (that do not exist). I think proper markdown output support would make this challenge at least a tiny bit easier since the jinja templates should be simpler to modify to fit any weird mkdocs→pdoc linking issues.

thanks for the work!

Originally posted by @zeelot in #203 (comment)

[mhils]

hey any chance we can create a ticket for the markdown output support? I saw there is one that was closed but the markdown branch is over a hundred commits behind and I would love to track it properly :)

Here we go. :)

there's quite a bit wrong with simply renaming html files to md files right now. I am using mkdocs but, for example, the search feature of mkdocs will pick up on the html and show a lot of nonsense in the results because of it. I'm also struggling with links not working because of differences in how mkdocs treats files as directories in the URL but pdoc links to html files (that do not exist). I think proper markdown output support would make this challenge at least a tiny bit easier since the jinja templates should be simpler to modify to fit any weird mkdocs→pdoc linking issues.

Integrating with mkdocs is definitely a bit quirky at the moment. Not sure if you have seen https://github.com/mitmproxy/pdoc/tree/main/examples/mkdocs, there are a few mkdocs-specific comments in the README.

Whatsoever, I agree that a direct Markdown export would work better here. If you are interested in building out Jinja2 templates for Markdown output I can rebase the markdown branch onto main, let me know what you think! 😃

[zeelot]

This is great, thank you! I have looked at the mkdocs example and have something half-working using similar setups. But I switched to the mkdocs material theme and things fell apart pretty quickly. It's just not something I want to continue hacking my way around but I would love to help contribute to this markdown export option. If you can rebase the old branch, I can continue digging. I'm also new to jinja2 templates so progress has been a bit slow but if you could point me in the right direction as to how I can help get a markdown template going, I'd love to give it a shot.

Do you think the easiest thing for now would be to simply keep using your latest release and use the --template-directory option to create some new html templates that simply output markdown? Would that be a good enough starting point for you to then take and bake into your markdown branch? Any tips about the particular sections or macros and such I should be looking to replace with markdown versions?

[mhils]

Excellent.

Do you think the easiest thing for now would be to simply keep using your latest release and use the --template-directory option to create some new html templates that simply output markdown?

Yes, exactly. The best way to start is probably to tinker with module.html.jinja2, https://jinja.palletsprojects.com/en/3.0.x/templates/ is a handy reference if you are new to Jinja2. The "meat" is the module_contents block which calls a bunch of macros defined earlier in the file. Feel free to leave out any bits you consider unnecessary, I don't think we have to mirror everything from the HTML templates.

Would that be a good enough starting point for you to then take and bake into your markdown branch?

If you replace the default templates and polish them to a nice Markdown output I can take care of everything else. :)

[zeelot]

I think I'm making some good progress on this but would love answers to a couple confusing bits.

Could you help me understand what link() does and where I can find the definition of it? I don't see it in the jinja docs:

<span class="base">{{ base[:2] | link(text=base[2]) }}</span>

Similarly, I'm not sure how linkify works exactly either. I've pulled most of these out but want to make sure I add any equivalent markdown versions.

[zeelot]

created a PR in my project with the latest version of my templates here: codeghetti/seagulls-py#39

Would love some initial review and feedback :)

Some notes:

• I'm being very specific about how I want to display the docs on my project
• I'm using markdown extensions needed to add attributes to elements and in order to render admonitions

Are things moving in the right direction? Is this too tightly coupled to certain things?

[mhils]

Could you help me understand what link() does and where I can find the definition of it? I don't see it in the jinja docs:

pdoc/pdoc/render_helpers.py

Lines 135 to 175 in fd890b6

	@pass_context
	def linkify(context: Context, code: str, namespace: str = "") -> str:
	"""
	Link all identifiers in a block of text. Identifiers referencing unknown modules or modules that
	are not rendered at the moment will be ignored.
	A piece of text is considered to be an identifier if it either contains a `.` or is surrounded by `<code>` tags.
	"""
	def linkify_repl(m: re.Match):
	text = m.group(0)
	identifier = removesuffix(text, "()")
	# Check if this is a local reference within this module?
	mod: pdoc.doc.Module = context["module"]
	for qualname in qualname_candidates(identifier, namespace):
	doc = mod.get(qualname)
	if doc and context["is_public"](doc).strip():
	return f'<a href="#{qualname}">{text}</a>'
	# Find the parent module.
	try:
	module, qualname = split_identifier(context["all_modules"], identifier)
	except ValueError:
	return text
	else:
	if qualname:
	qualname = f"#{qualname}"
	return f'<a href="{relative_link(context["module"].modulename, module)}{qualname}">{text}</a>'
	return Markup(
	re.sub(
	r"""
	(?<!/)(?!\d)[a-zA-Z_0-9]+(?:\.(?!\d)[a-zA-Z_0-9]+)+(?:\(\))? # foo.bar
	|
	(?<=<code>)(?!\d)[a-zA-Z_0-9]+(?:\(\))?(?=</code>) # `foo` or `foo()`
	""",
	linkify_repl,
	code,
	flags=re.VERBOSE,
	)
	)

pdoc/pdoc/render_helpers.py

Lines 178 to 202 in fd890b6

	@pass_context
	def link(context: Context, spec: tuple[str, str], text: Optional[str] = None) -> str:
	"""Create a link for a specific `(modulename, qualname)` tuple."""
	mod: pdoc.doc.Module = context["module"]
	modulename, qualname = spec
	# Check if the object we are interested is also imported and re-exposed in the current namespace.
	doc = mod.get(qualname)
	if doc and doc.taken_from == spec and context["is_public"](doc).strip():
	if text:
	text = text.replace(f"{modulename}.", f"{mod.modulename}.")
	modulename = mod.modulename
	if mod.modulename == modulename:
	fullname = qualname
	else:
	fullname = removesuffix(f"{modulename}.{qualname}", ".")
	if qualname:
	qualname = f"#{qualname}"
	if modulename in context["all_modules"]:
	return Markup(
	f'<a href="{relative_link(context["module"].modulename, modulename)}{qualname}">{text or fullname}</a>'
	)
	return text or fullname

You can probably change those to emit Markdown instead and then add a | to_html filter in the HTML template.

Would love some initial review and feedback :)

Thanks, this looks good overall! I feel the "View Source" part is probably a bit too much for a default template, but maybe we can convert show_source to Optional[bool] with None meaning the template decides (show soure for HTML, but not for Markdown).

We probably also want to share is_public between templates, I'll have to think about how to do this best. :)

[zeelot]

You can probably change those to emit Markdown instead and then add a | to_html filter in the HTML template.
How would I go about doing that? Is that something I can overwrite similarly to template files or is this started to enter into the area of making changes to a fork of your code? It might be time for me to actually set up a development environment for your code instead of just overwriting templates for much longer :) there are definitely things in my jinja files that are getting around the fact that these templates render html by default!

I can give this some more attention after work tonight.

[zeelot]

I just pushed a version of the markdown template that I am actually decently happy with. I know it's very specific to mkdocs and my theme so maybe we can figure out what I can do to abstract it a bit more or give folks control over some of these as options?

[mhils]

Thanks! The pdoc bits look great here. I think we fundamentally need to answer who the pdoc output is for. Is it for human consumption, or is it for other tools like mkdocs? For now I would be fine with declaring mkdocs as the primary target, so the specializations here (admonitions, code blocks for highlighting) are fine with me. We should tackle link/Linkify though, that is a major usability improvement for API consumers :)

[mhils]

@zeelot made some great progress in codeghetti/seagulls-py#39 - I think we have a reasonable template to start with. 🥳

I have a PhD thesis to write at the moment, so it will take me a while to do the rest of the pdoc lifting. If someone needs this urgently, @zeelot's project has the relevant template files which you can just drop in for the HTML one to get Markdown output. :)

[mhils]

I have pondered on this for a quite a bit over the last few months and have come to the conclusion that I'd like to declare pure Markdown export as out of scope for the time being. Here's my reasoning:

• The HTML export is in many ways superior to what raw Markdown can potentially achieve. For example, we now have syntax highlighting in the function definitions, which is hard to achieve with Markdown. Having only one output format allows us to specialize on that and exploit its benefits.
• The vast majority of pdoc user that I'm aware of uses pdoc as a standalone documentation system with some adjustments to the default template. There is only a small subset of users that embeds pdoc's docs into other content systems (mkdocs, hugo, ...). For the small subset which embeds pdoc, the majority is better served by embedding the generated HTML instead of Markdown. The remaining users are mostly on mkdocs, which does not take raw HTML very well. For this use case, @pawamoy et al.'s mkdocstrings is a wonderful alternative that integrates well into the ecosystem. Long story short, raw Markdown output is really a niche use case.
• A second output format requires a significant amount of maintenance, which is at odds with keeping this project sustainable.

Long story short, as a maintainer I have the unfortunate duty of making some hard calls, and not adding Markdown output is one of them. @zeelot's template will continue to function for those who want this functionality, but it's not becoming part of pdoc itself.

[patrick91]

@mhils thanks for the update!