big ol' batch of features

- fixed temp doxyfile being written to disk when using `--dry` - added command-line option `--doxygen` - added command-line option `--werror` - added `[set_parent_class]` - added `[add_parent_class]` - added `[remove_parent_class]` - added config file options for `images` and `examples` - added downloading of tagfiles specified by URIs - improved re-routing of Doxygen and m.css warning and error messages - refactored config file schema to better group source options - actually wrote the README lmao
marzer · May 9, 2021 · 98c9756 · 98c9756
1 parent 8ffd81d
commit 98c9756
Show file tree

Hide file tree

Showing 8 changed files with 693 additions and 282 deletions.
diff --git a/.editorconfig b/.editorconfig
@@ -9,6 +9,9 @@ end_of_line = lf
 trim_trailing_whitespace = true
 charset = utf-8
 
+[*.{md,markdown}]
+trim_trailing_whitespace = false
+
 [*.{gitattributes,yml,vcxproj,vcxproj.filters,sln,rc,clang-format}]
 indent_style = space
 

diff --git a/README.md b/README.md
@@ -1,6 +1,127 @@
 # poxy
 Documentation generator for C++ based on Doxygen and [mosra/m.css](https://mcss.mosra.cz/).
 
-```
+ - [Overview](#overview)
+ - [Example](#example)
+ - [Installation](#installation)
+ - [Usage](#usage)
+ - [Migrating from Doxygen](#migrating-from-doxygen)
+ - [Config file options](#config-file-options)
+
+<br><br>
+
+## Overview
+[mosra/m.css] is a Doxygen-based documentation generator that significantly improves on Doxygen's default output
+by controlling some of Doxygen's more unruly options, supplying it's own slick HTML+CSS generation and adding
+a fantastic live search feature. **Poxy** builds upon both by:
+- Moving the configuration out into a TOML file
+- Preprocessing the Doxygen XML to fix a bunch of Doxygen _~~bugs~~_ quirks
+- Postprocessing the generated HTML to improve syntax highlighting and add a few other improvements
+- Allowing source, image and example directories to be recursive or shallow on a per-directory basis
+- Automatically defining C++ language feature macros based on your project's target C++ version
+- Automatically integrating the cppreference.com doxygen tagfile
+- Providing a number of additional built-in doxygen `@alias` commands
+- Giving more control over the HTML inline using square-bracket `[tags][/tags]`
+- Quite a bit more!
+
+<br><br>
+
+## Example
+The homepage + documentation for [toml++] is built using poxy:
+- homepage: [marzer.github.io/tomlplusplus](https://marzer.github.io/tomlplusplus/)
+- config file: [`poxy.toml`](https://github.com/marzer/tomlplusplus/blob/master/docs/poxy.toml)
+
+<br><br>
+
+## Installation
+### Prerequisites:
+- Python 3
+- Doxygen (preferably a version from this decade, though most will be OK)
+### Then:
+```sh
 pip install poxy
 ```
+
+<br><br>
+
+## Usage
+Poxy is a command-line application.
+```
+poxy [-h] [-v] [--dry] [--threads <N>] [--m.css <path>] [--doxygen <path>] [--werror] [config]
+
+Generate fancy C++ documentation.
+
+positional arguments:
+  config            a path to a poxy.toml, Doxyfile.mcss, Doxyfile, or a directory containing one/any/all (default: ./)
+
+optional arguments:
+  -h, --help        show this help message and exit
+  -v, --verbose     enables very noisy diagnostic output
+  --dry             does a 'dry run' only, stopping after emitting the effective Doxyfile
+  --threads <N>     sets the number of threads to use (default: automatic)
+  --m.css <path>    specifies the version of m.css to use (default: uses the bundled one)
+  --doxygen <path>  specifies the Doxygen executable to use (default: finds Doxygen on system path)
+  --werror          always treats warnings as errors regardless of config file settings
+```
+The basic three-step to using Poxy is similar to Doxygen:
+1. Create your `poxy.toml` (Poxy's answer to the `Doxyfile`)
+2. Invoke Poxy on it: `poxy path/to/poxy.toml` (or simply `poxy` if the cwd contains the config file)
+3. See your HTML documentation `<cwd>/html`
+
+ℹ&#xFE0F; If there exists a `Doxyfile` or `Doxyfile-mcss` in the same directory as your `poxy.toml` it will be loaded
+first, then the Poxy overrides applied on top of it. Otherwise a 'default' Doxyfile is used as the base.
+
+<br><br>
+
+## Config file options
+
+For a self-contained `poxy.toml` example to copy and paste from, see [the one used by toml++](https://github.com/marzer/tomlplusplus/blob/master/docs/poxy.toml).
+
+For a full list of options, with full descriptions, schemas and usage examples, see the [Configuration options] wiki page.
+
+<br><br>
+
+## Migrating from Doxygen
+Generally the relevant `Doxyfile` options will have a corresponding `poxy.toml` option
+(or be replaced by something more specific) so migration is largely a transcription and box-ticking exercise,
+though there are a few gotchas:
+
+#### **⚠&#xFE0F; The majority of Doxygen's options are controlled by Poxy**
+Very few of the configurable options from the Doxyfile remain untouched by Poxy. This is intentional;
+[m.css] is opinionated, and Poxy even moreso. There are a few instances where information can flow from a Doxyfile to
+Poxy, but these situations are few, and all are documented explicitly on the [Configuration options] wiki page.
+
+#### **⚠&#xFE0F; All relative input paths are relative to the config file, _not_ CWD**
+This is in contrast to Doxygen, which has all paths be relative to the Doxygen process' current working directory
+regardless of where the Doxyfile was. I've always personally found that to be nothing but a source of error,
+so Poxy does away with it.
+
+#### **⚠&#xFE0F; Output is always emitted to CWD**
+Poxy always emits the output html to `<cwd>/html`. This is largely to simplify the HTML post-process step.
+
+#### **⚠&#xFE0F; Poxy config files are self-contained**
+There is no equivalent to Doxygen's `@INCLUDE`. If your project is structured in such a way that an N-levels-deep
+Doxyfile hierarchy is necessary, Poxy isn't for you.
+
+<br><br>
+
+## Why the name "Poxy"?
+
+Originally it was simply called "dox", but there's already a C++ documentation project with that name, so I smashed
+"python" and "dox" together and this is what I came up with.
+
+Also "poxy" can be slang for cheap, inferior, poor quality, etc., which I thought was funny.
+
+<br><br>
+
+## License and Attribution
+This project is published under the terms of the [MIT license](https://github.com/marzer/poxy/blob/main/LICENSE.txt).
+
+Significant credit must go to Vladimír Vondruš ([mosra]) and his amazing [m.css] framework. Poxy bundles a fork of m.css, used per the [MIT/Expat license](https://github.com/mosra/m.css/blob/master/COPYING) (which can also be found in the installed python package).
+
+[m.css]: https://mcss.mosra.cz/documentation/doxygen/
+[mosra]: https://github.com/mosra
+[mosra/m.css]: https://mcss.mosra.cz/documentation/doxygen/
+[toml++]: https://marzer.github.io/tomlplusplus/
+[C++ feature test macros]: https://en.cppreference.com/w/cpp/feature_test
+[Configuration options]: https://github.com/marzer/poxy/wiki/Configuration-options
diff --git a/poxy/doxygen.py b/poxy/doxygen.py
@@ -76,9 +76,11 @@ def _format_for_doxyfile(val):
 
 class Doxyfile(object):
 
-	def __init__(self, doxyfile_path, cwd=None, logger=None):
+	def __init__(self, doxyfile_path, cwd=None, logger=None, doxygen_path=None, flush_at_exit=True):
 		self.__logger=logger
 		self.__dirty=True
+		self.__text = ''
+		self.__autoflush=bool(flush_at_exit)
 
 		# the path of the actual doxyfile
 		self.path = coerce_path(doxyfile_path).resolve()
@@ -87,7 +89,8 @@ def __init__(self, doxyfile_path, cwd=None, logger=None):
 		self.__cwd = Path.cwd() if cwd is None else coerce_path(cwd).resolve()
 		assert_existing_directory(self.__cwd)
 
-		self.__text = ''
+		# doxygen itself
+		self.__doxygen = r'doxygen' if doxygen_path is None else coerce_path(doxygen_path)
 
 		# read in doxyfile
 		if self.path.exists():
@@ -100,7 +103,7 @@ def __init__(self, doxyfile_path, cwd=None, logger=None):
 		else:
 			log(self.__logger, rf'Warning: doxyfile {self.path} not found! A default one will be generated in-memory.', level=logging.WARNING)
 			result = subprocess.run(
-				r'doxygen -s -g -'.split(),
+				[str(self.__doxygen), r'-s', r'-g', r'-'],
 				check=True,
 				capture_output=True,
 				cwd=self.__cwd,
@@ -118,11 +121,11 @@ def cleanup(self):
 		if 1:
 			log(self.__logger, rf'Invoking doxygen to clean doxyfile')
 			result = subprocess.run(
-				r'doxygen -s -u -'.split(),
+				[str(self.__doxygen), r'-s', r'-u', r'-'],
 				check=True,
 				capture_output=True,
 				cwd=self.__cwd,
-				encoding='utf-8',
+				encoding=r'utf-8',
 				input=self.__text
 			)
 			self.__text = result.stdout.strip()
@@ -204,5 +207,5 @@ def __enter__(self):
 		return self
 
 	def __exit__(self, type, value, traceback):
-		if traceback is None:
+		if traceback is None and self.__autoflush:
 			self.flush()
diff --git a/poxy/fixers.py b/poxy/fixers.py
@@ -23,8 +23,21 @@ class CustomTags(object):
 	'''
 	Modifies HTML using custom square-bracket [tags].
 	'''
-	__double_tags = re.compile(r"\[\s*(span|div|aside|code|pre|h1|h2|h3|h4|h5|h6|em|strong|b|i|u|li|ul|ol)(.*?)\s*\](.*?)\[\s*/\1\s*\]", re.I | re.S)
-	__single_tags = re.compile(r"\[\s*(/?(?:span|div|aside|code|pre|emoji|(?:parent_)?set_name|(?:parent_)?(?:add|remove|set)_class|br|li|ul|ol|(?:html)?entity))(\s+[^\]]+?)?\s*\]", re.I | re.S)
+	__double_tags = re.compile(
+		r'\[\s*('
+			+ r'span|div|aside|code|pre|h1|h2|h3|h4|h5|h6|em|strong|b|i|u|li|ul|ol'
+			+ r')(.*?)\s*\](.*?)\[\s*/\1\s*\]',
+		re.I | re.S
+	)
+	__single_tags = re.compile(
+		r'\[\s*(/?(?:'
+			+ r'img|span|div|aside|code|pre|emoji'
+			+ r'|(?:parent_)?set_(?:parent_)?(?:name|class)'
+			+ r'|(?:parent_)?(?:add|remove)_(?:parent_)?class'
+			+ r'|br|li|ul|ol|(?:html)?entity)'
+			+ r')(\s+[^\]]+?)?\s*\]',
+		re.I | re.S
+	)
 	__allowed_parents = ('dd', 'p', 'h1', 'h2', 'h3', 'h4', 'h5', 'h6', 'li', 'aside', 'td')
 
 	@classmethod
@@ -60,7 +73,10 @@ def __single_tags_substitute(cls, m, out, context):
 				cp = context.emoji[tag_content][0]
 				return f'&#x{cp:X};&#xFE0F;'
 			return ''
-		elif tag_name in ('add_class', 'remove_class', 'set_class', 'parent_add_class', 'parent_remove_class', 'parent_set_class'):
+		elif tag_name in (
+				r'add_class', r'remove_class', r'set_class',
+				r'parent_add_class', r'parent_remove_class', r'parent_set_class',
+				r'add_parent_class', r'remove_parent_class', r'set_parent_class'):
 			classes = []
 			if tag_content:
 				for s in tag_content.split():
@@ -69,7 +85,7 @@ def __single_tags_substitute(cls, m, out, context):
 			if classes:
 				out.append((tag_name, classes))
 			return ''
-		elif tag_name in ('set_name', 'parent_set_name'):
+		elif tag_name in (r'set_name', r'parent_set_name', r'set_parent_name'):
 			if tag_content:
 				out.append((tag_name, tag_content))
 			return ''
@@ -99,25 +115,25 @@ def __call__(self, doc, context):
 						parent = tag.parent
 						new_tags = soup.replace_tag(tag, str(replacer))
 						for i in range(len(replacer)):
-							if replacer[i][0].startswith('parent_'):
+							if replacer[i][0].find(r'parent_') != -1:
 								if parent is None:
 									continue
-								if replacer[i][0] == 'parent_add_class':
+								if replacer[i][0] in (r'parent_add_class', r'add_parent_class'):
 									soup.add_class(parent, replacer[i][1])
-								elif replacer[i][0] == 'parent_remove_class':
+								elif replacer[i][0] in (r'parent_remove_class', r'remove_parent_class'):
 									soup.remove_class(parent, replacer[i][1])
-								elif replacer[i][0] == 'parent_set_class':
+								elif replacer[i][0] in (r'parent_set_class', r'set_parent_class'):
 									soup.set_class(parent, replacer[i][1])
-								elif replacer[i][0] == 'parent_set_name':
+								elif replacer[i][0] in (r'parent_set_name', r'set_parent_name'):
 									parent.name = replacer[i][1]
 							elif len(new_tags) == 1 and not isinstance(new_tags[0], soup.NavigableString):
-								if replacer[i][0] == 'add_class':
+								if replacer[i][0] == r'add_class':
 									soup.add_class(new_tags[0], replacer[i][1])
-								elif replacer[i][0] == 'remove_class':
+								elif replacer[i][0] == r'remove_class':
 									soup.remove_class(new_tags[0], replacer[i][1])
-								elif replacer[i][0] == 'set_class':
+								elif replacer[i][0] == r'set_class':
 									soup.set_class(new_tags[0], replacer[i][1])
-								elif replacer[i][0] == 'set_name':
+								elif replacer[i][0] == r'set_name':
 									new_tags[0].name = replacer[i][1]
 
 						continue
@@ -247,10 +263,10 @@ def __call__(self, doc, context):
 
 class StripIncludes(object):
 	'''
-	Strips #include <paths/to/headers.h> based on context.strip_includes.
+	Strips #include <paths/to/headers.h> based on context.sources.strip_includes.
 	'''
 	def __call__(self, doc, context):
-		if doc.article is None or not context.strip_includes:
+		if doc.article is None or not context.sources.strip_includes:
 			return False
 		changed = False
 		for include_div in doc.article.find_all(r'div', class_=r'm-doc-include'):
@@ -261,7 +277,7 @@ def __call__(self, doc, context):
 			if not (text.startswith('<') and text.endswith('>')):
 				continue
 			text = text[1:-1].strip()
-			for strip in context.strip_includes:
+			for strip in context.sources.strip_includes:
 				if len(text) < len(strip) or not text.startswith(strip):
 					continue
 				if len(text) == len(strip):
@@ -360,7 +376,7 @@ def __colourize_compound_def(cls, tags, context):
 		assert len(tags) == 1 or tags[-1].string != '::'
 		full_str = ''.join([tag.get_text() for tag in tags])
 
-		if context.highlighting.enums.fullmatch(full_str):
+		if context.code_blocks.enums.fullmatch(full_str):
 			soup.set_class(tags[-1], 'ne')
 			del tags[-1]
 			while tags and tags[-1].string == '::':
@@ -369,7 +385,7 @@ def __colourize_compound_def(cls, tags, context):
 				cls.__colourize_compound_def(tags, context)
 			return True
 
-		if context.highlighting.types.fullmatch(full_str):
+		if context.code_blocks.types.fullmatch(full_str):
 			soup.set_class(tags[-1], 'ut')
 			del tags[-1]
 			while tags and tags[-1].string == '::':
@@ -378,7 +394,7 @@ def __colourize_compound_def(cls, tags, context):
 				cls.__colourize_compound_def(tags, context)
 			return True
 
-		while not context.highlighting.namespaces.fullmatch(full_str):
+		while not context.code_blocks.namespaces.fullmatch(full_str):
 			del tags[-1]
 			while tags and tags[-1].string == '::':
 				del tags[-1]
@@ -496,17 +512,17 @@ def __call__(self, doc, context):
 						or isinstance(prev, soup.NavigableString)
 						or 'class' not in prev.attrs):
 						continue
-					if ('s' in prev['class'] and context.highlighting.string_literals.fullmatch(span.get_text())):
+					if ('s' in prev['class'] and context.code_blocks.string_literals.fullmatch(span.get_text())):
 						soup.set_class(span, 'sa')
 						changed_this_block = True
-					elif (prev['class'][0] in ('mf', 'mi', 'mb', 'mh') and context.highlighting.numeric_literals.fullmatch(span.get_text())):
+					elif (prev['class'][0] in ('mf', 'mi', 'mb', 'mh') and context.code_blocks.numeric_literals.fullmatch(span.get_text())):
 						soup.set_class(span, prev['class'][0])
 						changed_this_block = True
 
 				# preprocessor macros
 				spans = code_block('span', class_=('n', 'nl', 'kt', 'nc', 'nf'), string=True)
 				for span in spans:
-					if context.highlighting.macros.fullmatch(span.get_text()):
+					if context.code_blocks.macros.fullmatch(span.get_text()):
 						soup.set_class(span, 'm')
 						changed_this_block = True