Validating code in documentation

[lilatomic]

The code samples in the documentation may not be updated when the code they reference is. The case I found was with the plugin howto docs, where we changed the code but not the samples for writing a plugin. We may also not be running linters or formatters on them.

I think the main value we gain is in simple validations, like if the sample is syntactically valid and if all the symbols are importable. I think that will do most of the work of prompting us to update the samples when the code they reference updates.

---

I see several distinct types of code samples:

1. full usage examples: These include a BUILD file and the relevant supporting samples. These use Pants as a cli tool.
2. internal usage examples: By example, the plugin howtos. They need to reference Pants as a python package
3. fragment samples: incomplete fragments. The challenge here is that they cannot stand on their own; if we want to validate them, we need to provide some scaffolding or an adapter. This might be as simple as adding the imports, but might also involve creating supporting classes or function bodies left blank.

I think these 3 cases need to be treated separately. Justifying, for the full usage examples, we may have to spin up separate rule_runners, as the different examples might have conflicting Pants configurations. At a minimum, in some way we'd need to spin up a Pants instance to test them. The internal usage examples might be suitable to extract in whole from the markdown, but the incomplete examples will need to have their code synthesised together with supporting information.

---

Options:

• Pull code out of markdown into synthetic targets.
• Pull code into markdown from full files
• Use Jupyter book, which would allow us to write Jupyter notebooks along with myst docs

I see type 1 (full examples) as a challenge to all of these, but it's reasonably straightforward to just throw them all at a rule_runner

[benjyw]

This is a great use case to take into account when we evaluate replacements for our current doc publishing via readme.com