[RFC] Migrate the PPL docs from rST to Markdown

[Swiddis]

Problem Statement
While reStructuredText (rST) has lots of flexible features for documentation, including custom directives and support for better organization schemes, we're hitting issues with portability. The documentation site runs on Markdown, as well as sources like the upcoming PPL playground. Since previous proposals to self-host documentation from our rST were rejected, we're left with a lot of work to migrate the rST documents to Markdown to host on the official OpenSearch site.

Current State
Our docs are primarily in rST. They're being separately migrated to Markdown as part of two separate projects. This is duplicated effort that will need to be repeated for all future documentation changes.

Long-Term Goals

• Establish a single source of truth for documentation
• Require no manual migration work to get docs from this repo to downstream sources.
• Finalize the format for how we want to document PPL features.

Proposal
Migrate the documentation from rST to Markdown. The bulk of the work can be done with automatic conversion. There will be some other work left over at the boundaries where the conversion is clunky.

@kylehounslow noted that it would be easier/more deterministic to do rST -> HTML -> Markdown instead of rST -> Markdown directly. We can also split all the PPL code blocks into separate command blocks and result blocks (with appropriate doctest tweaks). This can be done in a later stage after the initial migration.

@ritvibhatt may have prior art on this as part of the migration for the documentation site.

Alternative
rST does have some valuable features and I've advocated for it before in projects that wanted more power than what Markdown provides. If we want to leverage the features of rST instead of migrating, we should pursue an avenue for hosting these pages directly. Otherwise, we keep paying the migration cost and lose the rST features anyway, in order to maintain the downstream doc hosting.

[LantaoJin]

I asked a same question several years ago and get answer that we need it for doctest. cc @penghuo .
My thought is how above add a tool or script to generate Markdown file from rST files instead of migration. We can generate Markdown files on demand.

[Swiddis]

Discussed with team in triaging meeting:

I think that we ultimately want one source of truth, I don't want any dev flow that involves making documentation changes in multiple places. So that leaves us with two options: a dynamic migration script (as Lantao mentioned) and a one-time migration that's copied around.

IMO a migration script is risky as we lose determinism with the output. Visual bugs might not get reported for a long time and we need to handle migrating nontrivial rST->Markdown constructs automatically instead of figuring it out once. We end up needing to maintain the migration code. (This could be worth it if we need to output in multiple formats, @ritvibhatt was mentioning we might need a custom XML conversion for AWS docs?)

I know when we were discussing approaches for maintaining documentation long-term, I brought up the idea of having a versioned index in mainline and using migration scripts to ship it everywhere, and this was almost-unanimously voted against as too complex.

---

The same doctest concern was brought up in the meeting, so I looked into it more. Doctest supports arbitrary parsers, we just need to implement a DocTestParser and it'll work as-is. It looks like Sybil provides a markdown parser out-of-the-box. Writing our own would work with @kylehounslow's split-blocks suggestion.

[dai-chen]

For your reference, here is the design doc that explains why and benefits of doctest with RST: https://github.com/opensearch-project/sql/blob/main/docs/dev/testing-doctest.md

[LantaoJin]

For your reference, here is the design doc that explains why and benefits of doctest with RST: https://github.com/opensearch-project/sql/blob/main/docs/dev/testing-doctest.md

Years ago, I had the same question: "Since we already have UT and IT, why do we need to test queries in the documentation?" Until one day (before the doctest for Calcite were enabled), we introduced an incorrect PPL query into the documentation, and users actually tried it and reported failure. Yes, the doctest is useful, to protect us from writing incorrect queries in user doc. But at the same time, how serious is it that a few isolated errors appear in the user doc of the open-source project? I'm open to the proposal if a markdown solution could benefit us more.

[anasalkouz]

Maintaining two format is painful. If there a way we can enable doctest with markdown, then this might be the right direction moving forward.

[Swiddis]

Yes, the doctest is useful, to protect us from writing incorrect queries in user doc. But at the same time, how serious is it that a few isolated errors appear in the user doc of the open-source project?

I agree that there's a considerable maintenance burden for it today. Over the years I've had to resolve many issues that boiled down to "doctest is broken" instead of "doctest's tests are broken." This could overall be an opportunity to simplify our model, but I'm not sure off instinct how to quantify the exact maintainability problems and how to fix them.

I do think doctest has value as a lightweight E2E test that's guaranteed to only use externally accessible endpoints. In general, for running some form of automated testing on new environments (serverless, new CLI), I've found it easier to hack the doctest runner for the new env than to try and get integ tests to run.

If you have a suggestion for how to reduce the debt while keeping that value, I'm open to it.

[kylehounslow]

I've drafted a PR for migrating PPL docs to markdown: #4912
TL;DR:

1. Markdown conversion was trivial and doctests were updated to successfully run against markdown code-blocks.
2. Existing GitHub docs pages remain intact and unchanged. See my fork here
3. I've validated that automatic export of PPL docs to main OpenSearch documentation is possible via python script. Only some metadata is injected for Jekyll compatibility.

This means when sql user/docs/ is updated, the export script can be run and devs open PR using the diff. Any feedback given in documentation-website PR will be applied back to this repo and script re-run. This sql repo docs will be the single source of truth.