Code Snippet Testing

[bmorelli25]

Summary

Support the ability to test code snippets.

See https://github.com/elastic/elasticsearch/tree/main/docs#test-code-snippets for more information.

Related issue that describes functionality in Asciidoc system: elastic/docs#2867

Partially related to #7.

[leemthompo]

Re https://elastic.slack.com/archives/CJE3WCFEH/p1741162012377709?thread_ts=1741124580.043689&cid=CJE3WCFEH

I guess we'll need snippet testing in the elasticsearch repo and docs-content at the very least. Elasticsearch should be back online as soon as the test comments are restored.

[lcawl]

There's a thread about some automated testing that was occurring in the ECK docs too here: https://elastic.slack.com/archives/C05PP2LEC1X/p1741273388324539

[leemthompo]

Current state of play for elasticsearch snippets:

• @colleenmcginnis wrote a script to restore the comments to the files in the elasticsearch repo
• @georgewallace has a PR to add those back to the docs: add es code comments for testing elasticsearch#124283

To restore the existing snippet testing framework to the elasticsearch repo, we'll need the @elastic/es-delivery team to update the parsing logic to parse .md files using the new comment syntax.

As @szabosteve mentioned elsewhere this probably just requires some minimal updates to the MDX logic added for a previous docs migration test.

[brianseeders]

I've started work on making the markdown docs work with snippet testing for Elasticsearch using the markdown from elastic/elasticsearch#124283. I'm not familiar with how this stuff works, so am learning as I go.

I discovered an issue related to the fact that the docs content was also re-organized as part of the markdown conversion.

https://github.com/elastic/elasticsearch/blob/v9.0.0-beta1/docs/reference/vectors/vector-functions.asciidoc?plain=1#L73

See this TESTSETUP comment. TESTSETUP has to be used on the first snippet in a file, like it is here, to denote setup for all of the snippets in the file.

https://github.com/elastic/elasticsearch/blob/d0f24003a84eb8139144feab44b0a1ca46d74104/docs/reference/query-languages/query-dsl-script-score-query.md?plain=1#L373

This is the new location of that content. It's been merged with other content/snippets in a new file. This results in the following error message when trying to run the tests:

TESTSETUP: wasn't first. TESTSETUP can only be used in the first snippet of a document.

So, moving these snippets around is basically like taking test cases out of their test suites and putting them into other ones, it doesn't really work.

Not sure what the solution is or just how big the impact is at the moment, just wanted to get a note here as it's close to the end of my day.

[georgewallace]

@bmorelli25 adding a link to the thread where we are left at now. https://elastic.slack.com/archives/CJE3WCFEH/p1750707001785249?thread_ts=1750327877.680069&cid=CJE3WCFEH. Brian did a bunch of work to try to get this working but with the drift with the migration and the conversion there were multiple issues that made it hard to determine the source of the problem.

[bmorelli25]

cc @KOTungseth for visibility