proposal: testable markdown documentation

[thehowl]

From #1913.

I would prefer if testscript / txtar remained an internal tool. Discussing with @gfanton on the review meeting, we may still benefit from unifying how txtar works (rather than having two differing impl's in gnoland and cmd/gno), and having it as a command (which we can still get coverage information for).

However, long-term, I don't like to use txtar as documentation. While they are decently straightforward to use, thanks to @gfanton's acute design taste :), I think we should try to have something better to make testable examples.

An idea that sprung up, was to have markdown files for documentation of a realm, with the code blocks being testable.

@moul: Yes, or alternatively, provide testable examples and adhere to documentation standards that include comments in .gno files.

We need to have good support for testable examples our tests, same as Go. From this issue and ensuing discussion, with @gfanton we discussed the possibility of having a way to create tests as markdown files; which serve primarily as documentation, but can show usage examples using gnokey and which can be tested.

This should be in opposition to using txtar tests for public usage; keeping them mostly for internal testing, but allowing to document how a realm is practically used publicly.

[AnhVAR]

@thehowl i think we can make test file in markdown files like example: https://github.com/Jc2k/pytest-markdown/blob/main/tests/integration/example.md. After that we can use github action to write report in same as markdown file to show test coverage and pass/fail test case. https://github.com/becheran/go-testreport

[notJoon]

If we think about it simply, I can try the following approach. After parsing, generate a temporary .gno file with the corresponding code block, execute it, and then output the result (dynamically inserting it into the document). Also, It would be good to add a cache to reduce the unnecessary operations.

flowchart TD
    A[Parse Markdown Document] --> B[Extract Code Blocks]
    B --> C{Check Code Blocks}
    C -->|None| D[End]
    C -->|Exists| E[Preparation Before Code Execution]
    E --> F[Initialize Error Handling and Logging Mechanisms]
    F --> G[Set Execution Environment Resource Limits]
    G --> H[Execute Code and Monitor]
    H --> I{Check Cache for Result}
    I -->|Cached| J[Use Cached Result]
    I -->|No Cache| K[Execute Code]
    K --> L[Process Results]
    L --> M{Check for Errors}
    M -->|Errors Present| N[Log Errors and Provide Feedback to User]
    M -->|No Errors| O[Log and Save Results]
    J --> P[Insert Results into Markdown Document]
    O --> P
    N --> D

    style D fill:#f96,stroke:#333,stroke-width:2px
    style J fill:#bbf,stroke:#333,stroke-width:2px
    style O fill:#bbf,stroke:#333,stroke-width:2px
    style N fill:#fbb,stroke:#333,stroke-width:2px

Loading

[moul]

I suggest that we:

• Support testable examples, similar to Go, so we can potentially replace most of the txtar needs while providing better documentation automatically.
• Consider writing a library that will allow using Go to test things, while generating working gnokey commands. This could be something similar to the library I wrote for Go/HTTP: https://github.com/moul/http2curl. import "test/gnokey"; gnokey.MaketxCall(params...)