[RFC] Standardizing New Feature Documentation in OpenSearch PPL

[anasalkouz]

Problem Statement

Currently, while new features are highlighted in release notes, there's no standardized way to indicate new features or enhancements in the OpenSearch PPL language documentation. This makes it difficult for users to:

1. Identify which features are available in their version.
2. Track when specific enhancements were introduced.
3. Understand version compatibility requirements.

Current State

Current PPL Documentation Examples

1. New Commands (e.g., regex command):

Version
-------
This command was introduced in OpenSearch 3.3.

2. Enhanced Features (e.g., fields command):

Enhanced Features (Version 3.3.0)
--------------------------------

Proposal

1. Standardized Version Notation

I am proposing a format that is commonly used in other documentation softwares like Sphinx and Python.

For all PPL commands and functions, we should introduce a consistent version information and command status block as following:

Command Name
===========

Version Information
-----------------
.. versionadded:: 3.3.0
    Initial command implementation

.. versionchanged:: 3.4.0
    Added support for feature X

Status
------
[Stable|Experimental]

Description
----------
[Command description]

Syntax
------
[Command syntax]

Parameters
---------
[Parameter details]

Examples
--------
[Usage examples]

2. Version Compatibility Matrix

Add a version compatibility matrix in the main PPL index documentation:

Command	Added In	Modified	Status
regex	3.3.0	-	Stable
fields	1.0.0	3.3.0	Stable

[Swiddis]

I like the versionadded and versionchanged tags, though we'll need to implement handling for them when we host the docs on the web.

I'm not sure about that compatibility matrix, it doesn't say much on its own (fields was modified, but how?) and won't handle if/when we update commands again down the road. If we want a version matrix like this, it should be broken down by feature instead ("field negation: added in 3.3.0"). Would this be better handled by just compiling all the versionadded and versionchanged contents into a changelog file?

If we're introducing the stable/experimental split here, we could also have a versionstabilized? Though I'm not sure how much this matters, since we don't have a specific definition of what those terms mean for us.

[anasalkouz]

I'm not sure about that compatibility matrix, it doesn't say much on its own (fields was modified, but how?) and won't handle if/when we update commands again down the road. If we want a version matrix like this, it should be broken down by feature instead ("field negation: added in 3.3.0"). Would this be better handled by just compiling all the versionadded and versionchanged contents into a changelog file?

Version Matrix view will give a wholistice view for all commands when they are available in the main index file. Instead of just a list them. For simplfication, we can remove the modified column

If we're introducing the stable/experimental split here, we could also have a versionstabilized? Though I'm not sure how much this matters, since we don't have a specific definition of what those terms mean for us.

We can define our own definition of experimental, but in general, I do recommend introduce any new commands to be experimental for couple of months, until we received enough feedback before finalize the commands api params and once we mark the command as stable then we need to ensure backward compatibility. Otherwise, we can ignore this status state and consider all commands are stable

[Swiddis]

Removing "modified" in the matrix is fine, treating the command pages as the primary source for history is good. I also like the definition of experimental as "we reserve the right to change how specific parameters work," we should add it somewhere to the docs to formalize this. (There's similar discussion for OS core, it doesn't seem to have a conclusive answer yet)

For the matrix: if all we need is to mark initial versions and experimental commands, I think it's better to put this in the command list directly, and combine it with a one-sentence description on the commands. The new index would look something like:

Commands

• Command syntax reference
• ad command (1.3+): Anomaly detection based on random-cut forests.
• append command (3.3+, experimental): Append the results of a subsearch to the current search.
• appendcol command (3.1+, experimental): Affix the columns of a subsearch to the current search.
• bin command (3.3+, experimental): Map values into consistent buckets for grouping.
• dedup command (1.0+): Deduplicate documents by specified fields.
  ...

[RyanL1997]

Hi @anasalkouz, thanks for putting this together. Generally I agree with the direction that we should standardize the doc template, and I just wanna drop some thoughts on the long term maintenance of these docs:

1. About the version information

For all PPL commands and functions, we should introduce a consistent version information and command status block as following:

  Version Information
-----------------
.. versionadded:: 3.3.0
    Initial command implementation

.. versionchanged:: 3.4.0
    Added support for feature X

I think the above is a good idea, however, I was wondering how should we handle commands with multiple modifications across versions? Listing every .. versionchanged:: entry could clutter the documentation for older commands. Or Limit versionchanged entries to the most recent 2-3 modifications, with a note directing users to the release notes for a complete history?

something like this?

Version Information
-----------------
.. versionadded:: 2.19.0
    Initial command implementation
.. versionchanged:: 3.3.0
    Added support for feature X
.. versionchanged:: 3.4.0
    Improved performance for large datasets

.. note::
    For a complete history of changes, refer to the `OpenSearch SQL Plugin Release Notes <link-to-release-notes>`_.

2. Future deprecation

Add a version compatibility matrix in the main PPL index documentation:

nit: I think we may need to also add a column for future deprecation of some old command in the future.