[PROPOSAL] Merge ML APIs to opensearch-py

[nathaliellenaa]

What are you proposing?

In regard to this proposal, we want to merge ML APIs from opensearch-py-ml to opensearch-py.

What problems are you trying to solve?

Currently, users need to install and manage two different packages when they want to use ML functionalities. By merging ML APIs to opensearch-py, users who want to use ML functionalities would only need to install and manage a single package, reducing their effort and confusion.

What is the developer experience going to be?

The coordination of updates and releases for ML features will be easier. With ML APIs integrated into opensearch-py, the codebase management will be simplified. There is also opportunity to refactor and optimize ML APIs during the merger process.

What is the user experience going to be?

Users only require a single package installation for both core OpenSearch operations and ML functionalities. It will simplify dependency management in their projects.

Are there breaking changes to the User Experience?

This change should not introduce breaking changes to the existing user experience. Users who currently use opensearch-py-ml for ML APIs might need to update their import statements, but the functionality and method signatures should remain the same.

What will it take to execute?

• Move ML API code from opensearch-py-ml to opensearch-py, ensuring proper integration and maintaining existing functionality.
• Develop and run comprehensive tests to ensure all ML APIs work correctly in the new environment and don't interfere with existing opensearch-py features.
• Update all relevant documentation, including API references, user guides, and migration guides for existing opensearch-py-ml users.
• Determine appropriate versioning strategy for the updated opensearch-py package.

[dblock]

Seems reasonable. Note that the raw API calls to OpenSearch should be generated from spec in here, and not hand-written.

[saimedhi]

• Connected with @nathaliellenaa to show where and how to add specs. Suggested to start with adding ML client 1 API spec in https://github.com/opensearch-project/opensearch-api-specification/tree/main. And can see how generated API looks in opensearch-py.
• Need to decide where these APIs need to be present in opensearch-py client. (under which namespace or namespaces?)

CC: @minalsha

[dblock]

@nathaliellenaa The ML plugin APIs are in https://github.com/opensearch-project/opensearch-api-specification/blob/main/spec/namespaces/ml.yaml with tests in https://github.com/opensearch-project/opensearch-api-specification/tree/main/tests/plugins/ml. According to opensearch-project/opensearch-api-specification#168 some are missing, which is where I'd start.

• DELETE /_plugins/_ml/agents/{}
• DELETE /_plugins/_ml/connectors/{}
• DELETE /_plugins/_ml/controllers/{}
• DELETE /_plugins/_ml/memory/{}
• DELETE /_plugins/_ml/model_groups/{}
• DELETE /_plugins/_ml/models/{}
• DELETE /_plugins/_ml/tasks/{}

Then any of these that exist in opensearch-py-ml should be deleted ensuring backwards compatibility, and use the generated code in opensearch-py.

[nathaliellenaa]

I'll start by adding the missing Models APIs along with the tests.