[META] Support asynchronous operations on an extension

[cwperks]

Anomaly Detection (AD) backend is currently being converted from a plugin to an extension. AD runs jobs in the background to monitor indices in a cluster for a detector. In the plugin model, AD serializes the user (including roles and backend roles) upon detector creation and saves it as part of the detector metadata. When it comes time to run the detector, AD performs roles injection to evaluate a dedicated user (called plugin) with the roles stored in the detector’s metadata to determine if the original user could perform the action. This will not work for AD running as an extension.

More generally, outside of the Anomaly Detection use-case, guidance needs to be provided for extensions developers on how to implement extensions that want to interact with the OpenSearch cluster asynchronously. When implementing microservices today, developers will typically configure the OpenSearch client used in the microservice with a username and password that is defined in the internal user list of OpenSearch. This will not work for extensions. The password will never be shared with an extension. An alternative approach to asynchronous jobs needs to be developed.

See example from the user guide of opensearch-py: https://github.com/opensearch-project/opensearch-py/blob/main/USER_GUIDE.md#creating-a-client for configuring a client to connect with a secure cluster.

Resolves:

• As a user, my requests to extensions that run background tasks require an identity to execute

---

Open Question: For anomaly detection as an extension, will the asynchronous tasks be run from nodes within the cluster or will they be run on the extension? How does scheduling work with extensions?

List of tasks to complete to support this feature:

• [Decision Doc] Async Operations for Extensions #2574 - See [Decision Doc] Async Operations for Extensions #2574 (comment) for follow-up design proposal
• [Extensions] Create a single secure index to store identity information for scheduled jobs #2624
• [Extensions] Issue access token on behalf of user stored in scheduled job identity index #2603
• [Extensions] Create a way to identify if an Extension REST Handler will create a scheduled job #2623
• [Extensions] Save user details to scheduled job identity index on postIndex of a plugin/extension scheduling a new job #2625
• [Extensions] Create an interface that Job Scheduler can use to manage user info associated with a scheduled job #2626
• [Extensions] Modify Job Schedule's ScheduleParser to read in the refresh token field #2627
• [Extensions] Delete user details to scheduled job identity index on postDelete of a plugin/extension deleting a scheduled job #2776