Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Design and support for initial extension point (IndicesModule) #2711

Closed
saratvemulapalli opened this issue Apr 1, 2022 · 5 comments
Closed

Comments

@saratvemulapalli
Copy link
Member

saratvemulapalli commented Apr 1, 2022

Add extension support for Indices Module as the first extension point.

@saratvemulapalli saratvemulapalli changed the title Add support for initial extension point (IndicesModule) Design and support for initial extension point (IndicesModule) Apr 5, 2022
@saratvemulapalli
Copy link
Member Author

Adding support to an extension point, we could do a POC for IndexModule.
Commit: saratvemulapalli@2d068df

IndexModule is the extension point for all index level operations.

Plugins work today as they get invoked by IndexService and they have access to IndexModule object.

IndexModule object is an interface to IndexService, where plugins can use all the public methods to add EventListeners, get EngineFactory etc

The problem is, with extension running outside of the process of OpenSearch. Now they dont have direct access to IndexModule object.
Taking an example: A plugin could get MergeStatsIndexModule.getEngineFactory.getMergeStats()

With extensions, they dont have access to this information. So we have to build in custom interfaces for every publicly accessible information for each extension point (IndexModule in this example).

**
 * IndexModule represents the central extension point for index level custom implementations like:
 * <ul>
 *     <li>{@link Similarity} - New {@link Similarity} implementations can be registered through
 *     {@link #addSimilarity(String, TriFunction)} while existing Providers can be referenced through Settings under the
 *     {@link IndexModule#SIMILARITY_SETTINGS_PREFIX} prefix along with the "type" value.  For example, to reference the
 *     {@link BM25Similarity}, the configuration {@code "index.similarity.my_similarity.type : "BM25"} can be used.</li>
 *      <li>{@link IndexStorePlugin.DirectoryFactory} - Custom {@link IndexStorePlugin.DirectoryFactory} instances can be registered
 *      via {@link IndexStorePlugin}</li>
 *      <li>{@link IndexEventListener} - Custom {@link IndexEventListener} instances can be registered via
 *      {@link #addIndexEventListener(IndexEventListener)}</li>
 *      <li>Settings update listener - Custom settings update listener can be registered via
 *      {@link #addSettingsUpdateConsumer(Setting, Consumer)}</li>
 * </ul>
 */

@owaiskazi19
Copy link
Member

Possible Solution to the above problem:

  1. Create an extension.yml file and pass only the extension points, its listeners and the method extension wants to invoke. Based on the request provided by yml file handle it on the OpenSearch end.
  2. Use RPC calls to invoke remote methods of OpenSearch from the extension.
  3. Use JNA to call native functions using natural Java method invocation.

@saratvemulapalli
Copy link
Member Author

Will close this issue as we were able to get IndicesModule up for Latency Tester.
Will copy the comment in a dedicated issue.

@saratvemulapalli
Copy link
Member Author

Code: saratvemulapalli@2d068df has been merged for first extension point.

@saratvemulapalli
Copy link
Member Author

PR #3107 merged to feature branch!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants