You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
tl'dr - create a common search interface, micronaut-search. I can work on creating solr clients (micronaut-search-solr) and make this one be for ES (micronaut-search-opensearch).
I would then create the following for micronaut-search:
grpc interface for searching
REST interface for search
common admin features that integrate with search
Proposal: Creating a Unified Search Interface for Micronaut
I'm proposing we develop a common search interface, micronaut-search, for Micronaut applications. This would simplify the integration of various search engines, such as Solr and Elasticsearch.
My Role:
Create common search components that work on both solr-clients and elastic-search clients. (others such as vespa pinecone etc are possible too, but ES and solr have nearly identical setups since they're both based on lucene)
What I need help with:
Solr Client: Solr client module (micronaut-search-solr). Is this a difficult ask?
Elasticsearch Client: this client, but it would have some capability to interact with a generic search module
Motivation:
Simplify Search Integration: Indexing and searching can be complex. I aim to streamline this process for Micronaut developers. Create common indexing components: Right now this is weak as hell in the market and requires ETL transform-types. I am trying to simplify that Sentence embedding integration: Unfortunately for us, almost all good embedding calculations are either in python or require JNI. We would create an interface that would bridge that (as I've already done with both openapi and grpc) Job management: I'd integrate with the job runner on micronaut to create a job manager for indexing and searching. common search interface: get search results OOTB with either search engine
Current Progress:
Indexer Development: I've already developed an indexer that simplifies the creation of vector indexes on Solr. I'm going to do the same with elastic search.
Feature Integration: I'm working on integrating features like automatic Solr document creation from Google Protocol Buffers and Tika parsing from web crawling. I already have unit tests that demonstrate this with wikipedia and tika.
Next Steps:
Generic Search Client: create a more generic search client to support both Solr and Elasticsearch.
Advanced Features: This would enable us to explore advanced features like vector and semantic search.
Request for Collaboration:
I'm seeking assistance with the client code development. If we can create a shared client, it will make the project more attractive to a wider audience.
Thoughts?
The text was updated successfully, but these errors were encountered:
Feature description
tl'dr - create a common search interface, micronaut-search. I can work on creating solr clients (micronaut-search-solr) and make this one be for ES (micronaut-search-opensearch).
I would then create the following for micronaut-search:
Proposal: Creating a Unified Search Interface for Micronaut
I'm proposing we develop a common search interface, micronaut-search, for Micronaut applications. This would simplify the integration of various search engines, such as Solr and Elasticsearch.
My Role:
Create common search components that work on both solr-clients and elastic-search clients. (others such as vespa pinecone etc are possible too, but ES and solr have nearly identical setups since they're both based on lucene)
What I need help with:
Solr Client: Solr client module (micronaut-search-solr). Is this a difficult ask?
Elasticsearch Client: this client, but it would have some capability to interact with a generic search module
Motivation:
Simplify Search Integration: Indexing and searching can be complex. I aim to streamline this process for Micronaut developers.
Create common indexing components: Right now this is weak as hell in the market and requires ETL transform-types. I am trying to simplify that
Sentence embedding integration: Unfortunately for us, almost all good embedding calculations are either in python or require JNI. We would create an interface that would bridge that (as I've already done with both openapi and grpc)
Job management: I'd integrate with the job runner on micronaut to create a job manager for indexing and searching.
common search interface: get search results OOTB with either search engine
Current Progress:
Indexer Development: I've already developed an indexer that simplifies the creation of vector indexes on Solr. I'm going to do the same with elastic search.
Feature Integration: I'm working on integrating features like automatic Solr document creation from Google Protocol Buffers and Tika parsing from web crawling. I already have unit tests that demonstrate this with wikipedia and tika.
Next Steps:
Generic Search Client: create a more generic search client to support both Solr and Elasticsearch.
Advanced Features: This would enable us to explore advanced features like vector and semantic search.
Request for Collaboration:
I'm seeking assistance with the client code development. If we can create a shared client, it will make the project more attractive to a wider audience.
Thoughts?
The text was updated successfully, but these errors were encountered: