Comparative analysis of indexing technologies for the Finesse project

[leejaeka]

The objective is to conduct a detailed comparison of indexing technologies. Currently, we have integrated a PostgreSQL based index as ailab-db and an Azure AI Search index as azure-db. Our next step is to explore LlamaIndex, a promising solution that combines the advantages of open-source flexibility and high performance. The goal is to implement LlamaIndex and integrate it alongside the existing technologies in finesse-backend and finesse-frontend, enabling direct, side-by-side comparisons.

Tasks

• Build an index on our PostgreSQL db using llamaindex
• Updated notebook llamaindex-db#3
• Translate Notebook Code into Standalone Packages in llamaindex-db llamaindex-db#4
• Integrate Llamaindex in finesse-backend finesse-backend#51
• Integrate LlamaIndex in finesse-frontend #79

Acceptance Criteria

• 1. llamaindex search functions and operates as expected
• 2. Well documented

[rngadam]

@leejaeka to get it to the backlog of DevSecOps, the Projects need to be set to DevSecOps. In addition, the issue should be associated to the product backlog.

We try to minimize the use of labels (don't know how devops made it there as the only one we use as of now is user-visible).

@ThomasCardin where does the devops label come from?

[ibrahim-kabir]

This is an epic, moving this to finesse-frontend