How to ingest/index existing data from database

[vicperdana]

Context / Scenario

Have an existing Postgres database to connect and ingest data from so that we can interact and ask questions via Semantic Kernel Plugin. I've used the connector successfully but looking for ways to "load" the data.

Question

Hi, is there any example on how to ingest existing dataset from database? In my case I'd like to ingest data from Postgres and ask the data via SK plugin.

This is the closest example I find but it covers documents and not existing database or schema.

[dluc]

when asking for answers, KM looks up a "Memory DB" containing memory records. Records can be stored on Azure AI Search, Qdrant, ElasticSearch, Postgres, SQL Server, etc. Records must follow a specific schema, so usually you would use KM first to ingest these records (uploading files, text, web pages). However, with Postgres, you can configure the memory DB connector to read data from your tables, as long as they provide the needed fields. Some info here: https://github.com/microsoft/kernel-memory/tree/main/extensions/Postgres

Summarizing:

• if you're using Postgres and your tables contain the needed fields, which includes a vector field, you might be able to reuse the existing tables as Memory Indexes, containing Memory Records.
• if your Postgres doesn't have the needed columns (or if you're using any other storage), you need to import the data into KM using ImportDocument/ImportText/ImportWebPage methods first - these methods will create the tables with the correct schema
• once memory records are available, Search and Ask methods will use to generate results/answers

[vicperdana]

However, with Postgres, you can configure the memory DB connector to read data from your tables, as long as they provide the needed fields.

Thanks @dluc, assuming we're using the same Postgres db server for the MemoryDb, does the data (to be indexed/read from) need to be in the same database? If not, I assume we can target by using the postgresconfig.

[dluc]

Thanks @dluc, assuming we're using the same Postgres db server for the MemoryDb, does the data (to be indexed/read from) need to be in the same database? If not, I assume we can target by using the postgresconfig.

I would look at the config class, you can manage schemas and add a prefix to tables created by KM. These should provide a way to isolate KM data from the rest.

[vicperdana]

Thanks @dluc I'll look to try this later. Closing this for now.