Support loading data into archival with Llama Index connectors

[sarahwooders]

Llama Index has implemented a variety of connectors for reading, parsing, and embedding data. This pull request integrates with Llama index to allow users to ingest more data sources into archival memory.

Data can be loaded into archival with the following command

memgpt load <CONNECTOR> --name <DATASET-NAME> [ARGS]

where supported connectors are currently directory and webpage.

Remaining to-dos before merging:

• Migrate MemGPT to query through LlamaIndex retrievers
• Add tests for loading into archival storage

[cpacker]

@sarahwooders are there example commands w/ dummy data I can test this with (eg for runtime error on my end)?

[sarahwooders]

It's not fully integrated into the CLI yet, but you can run the tests:

poetry install 
poetry run pytest -s tests

And also make sure memgpt still works OK.

Also just lmk if stylistically it seems OK before I refactor the CLI/config.

[cpacker]

@sarahwooders tests passed on my end, but I'm getting:

ModuleNotFoundError: No module named 'llama_index'

when testing backcompat with python main.py.

Should we add llama_index as a default dependency?

[sarahwooders]

Did you run poetry install and then poetry run python main.py? The poetry.lock is updated so you might need to make sure the environment has llama_index.

[cpacker]

I am a poetry scrub, I did not run with the poetry prefix

[cpacker]

Alright LGTM, backcompat looks fine. Will merge when you're back.

[cpacker]

LGTM 👩‍🍳