Add RAG for code, personal files, and conversations

[ErikBjare]

Source files could be:

• your projects folder (with code)
• your notes (in plaintext)
• past conversations
• previously web-retrieved documents

Adding this feature would return gptme to its roots: an agent which has context about you and your projects, to intelligently and effectively assist in your work.

Needs

• Configure where to look for source files
  • probably in config
  • could index the current working directory
    • would be problematic for large files, limit to git repos?
• Exclude functionality
  • exclude binary files
  • respect git-ignores
• Retrieval method
• indexing?

Prompting

The model should be aware of its RAG abilities in the prompt (possibly listing source folder config), and provided a tool for using it.

Inspiration

• Aider's code map
• PrivateGPT
• LocalGPT
• https://www.anthropic.com/news/contextual-retrieval
  • https://github.com/anthropics/anthropic-cookbook/blob/main/skills/contextual-embeddings/guide.ipynb

[robingenz]

Thank you so much for this great project! I'm really looking forward to this feature. If you want you can start a pool funding (e.g. with Polar). I would be happy to contribute.