ragKnowledge blows up prompt by retrieving entire documents.

[LinuxIsCool]

Describe the bug

ragKnowledge embeds entire documents and then retrieves them at runtime, blowing up prompts.

To Reproduce

Put a large (>1MB) text file in rag knowledge, and then message your agent with a large portion of that document so that it gets retrieved via ragKnowledge during runtime.

Expected behavior

Either don't embed entire documents or filter them out in runtime.

Additional context

There is some code duplication in ragKnowledge.ts between processFile and createKnowledge. I think those can be cleaned up and we must consdier 1. Do we need entire document embeddings? How to properly filter them out at runtime retrieval especially if they are too big?

I have a temporary fix in sqlite adapter doing this:

        let sql = `SELECT * FROM knowledge WHERE (isMain = 0) AND (agentId = ? OR isShared = 1)`;

I originally commented on this here: #1620 (comment)

[github-actions]

Hello @LinuxIsCool! Welcome to the elizaOS community. Thank you for opening your first issue; we appreciate your contribution. You are now an elizaOS contributor!