How to handle with multiple PDF's?

[barizon1]

Hi guys,
Just starting a new project in Flowise, and I need some help with PDFs. I want to use about 15 PDFs and want to avoid creating 15 PDF Nodes in the workflow.

What would you suggest to store it somewhere and use it in Flowise? What would be the best Node option? I tried to use the "folder with files," but got no results.

Thanks in advance!

[HenryHengZJ]

you can try selecting multiple files in one go. If you're looking to have metadata on each pdf file, the only way is to have 15 PDF Loader nodes. You can just simply duplicate them

[barizon1]

thanks for the answer. it would be possible to upload these documents directly to supabase for example? I wanted to handle many small txt files also. thanks!

[ahasibrm]

If you're looking to have metadata on each pdf file, the only way is to have 15 PDF Loader nodes. You can just simply duplicate them

Is it possible to have one loader node and invoke it sequentially, each time adjusting the metadata?

[rinaldo77]

I try now for 1 day to do exactly that. I have 1 German pdf and 1 English pdf file. As long as only one is connected to my TOOL agent it works, in both languages. As soon as both are connected I get TypeError: Cannot read property of undefined (reading 0). Any idea is very appreciated.

[rinaldo77]

I figured it out. It works with per-configured Document Store node.

[briansoegaard]

It's great you found a solution, @rinaldo77. Can you share some screendumps or your chatflow where you did this? Specifically, I'm looking for a solution where I can use the Upsert API to upsert different Docx or PDF files, but these two document loaders seems to be buggy. It's only the Text File Document Loader that works without error.
It's a showstopper for me so I have filed a bug on this: #3041

Thus, my question is if you or @barizon1 have found another method to provide this feature?

I really cross my fingers to get someone to look into this issue with the document loaders 🤞

[rinaldo77]

At the Flowise main menu you see Document Stores at the bottom. There you can create a pre-configured document store, select different doc loaders to load up different types of documents into one store. The pdf file Uploader has a bug. The file update does not work. If I had to upload an updated pdf file, I deleted the pdf uploader, selected the pdf Uploader again and then uploaded the updated file. I assume the upload works if the updated file has a different file name. It had no time to test that.
You can than drag the Document Store node in your chatflow, select the name you gave it and then it should work. I noticed that mentioning the Retriever Tool name (only small caps) in the Agents system prompt is critical.

[SamuelH77]

Related question to this. What tool have you found effective in calling the PDF documents? I was looking for the Conversational Retrieval Agent but don't see it as an option anymore. I get that same error with multiple documents and was wondering if it was the type of AI agent that was the issue.

I am curious if loading the PDFs into the Document Store in Flowise worked. I was also thinking Pinecone might be a way to store sets of documents.

Thanks!

[Crispin-Courtenay]

Pinecone has a new function called Assistant. As of release, it supports PDF and txt docs. It is brilliantly fast. Currently runs GPT or Claude.

I recommend using the paid version, as it removes many constraints, and if you are not in a production environment it is inexpensive.

https://www.pinecone.io/blog/pinecone-assistant/

[rinaldo77]

I am using all kind of agents and LLMs. I believe important is which retriever tool you are using, the correct Retriever Name and which vector store. The pre-configured Document Store works with different file types. I upsert it only in the Chat Flow.
I found Render Starter package with 1 GB disk the most cost effective solution.

[mxsteini]

I resolved the question like this:

1. I created an Documentstore with 2 loaders. Each loader sets metadata like this:
   
2. I created a retriever for each language:
   

This solution need > 24h.
Traps:

1. if the prompt is not correct, the retriever returns illegal schema
2. head up to your chunks. Check them in the preview. I had chunk with structural elements from website which returned as Top 1 in the retrieval.