What are the file attachments for?

[rcdailey]

I tried uploading a PDF containing a vacation itinerary. I tried asking GPT to summarize it but I didn't get an english response. I'm very confused on what the purpose of the file attachments are -- I thought they were a way to provide context to GPT instead of pasting it in the text field. Can someone explain if I'm understanding, and if not, why I get the response pictured below?

[nmorton13]

I've had that happen when something was wrong with the PDF.

[rcdailey]

I used the "Print to PDF" function on Windows, and I've never really had an issue with that. The PDF also seems to open just fine. So I'm not sure what could be wrong with it.

[danny-avila]

I'm looking into this soon. In my experience, it seems to happen when the semantic search returns nothing. For example, with your file, it would convert your words "summarize this itinerary" to vector embeddings, to perform a nearest similarity search on what's within the file, which it may not find any matches for those 3 words.

[NDolensek]

I'm also confused about the file feature.

In Google AI studio, Gemini Pro 1.5 accepts uploaded files like pdfs and fits them into the chat context. Any given message in a thread can ask a new question about the file and Gemini will provide a more or less relevant answer using the information contained in the file.

I've been testing the Studio Gemini Pro 1.5 and Flash 1.5 endpoints in Librechat and the behavior seems unexpected. Only the first message in a thread with a pdf attachment will receive an answer using info from the file. My next question in the thread will receive a perplexed response from the model, requesting the file to be sent.

Am I using the feature wrong or misunderstanding it? How can I get the file functionality as in Google AI Studio, in Librechat? @danny-avila Your answer above seems to indicate that the file functionality works like RAG on the Librechat side, instead of the file being passed directly to the API endpoint? Why? Is there a way to default to non-RAG behavior?

[danny-avila]

Direct file integration with Google/Gemini is not yet supported but planned.

LC deals with many different providers who all tend to handle files differently, or outright don't support certain filetypes as valid for the request payload.

The RAG API is one such way to universalize files by working within the vector space triggered by text.

Loading files as full text into the context is also planned and will be worked on soon: #4502

[Peveld]

I'm also confused. Files seem to be put into something like a RAG vector store but are not completely loaded into the chat context. So I'm getting different - worse results in Librechat than in ChatGPT. Tried it with an excel file where I was asking questions about the data. So does anybody know a way to get file contents completely into chat?

[danny-avila]

See my comment above for further information on this: #2951 (reply in thread)