[Bug]:RAG reports an error when the attached document is very large

[Passerby1011]

What happened?

I uploaded a very large txt document and RAG reported an error
Here are the logs, my api backend is load balanced between openai and azure
《西游记》.txt

Steps to Reproduce

Normal document upload
System performs search
Error reported

What browsers are you seeing the problem on?

Chrome

Relevant log output

2024-07-15 15:19:23,326 - uvicorn.error - INFO - Started server process [1]
2024-07-15 15:19:23,327 - uvicorn.error - INFO - Waiting for application startup.
2024-07-15 15:19:23,327 - uvicorn.error - INFO - Application startup complete.
2024-07-15 15:19:23,328 - uvicorn.error - INFO - Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)
2024-07-15 15:21:16,854 - httpx - INFO - HTTP Request: POST http://192.168.31.25:30000/v1/embeddings "HTTP/1.1 429 Too Many Requests"
2024-07-15 15:21:16,855 - openai._base_client - INFO - Retrying request to /embeddings in 0.963344 seconds
2024-07-15 15:21:26,191 - httpx - INFO - HTTP Request: POST http://192.168.31.25:30000/v1/embeddings "HTTP/1.1 429 Too Many Requests"
2024-07-15 15:21:26,192 - openai._base_client - INFO - Retrying request to /embeddings in 1.852032 seconds
2024-07-15 15:21:34,927 - httpx - INFO - HTTP Request: POST http://192.168.31.25:30000/v1/embeddings "HTTP/1.1 429 Too Many Requests"

Screenshots

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[berry-13]

it's hard to debug because it worked during my initial tests, but now it doesn't work for me either

[Passerby1011]

When the document is small, my RAG can run normally, but when using the document I provided, it throws an error. I suspect this is because the document is too large, with over seven hundred thousand Chinese characters

[Passerby1011]

I have found another file where the same situation occurs, it is a saved web page file, the original web page: https://github.com/ShiArthur03/ShiArthur03
File:
ShiArthur03_ShiArthur03.zip

[jumpei-tsutsui]

@berry-13 @danny-avila
I encountered the same problem in my environment. I am using Azure OpenAI's text-embedding-3-small model for RAG embeddings. I believe this issue occurs because the embedding model is being called with input that exceeds its token limit. Below is the error message I encountered:

ERROR - Error code: 429 - {'error': {'code': '429', 'message': 'Requests to the Embeddings_Create Operation under Azure OpenAI API version 2024-06-01 have exceeded call rate limit of your current OpenAI S0 pricing tier. Please retry after 86400 seconds. Please go here: https://aka.ms/oai/quotaincrease if you would like to further increase the default rate limit.'}}

It would be very helpful if the system could return an explicit error message indicating that the input text is too long due to token limits. Would it be possible to implement such a fix?