You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We should also batch requests to embedding model and vector database. One way is to move embedding&inserting into ProcessingExecutionStrategy.
For errors we should decide how to handle failed documents, for example create a default implementation gathering all failed documents into a list. User should be able to create custom error handling strategy.
Motivation
Current behaviour causes problems with rate limits / memory usage / too big requests to embedding models.
Additional context
No response
The text was updated successfully, but these errors were encountered:
Feature description
Right now ingest method flow looks like this:
We should also batch requests to embedding model and vector database. One way is to move embedding&inserting into ProcessingExecutionStrategy.
For errors we should decide how to handle failed documents, for example create a default implementation gathering all failed documents into a list. User should be able to create custom error handling strategy.
Motivation
Current behaviour causes problems with rate limits / memory usage / too big requests to embedding models.
Additional context
No response
The text was updated successfully, but these errors were encountered: