You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Push API currently returns the number of docs received, but this does not mean the documents are indexed, they are merely stored in the rocks-db.
It could be that the documents had a wrong format or other errors. Currently there is no way for the user to know the status of committed documents, except checking the server logs or check if they appear in search.
Async API design
The Push API is essential a async API. A typical pattern for async APIs is to return an id handle to the user with status code 202 when the documents have been securely stored. This handle can then be used to check the actual status of the submitted documents.
The endpoint to check could be a plain http returning some status in json, or websocket-like server sent events.
The async API is nice but may be hard to use in practice without a native Quickwit client or support from log collectors. I'm in favor of a sync API with throttling:
ensure index exists
parse docs
ensure docs match index schema
store tantivy docs in WAL
ack or (partially or fully) reject the batch
Things get tricky when updating index schemas. I have a few ideas in mind but this is probably worthy of a design doc.
Issue Outline
The Push API currently returns the number of docs received, but this does not mean the documents are indexed, they are merely stored in the rocks-db.
It could be that the documents had a wrong format or other errors. Currently there is no way for the user to know the status of committed documents, except checking the server logs or check if they appear in search.
Async API design
The Push API is essential a async API. A typical pattern for async APIs is to return an id handle to the user with status code 202 when the documents have been securely stored. This handle can then be used to check the actual status of the submitted documents.
The endpoint to check could be a plain http returning some status in json, or websocket-like server sent events.
Async API pattern: https://docs.microsoft.com/en-us/azure/architecture/patterns/async-request-reply
The text was updated successfully, but these errors were encountered: