Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Push API - Add mechanism to check status #1442

Closed
PSeitz opened this issue May 13, 2022 · 2 comments
Closed

Push API - Add mechanism to check status #1442

PSeitz opened this issue May 13, 2022 · 2 comments
Labels
enhancement New feature or request

Comments

@PSeitz
Copy link
Contributor

PSeitz commented May 13, 2022

Issue Outline

The Push API currently returns the number of docs received, but this does not mean the documents are indexed, they are merely stored in the rocks-db.

It could be that the documents had a wrong format or other errors. Currently there is no way for the user to know the status of committed documents, except checking the server logs or check if they appear in search.

Async API design

The Push API is essential a async API. A typical pattern for async APIs is to return an id handle to the user with status code 202 when the documents have been securely stored. This handle can then be used to check the actual status of the submitted documents.

The endpoint to check could be a plain http returning some status in json, or websocket-like server sent events.

Async API pattern: https://docs.microsoft.com/en-us/azure/architecture/patterns/async-request-reply

@guilload
Copy link
Member

guilload commented Jun 27, 2022

The async API is nice but may be hard to use in practice without a native Quickwit client or support from log collectors. I'm in favor of a sync API with throttling:

  1. ensure index exists
  2. parse docs
  3. ensure docs match index schema
  4. store tantivy docs in WAL
  5. ack or (partially or fully) reject the batch

Things get tricky when updating index schemas. I have a few ideas in mind but this is probably worthy of a design doc.

@guilload
Copy link
Member

See #3743 insead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants