Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize tsdb data stream timestamp parsing if ingest pipeline is used #84046

Closed
martijnvg opened this issue Feb 16, 2022 · 2 comments
Closed
Assignees
Labels
:Data Management/Data streams Data streams and their lifecycles >enhancement Team:Data Management Meta label for data/management team

Comments

@martijnvg
Copy link
Member

For tsdb data streams, the @timestamp field is used to determine to which backing index a data stream resolves to.

Currently in IndexAbstraction.DataStream#getWriteIndex(...) the source of the document is parsed in order to extract the @timestamp field. If an ingest pipeline is active then this parsing is redundant, because ingest parse the source of a document to into a map of maps in order for a pipeline to process it.

After the final pipeline has executed, ingest can extract the @timestamp field and attach it as String/Long to the IndexRequest class as a field. Then IndexAbstraction.DataStream#getWriteIndex(...) can check for whether that field has been set and if so use that parse it into a Instant instance.

@martijnvg martijnvg added >enhancement :Data Management/Data streams Data streams and their lifecycles labels Feb 16, 2022
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Feb 16, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@martijnvg
Copy link
Member Author

Implemented via #85387

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Data streams Data streams and their lifecycles >enhancement Team:Data Management Meta label for data/management team
Projects
None yet
Development

No branches or pull requests

3 participants