Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enforce default date formats for tsdb data streams #83517

Closed
martijnvg opened this issue Feb 4, 2022 · 1 comment · Fixed by #84699
Closed

Enforce default date formats for tsdb data streams #83517

martijnvg opened this issue Feb 4, 2022 · 1 comment · Fixed by #84699
Assignees
Labels
:Data Management/Data streams Data streams and their lifecycles Team:Data Management Meta label for data/management team

Comments

@martijnvg
Copy link
Member

Supporting custom date formats when resolving a tsdb data stream to a backing index makes the logic on the coordinating node complicated, because the target backing index is unknown. It is only known after parsing the @timestamp field, but in order to parse the timestamp field the date format should be known and this information is stored in the mapping of a backing index.

In order to avoid unnecessary complexity when coordinating a bulk request we need to make the following changes:

  • Only support strict_date_optional_time and strict_date_optional_time_nanos date formats for the @timestamp field in tsdb data streams. For now we don't expect that we need to support other date formats. In order to do this we also need to prohibit field mapping for the @timestamp field that allow any other date format for backing indices of tsdb data streams.
  • When resolving the backing index for tsdb data streams on the coordinating node, truncate millisecond and nanosecond precision from the the timestamp only for determining to what backing index the document should be indexed into. Also the DataStreamIndexSettingsProvider should be adjusted to truncate ms/ns precision when setting the index.time_series.start_time and index.time_series.end_time index settings. Note that, on the data node when documents get indexed the timestamp do get parsed into a date value that confirms with the field mapping of the @timestamp field.

Note that currently only strict_date_optional_time and ms since epoch provided as long is supported for tsdb data streams (see this TODO).

@martijnvg martijnvg added the :Data Management/Data streams Data streams and their lifecycles label Feb 4, 2022
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Feb 4, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@martijnvg martijnvg self-assigned this Feb 4, 2022
martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Mar 7, 2022
Truncate timestamp to second precision for resolving data stream to backing index only.
Also add support for parsing timestamps in date nanos format.

Closes elastic#83517
martijnvg added a commit that referenced this issue Mar 17, 2022
Truncate timestamp to second precision for resolving data stream to backing index only.
Also add support for parsing timestamps in date nanos format.

Closes #83517
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Data streams Data streams and their lifecycles Team:Data Management Meta label for data/management team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants