Enforce default date formats for tsdb data streams #83517
Labels
:Data Management/Data streams
Data streams and their lifecycles
Team:Data Management
Meta label for data/management team
Supporting custom date formats when resolving a tsdb data stream to a backing index makes the logic on the coordinating node complicated, because the target backing index is unknown. It is only known after parsing the
@timestamp
field, but in order to parse the timestamp field the date format should be known and this information is stored in the mapping of a backing index.In order to avoid unnecessary complexity when coordinating a bulk request we need to make the following changes:
strict_date_optional_time
andstrict_date_optional_time_nanos
date formats for the@timestamp
field in tsdb data streams. For now we don't expect that we need to support other date formats. In order to do this we also need to prohibit field mapping for the@timestamp
field that allow any other date format for backing indices of tsdb data streams.DataStreamIndexSettingsProvider
should be adjusted to truncate ms/ns precision when setting theindex.time_series.start_time
andindex.time_series.end_time
index settings. Note that, on the data node when documents get indexed the timestamp do get parsed into a date value that confirms with the field mapping of the@timestamp
field.Note that currently only
strict_date_optional_time
and ms since epoch provided as long is supported for tsdb data streams (see this TODO).The text was updated successfully, but these errors were encountered: