Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add time series related information to get data stream API #86395

Conversation

martijnvg
Copy link
Member

In case if a data stream is a time series data stream then include time series information.
This includes the continuous temporal ranges a time series data stream encapsulates.
This is computed based on combing the index.time_series.start_time and index.time_series.end_time
ranges of all backing indices of a time series data stream

Closes #83518

In case if a data stream is a time series data stream then include time series information.
This includes the continuous temporal ranges a time series data stream encapsulates.
This is computed based on combing the index.time_series.start_time and index.time_series.end_time
ranges of all backing indices of a time series data stream

Closes elastic#83518
@martijnvg martijnvg added the :Data Management/Data streams Data streams and their lifecycles label May 3, 2022
@martijnvg martijnvg marked this pull request as ready for review May 4, 2022 12:15
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label May 4, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@martijnvg
Copy link
Member Author

martijnvg commented May 4, 2022

This change add time_series snippet to get data streams api response if a data stream is a time series data stream.
Example GET /_data_stream run:

{
    "data_streams": [
        {
            "name": "k8s",
            "timestamp_field": {
                "name": "@timestamp"
            },
            "indices": [
                {
                    "index_name": ".ds-k8s-2022.05.04-000001",
                    "index_uuid": "1C1g7at-RW6vz0kHEGx1ZA"
                },
                {
                    "index_name": ".ds-k8s-2022.05.04-000002",
                    "index_uuid": "nuh8WMrLROmrzkRPb1fu1w"
                }
            ],
            "generation": 2,
            "status": "GREEN",
            "template": "k8s-template",
            "ilm_policy": "7-days-1gb-rollover",
            "hidden": false,
            "system": false,
            "allow_custom_routing": false,
            "replicated": false,
            "time_series": {
                "temporal_ranges": [
                    {
                        "start": "2022-05-04T12:14:53.000Z",
                        "end": "2022-05-04T12:43:26.000Z"
                    }
                ]
            }
        }
    ]
}

@sethmlarson sethmlarson added the Team:Clients Meta label for clients team label May 4, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/clients-team (Team:Clients)

@martijnvg martijnvg requested a review from nik9000 May 4, 2022 12:43
Copy link
Contributor

@sethmlarson sethmlarson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM from an API perspective

@martijnvg
Copy link
Member Author

LGTM from an API perspective

Thanks @sethmlarson for reviewing!

if (dataStream.getIndexMode() == IndexMode.TIME_SERIES) {
List<Tuple<Instant, Instant>> ranges = new ArrayList<>();
Tuple<Instant, Instant> current = null;
for (Index index : dataStream.getIndices()) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you fail if the indices aren't in time series order? Or warn or something? Just out of paranoia.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or sort them by start time.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The indices are in order in which they are created. (new data stream, rollover, rollover again -> 3 backing indices that are in order). So the start / end times should be in order.

I will add an assertion and ensure a warning log gets printed (if this ever should occur in production). Which I don't think should be the case.

@martijnvg martijnvg requested a review from nik9000 May 4, 2022 18:19
@martijnvg
Copy link
Member Author

@elasticmachine update branch

@martijnvg martijnvg merged commit f14da48 into elastic:master May 9, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Data streams Data streams and their lifecycles >non-issue Team:Clients Meta label for clients team Team:Data Management Meta label for data/management team v8.3.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Include time series temporal ranges to get data stream api for data streams in time series index mode
5 participants