Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Create a coordinating node level reader for tsdb #79197

Merged
merged 3 commits into from
Oct 15, 2021
Merged

Create a coordinating node level reader for tsdb #79197

merged 3 commits into from
Oct 15, 2021

Conversation

nik9000
Copy link
Member

@nik9000 nik9000 commented Oct 14, 2021

This creates an interface that reads data in a time series compatible
way on the coordinating node. We believe that it can one day smooth out
querying time series data at a high level.

Right now there is a single implementation of this interface that
targets standard indices very inefficiently. It delegates down to our
standard _search APIs, specifically composite, top_hits, and
search_after. It is our hope that when we have fancier TSDB support we
can use it to speed the API.

The API itself looks like:


void latestInRange(metric, from, to, callback);
void latestInRanges(metric, from, to, step, callback);
void valuesInRange(metric, from, to, callback);

@nik9000 nik9000 added v8.0.0 :StorageEngine/TSDB You know, for Metrics labels Oct 14, 2021
@nik9000
Create a coordinating node level reader for tsdb
26389ef 
This creates an interface that reads data in a time series compatible
way on the coordinating node. We believe that it can one day smooth out
querying time series data at a high level.

Right now there is a single implementation of this interface that
targets standard indices very inefficiently. It delegates down to our
standard `_search` APIs, specifically `composite`, `top_hits`, and
`search_after`. It is our hope that when we have fancier TSDB support we
can use it to speed the API.

The API itself looks like:
```
// The latest value for all time series in the range
void latestInRange(metric, from, to, callback);
// The latest value for all time series in ranges starting from
// `from`, st
void latestInRanges(metric, from, to, step, callback);
void valuesInRange(metric, from, to, callback);
```
@nik9000
Copy link
Member Author

nik9000 commented Oct 14, 2021

run elasticsearch-ci/part-1

@nik9000 nik9000 marked this pull request as ready for review October 15, 2021 13:30
@elasticmachine elasticmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Oct 15, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

@nik9000 nik9000 requested a review from imotov October 15, 2021 13:30
imotov
imotov approved these changes Oct 15, 2021
View reviewed changes
Copy link
Contributor

@imotov imotov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I think this is a good start, but since this is quite experimental I would like to confine to a more easily identifiable location. How about org.elasticsearch.timeseries.support?

server/src/internalClusterTest/java/org/elasticsearch/search/tsdb/TimeSeriesMetricsIT.java Outdated
* Side Public License, v 1.
*/

package org.elasticsearch.search.tsdb;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe org.elasticsearch.timeseries.support?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure!

@nik9000 nik9000 added the auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) label Oct 15, 2021
@nik9000 nik9000 mentioned this pull request Oct 15, 2021
Closed
@elasticsearchmachine elasticsearchmachine merged commit bd01654 into elastic:master Oct 15, 2021
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Oct 18, 2021
@weizijun
Merge branch 'upstream/master' into reuse-date-stream-timestamp
c4a4b07 
* upstream/master: (109 commits)
  Migrate custom role providers to licensed feature (elastic#79127)
  Remove stale AwaitsFix in InternalEngineTests (elastic#79323)
  Fix errors in RefreshListenersTests (elastic#79324)
  Reeable BwC Tests after elastic#79318 (elastic#79320)
  Mute BwC Tests for elastic#79318 (elastic#79319)
  Reenable BwC Tests after elastic#79308 (elastic#79313)
  Disable BwC Tests for elastic#79308 (elastic#79310)
  Adjust BWC for node-level field cap requests (elastic#79301)
  Allow total memory to be overridden (elastic#78750)
  Fix SnapshotBasedIndexRecoveryIT#testRecoveryIsCancelledAfterDeletingTheIndex (elastic#79269)
  Disable BWC tests
  Mute GeoIpDownloaderCliIT.testStartWithNoDatabases (elastic#79299)
  Add alias support to fleet search API (elastic#79285)
  Create a coordinating node level reader for tsdb (elastic#79197)
  Route documents to the correct shards in tsdb (elastic#77731)
  Inject migrate action regardless of allocate action (elastic#79090)
  Migrate to data tiers should always ensure a TIER_PREFERENCE is set (elastic#79100)
  Skip building of BWC distributions when building release artifacts (elastic#79180)
  Default ENFORCE_DEFAULT_TIER_PREFERENCE to true (elastic#79275)
  Deprecation of transient cluster settings (elastic#78794)
  ...

# Conflicts:
#	server/src/main/java/org/elasticsearch/index/IndexMode.java
#	server/src/test/java/org/elasticsearch/index/TimeSeriesModeTests.java
weizijun added a commit to weizijun/elasticsearch that referenced this pull request Oct 18, 2021
@weizijun
Merge branch 'upstream/master' into tsdb-deal-timestreamp
e3b7256 
* upstream/master: (521 commits)
  Migrate custom role providers to licensed feature (elastic#79127)
  Remove stale AwaitsFix in InternalEngineTests (elastic#79323)
  Fix errors in RefreshListenersTests (elastic#79324)
  Reeable BwC Tests after elastic#79318 (elastic#79320)
  Mute BwC Tests for elastic#79318 (elastic#79319)
  Reenable BwC Tests after elastic#79308 (elastic#79313)
  Disable BwC Tests for elastic#79308 (elastic#79310)
  Adjust BWC for node-level field cap requests (elastic#79301)
  Allow total memory to be overridden (elastic#78750)
  Fix SnapshotBasedIndexRecoveryIT#testRecoveryIsCancelledAfterDeletingTheIndex (elastic#79269)
  Disable BWC tests
  Mute GeoIpDownloaderCliIT.testStartWithNoDatabases (elastic#79299)
  Add alias support to fleet search API (elastic#79285)
  Create a coordinating node level reader for tsdb (elastic#79197)
  Route documents to the correct shards in tsdb (elastic#77731)
  Inject migrate action regardless of allocate action (elastic#79090)
  Migrate to data tiers should always ensure a TIER_PREFERENCE is set (elastic#79100)
  Skip building of BWC distributions when building release artifacts (elastic#79180)
  Default ENFORCE_DEFAULT_TIER_PREFERENCE to true (elastic#79275)
  Deprecation of transient cluster settings (elastic#78794)
  ...

# Conflicts:
#	rest-api-spec/src/yamlRestTest/resources/rest-api-spec/test/tsdb/10_settings.yml
#	server/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java
#	server/src/main/java/org/elasticsearch/common/settings/Setting.java
#	server/src/main/java/org/elasticsearch/index/IndexMode.java
#	server/src/test/java/org/elasticsearch/index/TimeSeriesModeTests.java
@wchaparro wchaparro assigned nik9000 and unassigned nik9000 Dec 16, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@imotov imotov imotov approved these changes

Assignees
No one assigned
Labels
auto-merge-without-approval Automatically merge pull request when CI checks pass (NB doesn't wait for reviews!) :StorageEngine/TSDB You know, for Metrics Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v8.0.0-beta1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@nik9000 @elasticmachine @imotov @jakelandis @elasticsearchmachine