Skip to content

Commit 791b9b1

Browse files
polyfractalpolyfractal
authored
[Rollup] Add new capabilities endpoint for concrete rollup indices (#30401)
This introduces a new GetRollupIndexCaps API which allows the user to retrieve rollup capabilities of a specific rollup index (or index pattern). This is distinct from the existing RollupCaps endpoint. - Multiple jobs can be stored in multiple indices and point to a single target data index pattern (logstash-*). The existing API finds capabilities/config of all jobs matching that data index pattern. - One rollup index can hold data from multiple jobs, targeting multiple data index patterns. This new API finds the capabilities based on the concrete rollup indices.
1 parent 14d7e2c commit 791b9b1

File tree

12 files changed

+1066
-21
lines changed

12 files changed

+1066
-21
lines changed

‎x-pack/docs/en/rest-api/rollup/rollup-index-caps.asciidoc‎

Lines changed: 161 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,161 @@
1+
[role="xpack"]
2+
[[rollup-get-rollup-index-caps]]
3+
=== Get Rollup Index Capabilities
4+
++++
5+
<titleabbrev>Get Rollup Index Caps</titleabbrev>
6+
++++
7+
8+
experimental[]
9+
10+
This API returns the rollup capabilities of all jobs inside of a rollup index (e.g. the index where rollup data is stored).
11+
A single rollup index may store the data for multiple rollup jobs, and may have a variety of capabilities depending on those jobs.
12+
13+
This API will allow you to determine:
14+
15+
1. What jobs are stored in an index (or indices specified via a pattern)?
16+
2. What target indices were rolled up, what fields were used in those rollups and what aggregations can be performed on each job?
17+
18+
==== Request
19+
20+
`GET {index}/_xpack/rollup/data`
21+
22+
//===== Description
23+
24+
==== Path Parameters
25+
26+
`index`::
27+
(string) Index or index-pattern of concrete rollup indices to check for capabilities.
28+
29+
30+
31+
==== Request Body
32+
33+
There is no request body for the Get Jobs API.
34+
35+
==== Authorization
36+
37+
You must have `monitor`, `monitor_rollup`, `manage` or `manage_rollup` cluster privileges to use this API.
38+
For more information, see
39+
{xpack-ref}/security-privileges.html[Security Privileges].
40+
41+
==== Examples
42+
43+
Imagine we have an index named `sensor-1` full of raw data. We know that the data will grow over time, so there
44+
will be a `sensor-2`, `sensor-3`, etc. Let's create a Rollup job, which stores it's data in `sensor_rollup`:
45+
46+
[source,js]
47+
--------------------------------------------------
48+
PUT _xpack/rollup/job/sensor
49+
{
50+
"index_pattern": "sensor-*",
51+
"rollup_index": "sensor_rollup",
52+
"cron": "*/30 * * * * ?",
53+
"page_size" :1000,
54+
"groups" : {
55+
"date_histogram": {
56+
"field": "timestamp",
57+
"interval": "1h",
58+
"delay": "7d"
59+
},
60+
"terms": {
61+
"fields": ["node"]
62+
}
63+
},
64+
"metrics": [
65+
{
66+
"field": "temperature",
67+
"metrics": ["min", "max", "sum"]
68+
},
69+
{
70+
"field": "voltage",
71+
"metrics": ["avg"]
72+
}
73+
]
74+
}
75+
--------------------------------------------------
76+
// CONSOLE
77+
// TEST[setup:sensor_index]
78+
79+
If at a later date, we'd like to determine what jobs and capabilities were stored in the `sensor_rollup` index, we can use the Get Rollup
80+
Index API:
81+
82+
[source,js]
83+
--------------------------------------------------
84+
GET /sensor_rollup/_xpack/rollup/data
85+
--------------------------------------------------
86+
// CONSOLE
87+
// TEST[continued]
88+
89+
Note how we are requesting the concrete rollup index name (`sensor_rollup`) as the first part of the URL.
90+
This will yield the following response:
91+
92+
[source,js]
93+
----
94+
{
95+
"sensor_rollup" : {
96+
"rollup_jobs" : [
97+
{
98+
"job_id" : "sensor",
99+
"rollup_index" : "sensor_rollup",
100+
"index_pattern" : "sensor-*",
101+
"fields" : {
102+
"node" : [
103+
{
104+
"agg" : "terms"
105+
}
106+
],
107+
"temperature" : [
108+
{
109+
"agg" : "min"
110+
},
111+
{
112+
"agg" : "max"
113+
},
114+
{
115+
"agg" : "sum"
116+
}
117+
],
118+
"timestamp" : [
119+
{
120+
"agg" : "date_histogram",
121+
"time_zone" : "UTC",
122+
"interval" : "1h",
123+
"delay": "7d"
124+
}
125+
],
126+
"voltage" : [
127+
{
128+
"agg" : "avg"
129+
}
130+
]
131+
}
132+
}
133+
]
134+
}
135+
}
136+
----
137+
// TESTRESPONSE
138+
139+
140+
The response that is returned contains information that is similar to the original Rollup configuration, but formatted
141+
differently. First, there are some house-keeping details: the Rollup job's ID, the index that holds the rolled data,
142+
the index pattern that the job was targeting.
143+
144+
Next it shows a list of fields that contain data eligible for rollup searches. Here we see four fields: `node`, `temperature`,
145+
`timestamp` and `voltage`. Each of these fields list the aggregations that are possible. For example, you can use a min, max
146+
or sum aggregation on the `temperature` field, but only a `date_histogram` on `timestamp`.
147+
148+
Note that the `rollup_jobs` element is an array; there can be multiple, independent jobs configured for a single index
149+
or index pattern. Each of these jobs may have different configurations, so the API returns a list of all the various
150+
configurations available.
151+
152+
153+
Like other APIs that interact with indices, you can specify index patterns instead of explicit indices:
154+
155+
[source,js]
156+
--------------------------------------------------
157+
GET /*_rollup/_xpack/rollup/data
158+
--------------------------------------------------
159+
// CONSOLE
160+
// TEST[continued]
161+

‎x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rollup/action/GetRollupCapsAction.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,7 @@ public void writeTo(StreamOutput out) throws IOException {
139139
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
140140
builder.startObject();
141141
for (Map.Entry<String, RollableIndexCaps> entry : jobs.entrySet()) {
142-
entry.getValue().toXContent(builder, params);
142+
entry.getValue().toXContent(builder, params);
143143
}
144144
builder.endObject();
145145
return builder;

‎x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/rollup/action/GetRollupIndexCapsAction.java‎

Lines changed: 195 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,195 @@
1+
/*
2+
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
3+
* or more contributor license agreements. Licensed under the Elastic License;
4+
* you may not use this file except in compliance with the Elastic License.
5+
*/
6+
package org.elasticsearch.xpack.core.rollup.action;
7+
8+
9+
import org.elasticsearch.action.Action;
10+
import org.elasticsearch.action.ActionRequest;
11+
import org.elasticsearch.action.ActionRequestBuilder;
12+
import org.elasticsearch.action.ActionRequestValidationException;
13+
import org.elasticsearch.action.ActionResponse;
14+
import org.elasticsearch.action.IndicesRequest;
15+
import org.elasticsearch.action.support.IndicesOptions;
16+
import org.elasticsearch.client.ElasticsearchClient;
17+
import org.elasticsearch.common.ParseField;
18+
import org.elasticsearch.common.Strings;
19+
import org.elasticsearch.common.io.stream.StreamInput;
20+
import org.elasticsearch.common.io.stream.StreamOutput;
21+
import org.elasticsearch.common.io.stream.Writeable;
22+
import org.elasticsearch.common.xcontent.ToXContent;
23+
import org.elasticsearch.common.xcontent.ToXContentObject;
24+
import org.elasticsearch.common.xcontent.XContentBuilder;
25+
import org.elasticsearch.xpack.core.rollup.RollupField;
26+
27+
import java.io.IOException;
28+
import java.util.Arrays;
29+
import java.util.Collections;
30+
import java.util.Map;
31+
import java.util.Objects;
32+
33+
public class GetRollupIndexCapsAction extends Action<GetRollupIndexCapsAction.Response> {
34+
35+
public static final GetRollupIndexCapsAction INSTANCE = new GetRollupIndexCapsAction();
36+
public static final String NAME = "indices:data/read/xpack/rollup/get/index/caps";
37+
public static final ParseField CONFIG = new ParseField("config");
38+
public static final ParseField STATUS = new ParseField("status");
39+
private static final ParseField INDICES_OPTIONS = new ParseField("indices_options");
40+
41+
private GetRollupIndexCapsAction() {
42+
super(NAME);
43+
}
44+
45+
@Override
46+
public Response newResponse() {
47+
return new Response();
48+
}
49+
50+
public static class Request extends ActionRequest implements IndicesRequest.Replaceable, ToXContent {
51+
private String[] indices;
52+
private IndicesOptions options;
53+
54+
public Request(String[] indices) {
55+
this(indices, IndicesOptions.STRICT_EXPAND_OPEN_FORBID_CLOSED);
56+
}
57+
58+
public Request(String[] indices, IndicesOptions options) {
59+
this.indices = indices;
60+
this.options = options;
61+
}
62+
63+
public Request() {}
64+
65+
@Override
66+
public IndicesOptions indicesOptions() {
67+
return options;
68+
}
69+
70+
@Override
71+
public String[] indices() {
72+
return indices;
73+
}
74+
75+
@Override
76+
public IndicesRequest indices(String... indices) {
77+
Objects.requireNonNull(indices, "indices must not be null");
78+
for (String index : indices) {
79+
Objects.requireNonNull(index, "index must not be null");
80+
}
81+
this.indices = indices;
82+
return this;
83+
}
84+
85+
@Override
86+
public void readFrom(StreamInput in) throws IOException {
87+
super.readFrom(in);
88+
this.indices = in.readStringArray();
89+
this.options = IndicesOptions.readIndicesOptions(in);
90+
}
91+
92+
@Override
93+
public void writeTo(StreamOutput out) throws IOException {
94+
super.writeTo(out);
95+
out.writeStringArray(indices);
96+
options.writeIndicesOptions(out);
97+
}
98+
99+
@Override
100+
public ActionRequestValidationException validate() {
101+
return null;
102+
}
103+
104+
@Override
105+
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
106+
builder.array(RollupField.ID.getPreferredName(), indices);
107+
builder.field(INDICES_OPTIONS.getPreferredName(), options);
108+
return builder;
109+
}
110+
111+
@Override
112+
public int hashCode() {
113+
return Objects.hash(Arrays.hashCode(indices), options);
114+
}
115+
116+
@Override
117+
public boolean equals(Object obj) {
118+
if (obj == null) {
119+
return false;
120+
}
121+
if (getClass() != obj.getClass()) {
122+
return false;
123+
}
124+
Request other = (Request) obj;
125+
return Arrays.equals(indices, other.indices)
126+
&& Objects.equals(options, other.options);
127+
}
128+
}
129+
130+
public static class RequestBuilder extends ActionRequestBuilder<Request, Response> {
131+
132+
protected RequestBuilder(ElasticsearchClient client, GetRollupIndexCapsAction action) {
133+
super(client, action, new Request());
134+
}
135+
}
136+
137+
public static class Response extends ActionResponse implements Writeable, ToXContentObject {
138+
139+
private Map<String, RollableIndexCaps> jobs = Collections.emptyMap();
140+
141+
public Response() {
142+
143+
}
144+
145+
public Response(Map<String, RollableIndexCaps> jobs) {
146+
this.jobs = Objects.requireNonNull(jobs);
147+
}
148+
149+
Response(StreamInput in) throws IOException {
150+
jobs = in.readMap(StreamInput::readString, RollableIndexCaps::new);
151+
}
152+
153+
public Map<String, RollableIndexCaps> getJobs() {
154+
return jobs;
155+
}
156+
157+
@Override
158+
public void writeTo(StreamOutput out) throws IOException {
159+
super.writeTo(out);
160+
out.writeMap(jobs, StreamOutput::writeString, (out1, value) -> value.writeTo(out1));
161+
}
162+
163+
@Override
164+
public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
165+
builder.startObject();
166+
for (Map.Entry<String, RollableIndexCaps> entry : jobs.entrySet()) {
167+
entry.getValue().toXContent(builder, params);
168+
}
169+
builder.endObject();
170+
return builder;
171+
}
172+
173+
@Override
174+
public int hashCode() {
175+
return Objects.hash(jobs);
176+
}
177+
178+
@Override
179+
public boolean equals(Object obj) {
180+
if (obj == null) {
181+
return false;
182+
}
183+
if (getClass() != obj.getClass()) {
184+
return false;
185+
}
186+
Response other = (Response) obj;
187+
return Objects.equals(jobs, other.jobs);
188+
}
189+
190+
@Override
191+
public final String toString() {
192+
return Strings.toString(this);
193+
}
194+
}
195+
}

0 commit comments

Comments
 (0)