Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allows multiple patterns to be specified for index templates #21009

Merged
merged 3 commits into from
Nov 10, 2016

Conversation

a2lin
Copy link
Contributor

@a2lin a2lin commented Oct 19, 2016

Index templates now take an list for the template: field instead of a string.

For example:

{
  "index_patterns":  ["c*", "d*"],
}

Instead of the previous form

{
  "template": "c*"
}

Template metadata from older versions of ES are still openable with this code, although I don't know how important that is since it's probably assumed that you can't just open a new elasticsearch release on an old one's data directory and expect it to work.

Closes #20690

@nik9000
Copy link
Member

nik9000 commented Oct 20, 2016

@a2lin, Sorry to let this sit for a few days. I've added this to my work queue (browser tabs...) and I promise I'll get to it before too long. My queue is getting longer than I'd like these days.

if(entry.getValue() instanceof String) {
template(Collections.singletonList((String) entry.getValue()));
} else if (entry.getValue() instanceof List) {
template((List<String>) entry.getValue());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is worth doing something like ((List<?>) entry.getValue()).stream().map(Object::toString).collect(toList()) so we are sure we have strings here. As you've written it now if someone sticks [0] in the template field then we'll cram an Integer in the list and we'll get ClassCastExceptions in weird places.

} else if (entry.getValue() instanceof List) {
template((List<String>) entry.getValue());
} else {
throw new IllegalArgumentException("Malformed [template] value, should be a string[]");
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe "should be a string or a list of strings".

@@ -449,7 +458,7 @@ public void readFrom(StreamInput in) throws IOException {
super.readFrom(in);
cause = in.readString();
name = in.readString();
template = in.readString();
template = in.readList(StreamInput::readString);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK! New fun things for you to learn about. The way we support mixed version cluster (like, during a rolling restart) is by adding branches to this code. So this needs to read like:

if (in.getVersion().onOrAfter(Version.5_1_0)) {
  template = in.readList(StreamInput::readString);
} else {
  template = singletonList(in.readString());
}

And the out needs to have the same branching. And you need to find some place to cram the test for this.

The specific version number that I chose is based on my expectation of which release will be the first to see this feature.

@@ -286,7 +289,13 @@ public PutIndexTemplateRequest source(Map templateSource) {
for (Map.Entry<String, Object> entry : source.entrySet()) {
String name = entry.getKey();
if (name.equals("template")) {
template(entry.getValue().toString());
if(entry.getValue() instanceof String) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is good that you've added single string support - this is needed so we can add this feature without breaking backwards compatibility. We could chose to delay this to 6.0 and only support lists here but I don't think that is a good idea anyway.

@@ -41,7 +42,7 @@ public PutIndexTemplateRequestBuilder(ElasticsearchClient client, PutIndexTempla
/**
* Sets the template match expression that will be used to match on indices created.
*/
public PutIndexTemplateRequestBuilder setTemplate(String template) {
public PutIndexTemplateRequestBuilder setTemplate(List<String> template) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could add a new method that takes a String does a singletoneList on the input for backwards compatibility but we don't tend to do that for these builders because we expect that users of the Java API can deal with these changes on upgrade because the compiler will catch it for them. In other words: we don't respect semver in the Java API.

@@ -85,7 +87,7 @@ private Table buildTable(RestRequest request, ClusterStateResponse clusterStateR
if (patternString == null || Regex.simpleMatch(patternString, indexData.name())) {
table.startRow();
table.addCell(indexData.name());
table.addCell(indexData.getTemplate());
table.addCell(String.format(Locale.ROOT, "[%s]", String.join(", ", indexData.getTemplate())));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd just do "[" + String.join(", ", indexData.getTemplate()) + "]". I was a big fan of String.format for a while but I've fallen out of love. And the compiler is quite good at optimizing + for Strings.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do really with it were indexData.getTemplate*s*() or indexData.getPatterns() though.

@@ -383,7 +384,7 @@ public void doAfterNodes(int numNodes, Client client) throws Exception {

ClusterState state = client().admin().cluster().prepareState().execute().actionGet().getState();
assertThat(state.metaData().index("test").mapping("type2"), notNullValue());
assertThat(state.metaData().templates().get("template_1").template(), equalTo("te*"));
assertThat(state.metaData().templates().get("template_1").template(), equalTo(Collections.singletonList("te*")));
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Personally I do a static import for singletonList but I know lots of folks don't. If you want to keep it as is that is fine with me, or if you want to do static import that is fine with me.

@@ -3,8 +3,8 @@

Index templates allow you to define templates that will automatically be
applied when new indices are created. The templates include both settings and
mappings, and a simple pattern template that controls whether the template
should be applied to the new index.
mappings, and a list of simple pattern templates that controls whether the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a simple pattern or a list of simple patterns?

@@ -15,7 +15,7 @@ For example:
--------------------------------------------------
PUT _template/template_1
{
"template": "te*",
"template": ["te*"],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I might make this one ["te*", "bar*"] but then leave the others as strings rather than lists.

@@ -64,7 +64,7 @@
- match:
$body: /
(^|\n)test_2 \s+
test-2\* \s+
\[test-2\*\] \s+
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you fix the \s+ so it lines up with the \s+ above it?

@a2lin a2lin force-pushed the support_array_template_patterns branch 4 times, most recently from 88ac860 to 24b4bf5 Compare October 28, 2016 09:06
@a2lin
Copy link
Contributor Author

a2lin commented Oct 28, 2016

@nik9000 Sorry for the delayed response! I think I fixed all the points.

There's a bit of oddness left, but I can fix #18922 (rename template to index_pattern in the rest api) after this one and that should change everything.

I've added the setTemplate(String indexPattern) api back to the PutIndexTemplateRequestBuilder, with @deprecated attached, I thought it might be nice since the method got renamed. I can change it back if this is against convention; I was unsure about your earlier comment on this part :)

For the in/out tests I used:

    public void testToBytes() throws IOException{
        BytesStreamOutput output = new BytesStreamOutput();
        output.setVersion(Version.V_5_0_0_rc1);
                IndexTemplateMetaData data = IndexTemplateMetaData.builder("foo")
            .template("bar")
            .order(1)
            .settings(Settings.builder()
                .put("setting1", "value1")
                .put("setting2", "value2"))
            .putAlias(newAliasMetaDataBuilder("alias-bar1")).build();
        data.writeTo(output);
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        output.bytes().writeTo(baos);
        System.out.println(Base64.getEncoder().encodeToString(baos.toByteArray()));
    }

and a similar variant for the PutIndexTemplateRequest to generate the base64 strings used as constants.

Thanks again for reviewing!

Copy link
Member

@nik9000 nik9000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've left a few very minor things. I think this is just about ready! Thanks for being so patient with me. I keep getting distracted and forgetting about this. Sorry!

Anyway, I mention rebasing in one of my minor points. If we didn't have any remaining issues I'd say to just squash and rebase and fix those points. But since we have some open stuff I'd wait on those points until later.

}
} else if (name.equals("order")) {
order(XContentMapValues.nodeIntegerValue(entry.getValue(), order()));
} else if ("version".equals(name)) {
if ((entry.getValue() instanceof Integer) == false) {
if (!(entry.getValue() instanceof Integer)) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually prefer == false or false == of !. Well, we have one core contributor who really prefers it and the rest of us are fine with it. false == is unlikely to be misread, whereas ! is small and can get lost if you scan the code quickly. So I'd change this back. It is weird, yes, but it is our kind of weird.

@@ -47,6 +47,8 @@

public class IndexTemplateMetaData extends AbstractDiffable<IndexTemplateMetaData> {

public static final Version V_5_1_0 = Version.fromId(5010099);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

BytesArray bytes = new BytesArray(Base64.getDecoder().decode(putRequestBytes));

try (StreamInput in = bytes.streamInput()) {
in.setVersion(Version.V_5_0_0_rc1);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you rebase you can use V_5_0_0 which is more correct, I think. Both are fine, but we really expect wire compatibility with 5.0.0, not 5.0.0-rc1.

BytesArray bytes = new BytesArray(Base64.getDecoder().decode(templateBytes));

try (StreamInput in = bytes.streamInput()) {
in.setVersion(Version.V_5_0_0_rc1);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment about the version constant here.

@@ -429,7 +429,7 @@ public XContentBuilder toXContent(XContentBuilder builder, Params params) throws
IndexTemplateMetaData templateMetaData = cursor.value;
builder.startObject(templateMetaData.name());

builder.field("template", templateMetaData.template());
builder.field("template", templateMetaData.patterns());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK. I think we should make this index_patterns or patterns. We're already making a breaking change here (string -> list) so let's make the output more readable while we are here. I'm unsure if this means we can't merge this to the 5.x branch or not.... I hadn't thought this through until now. It is is a fairly out of the way section of the code....

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is all the docs I could find on this API. So it is fairly vague about the return format.

@a2lin
Copy link
Contributor Author

a2lin commented Nov 1, 2016

@nik9000 I've addressed the bit about the boolean and also the ClusterState.
edit: Was a bit too eager with the IndexTemplateMetaData; changing the 'template' there to 'index_patterns' would be a breaking REST api change, so I left that for a follow-up.

Thanks again for the reviews! I'll rebase on top of master and fix the versions (& the VersionTests thing) if these changes seem okay.

@a2lin a2lin force-pushed the support_array_template_patterns branch from de12ba5 to 77df805 Compare November 1, 2016 07:48
@nik9000
Copy link
Member

nik9000 commented Nov 1, 2016

edit: Was a bit too eager with the IndexTemplateMetaData; changing the 'template' there to 'index_patterns' would be a breaking REST api change, so I left that for a follow-up.

I think you are already breaking there because you change from a string to a list. Honestly I think you should just change it. Either we say "this has to go into 6.0 because of this" or we say "this part of the API is so deep we're willing to break it in 5.1". I don't think you can get away with not switching to a list though.

If you are willing to make the IndexTemplateMetaData change then I think it is fine to go ahead and squash and rebase to pick up the Version constant.

We should also add a note to the breaking changes docs about the IndexTemplateMetaData format but I'm unsure if it should be in the 6.0 or 5.1 docs.

@a2lin a2lin force-pushed the support_array_template_patterns branch from 77df805 to 962dbb6 Compare November 3, 2016 09:36
@a2lin
Copy link
Contributor Author

a2lin commented Nov 3, 2016

@nik9000 Okay, I rebased and fixed things! (This change solves #18922 as well now since the metadata causes index_patterns to be exposed on GET).

A summary of the changes:

For the put_template api:

curl -XPUT 'localhost:9200/_template/template_1?pretty' -d'
{
  "index_patterns": ["zzz*"],
  "order": 1,
  "version": 1,
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "type1": {
      "_source": {
        "enabled": false
      },
      "properties": {
        "field1": {
            "type": "text",
            "store": "false"
        },
        "field2": {
            "type": "keyword",
            "store": "true"
        }
      }
    }
  }
}'

put_template also works with (in the string case)

"index_patterns": "zzz*"

and also with

"template": "zzz*"

giving the same get_template in each case. Thus, the legacy template field only takes a string, and index_patterns takes both string and array.

Directly specifying the parameters for put_template also works:
curl -XPUT 'localhost:9200/_template/template_2?index_patterns=zzz*,eee*&order=5' -d {}
The legacy parameter template still works as well:
curl -XPUT 'localhost:9200/_template/template_3?template=zzz*&order=5' -d {}

Support for template (at least as part of the source for PutIndexTemplateRequest) seems to be necessary to pass the testBeatsTemplatesBWC and testLogstashTemplatesBWC tests.

Performing a get_template operation for any of the above should return:

{
  "template_1": {
    "order": 1,
    "version": 1,
    "index_patterns": [
      "zzz*"
    ],
    "settings": {
      "index": {
        "number_of_shards": "1"
      }
    },
    "mappings": {
      "type1": {
        "_source": {
          "enabled": false
        },
        "properties": {
          "field1": {
            "store": "false",
            "type": "text"
          },
          "field2": {
            "store": "true",
            "type": "keyword"
          }
        }
      }
    },
    "aliases": {}
  }
}

The cat_templates api returns:

name       index_patterns order version
template_1 [zzz*]         1     1

and for multiple patterns:

name       index_patterns order version
template_3 [zzz*]         5
template_6 [zzz*, eee*]   1     1

As for the Java PutIndexRequestBuilder API:
setTemplate(String) still remains (but marked with deprecated), but there is a new method
setPatterns(List<String>) for future use.

Going to and from pre-(this patch):
From 5.0.0 towards 5.1.0, legacy 'template' fields in the metadata are upgraded to a list and placed as singletonLists under 'index_patterns'.

If you do a backwards roll from 5.1.0 back to 5.0.0, all of the template patterns will be seen as null (as they have been rewritten in the metadata to use the index_patterns field instead of the template field). A follow-up forward roll will not fix the issue.
Therefore, this change is only be safe in the roll-forward direction.

Sorry for the essay, and thanks again for all the reviews and comments!

@a2lin a2lin force-pushed the support_array_template_patterns branch from 962dbb6 to 512da39 Compare November 3, 2016 09:50
index template, and rename the field from 'template' to 'index_pattern'.

Closes elastic#20690
@a2lin a2lin force-pushed the support_array_template_patterns branch from 512da39 to ca18747 Compare November 3, 2016 09:58
@clintongormley
Copy link
Contributor

Hi @a2lin

Thanks for all of the effort you've put into this. It'd be good to have some deprecation logging when the user specifies template instead of index_pattern.

@nik9000
Copy link
Member

nik9000 commented Nov 3, 2016

It'd be good to have some deprecation logging when the user specifies template instead of index_pattern.

See ParseField#withAllDeprecated.

@nik9000
Copy link
Member

nik9000 commented Nov 3, 2016

If you do a backwards roll from 5.1.0 back to 5.0.0,

So so so so much stuff is likely to break if you try to downgrade.

On the other hand, in a mixed version cluster you might get this behavior you describe if you query a node on the old version.

@nik9000
Copy link
Member

nik9000 commented Nov 3, 2016

@clintongormley what is your opinion on the target branch for this? I'd love the change to go into 5.x but it breaks backwards compatibility on the return for _cat/templates and _cluster/state, renaming the template field to index_patterns and making it a list. I figured since we were already changing from a string to a list then the rename is worth doing too. I certainly see good arguments for targeting this for release with 5.1 and for targeting it as 6.0 only.

Copy link
Member

@nik9000 nik9000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left two super minor things and @clintongormley left one a request to add deprecation logging when you use the old parameter.

@@ -38,11 +40,16 @@ public PutIndexTemplateRequestBuilder(ElasticsearchClient client, PutIndexTempla
super(client, action, new PutIndexTemplateRequest(name));
}

@Deprecated
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you copy the javadoc on the method below to this method as well? Including the @Deprecate javadoc annotation telling folks to use #setPatterns instead?

}
} else if (token.isValue()) {
// This is for roll-forward bwc with (#21009)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd change the comment to something like "prior to 5.1 Elasticsearch only supported a single index pattern and called it template."

@a2lin
Copy link
Contributor Author

a2lin commented Nov 4, 2016

@nik9000 I've completed the changes. I found it easier to just use DeprecationLogger directly, though. I'll leave swapping the ones in the source and fromXContent methods out for ParseField#withAllDeprecated for a follow-up (converting these to ObjectParser)?

Copy link
Member

@nik9000 nik9000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we have some failing documentation tests. You should be able to reproduce with gradle docs:check. Can you fix them?

@nik9000
Copy link
Member

nik9000 commented Nov 7, 2016

Can you fix them?

Debugging docs tests can be a bit of a trip. Have a look in docs/README.asciidoc for some hints on what is going on.

@nik9000
Copy link
Member

nik9000 commented Nov 7, 2016

I think that the docs failures were caused by the deprecation logging. Deprecation logging causes Warning HTTP headers to be returned with the response. The tests fail unless those headers match exactly what is declared in the text. I added that failing a while back so that if we deprecated something then the build would fail and we'd either convert the snippets to the new syntax or we'd mark the snippets as being deprecated. The build failure forces us to look at the document. We were missing a lot of things and this helped....

Anyway, I think you can fix these failures by using the new name.

@a2lin
Copy link
Contributor Author

a2lin commented Nov 7, 2016

@nik9000 Sorry about that :(. gradle docs:check passes for me on this new version.

@nik9000 nik9000 removed the review label Nov 10, 2016
@nik9000
Copy link
Member

nik9000 commented Nov 10, 2016

@nik9000 Sorry about that :(. gradle docs:check passes for me on this new version.

No problem! It is sneaky. I'm retesting one last time locally to be super ultra paranoid but everything is looking good so far.

@nik9000 nik9000 merged commit 0219a21 into elastic:master Nov 10, 2016
@nik9000
Copy link
Member

nik9000 commented Nov 10, 2016

Merged! Thanks again @a2lin!

@nik9000
Copy link
Member

nik9000 commented Nov 11, 2016

I still owe a breaking changes doc patch for this. I'll write one in the morning.

@a2lin
Copy link
Contributor Author

a2lin commented Nov 11, 2016

Thanks for all the help and reviews!

@nik9000
Copy link
Member

nik9000 commented Nov 11, 2016

Breaking changes docs: 7c38867

jasontedor added a commit that referenced this pull request Nov 11, 2016
* master:
  ShardActiveResponseHandler shouldn't hold to an entire cluster state
  Ensures cleanup of temporary index-* generational blobs during snapshotting (#21469)
  Remove (again) test uses of onModule (#21414)
  [TEST] Add assertBusy when checking for pending operation counter after tests
  Revert "Add trace logging when aquiring and releasing operation locks for replication requests"
  Allows multiple patterns to be specified for index templates (#21009)
  [TEST] fixes rebalance single shard check as it isn't guaranteed that a rebalance makes sense and the method only tests if rebalance is allowed
  Document _reindex with random_score
@m9aertner
Copy link
Contributor

Sorry for asking (and not changing this myself via pull request), it would be great to add a statement to the docs just what that "pattern" can be. I was hoping for a "regexp", but it looks like it is in fact a "glob". Most examples out there suggest it might be a "suffix asterisk" only, Thanks.

@@ -286,7 +296,20 @@ public PutIndexTemplateRequest source(Map templateSource) {
for (Map.Entry<String, Object> entry : source.entrySet()) {
String name = entry.getKey();
if (name.equals("template")) {
template(entry.getValue().toString());
// This is needed to allow for bwc (beats, logstash) with pre-5.0 templates (#21009)
Copy link
Contributor

@ruflin ruflin Dec 6, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nik9000 I assume you mean here pre-6.0 templates. Or will there be support for indext-patterns in the 5.x releases?

exekias added a commit to exekias/beats that referenced this pull request Apr 20, 2017
The change was introduced in
elastic/elasticsearch#21009 and deprecates
old `template` field.

This change uses the new `index_patterns` field and keeps
compatibility with `template` for older version
ruflin pushed a commit to elastic/beats that referenced this pull request Apr 20, 2017
The change was introduced in
elastic/elasticsearch#21009 and deprecates
old `template` field.

This change uses the new `index_patterns` field and keeps
compatibility with `template` for older version
@clintongormley clintongormley added :Data Management/Indices APIs APIs to create and manage indices and templates and removed :Index Templates labels Feb 13, 2018
jtibshirani added a commit that referenced this pull request Jan 10, 2020
The `template` field was deprecated in 6.0 in favor of `index_patterns`, and can
now be removed.

Relates to #21009.
SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this pull request Jan 23, 2020
The `template` field was deprecated in 6.0 in favor of `index_patterns`, and can
now be removed.

Relates to elastic#21009.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>breaking :Data Management/Indices APIs APIs to create and manage indices and templates v6.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants