Put Lifecycle method in ILM Client causes 400 Bad Request

[raynigon]

The PUT Lifecycle Endpoint send the name as a property in the body, which is not accepted by Elasticsearch.
This causes a 400 Bad Request Response.
Therefore this method is not usable.

Code

Application

client.ilm().putLifecycle(it -> {
    it.name(policy.name())
    it.policy(policy)
});

Request

Endpoint: PUT /_ilm/policy/{id}

{
  "policy": {
    "phases": {...},
    "name": "{id}"
  }
}

Response

Status: 400 Bad Request

{
  "error": {
    "root_cause": [
      {
        "type": "x_content_parse_exception",
        "reason": "[1:94] [lifecycle_policy] unknown field [name]"
      }
    ],
    "type": "x_content_parse_exception",
    "reason": "[1:101] [put_lifecycle_request] failed to parse field [policy]",
    "caused_by": {
      "type": "x_content_parse_exception",
      "reason": "[1:94] [lifecycle_policy] unknown field [name]"
    }
  },
  "status": 400
}

[oliverlockwood]

Yup, this leaves people forced to continue using the RHLC to set lifecycle policies.

[raynigon]

I updated my cluster to 8.1.3 and tried to use the RHLC but this did not work due to an incompatibility (see Stack Trace below). But I can not fully replace the RHLC because of this issue.
This is a big blocker!

java.io.IOException: Unable to parse response body for Response
{requestLine=GET /_index_template/****?master_timeout=30s HTTP/1.1, host=****, response=HTTP/1.1 200 OK}
        at ****
        at ****
        at ****
        ... N more
Caused by: org.elasticsearch.xcontent.XContentParseException: [1:248] [index_templates] failed to parse field [index_templates]
	at org.elasticsearch.xcontent.ObjectParser.throwFailedToParse(ObjectParser.java:609)
	at org.elasticsearch.xcontent.ObjectParser.parseValue(ObjectParser.java:604)
	at org.elasticsearch.xcontent.ObjectParser.parseArray(ObjectParser.java:597)
	at org.elasticsearch.xcontent.ObjectParser.parseSub(ObjectParser.java:630)
	at org.elasticsearch.xcontent.ObjectParser.parse(ObjectParser.java:320)
	at org.elasticsearch.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:161)
	at org.elasticsearch.xcontent.ConstructingObjectParser.apply(ConstructingObjectParser.java:153)
	at org.elasticsearch.client.indices.GetComposableIndexTemplatesResponse.fromXContent(GetComposableIndexTemplatesResponse.java:75)
	at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2484)
	at org.elasticsearch.client.RestHighLevelClient.lambda$performRequestAndParseEntity$9(RestHighLevelClient.java:2118)
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:2188)
	... 31 more
Caused by: org.elasticsearch.xcontent.XContentParseException: [1:248] [named_index_template] failed to parse field [index_template]
	at org.elasticsearch.xcontent.ObjectParser.throwFailedToParse(ObjectParser.java:609)
	at org.elasticsearch.xcontent.ObjectParser.parseValue(ObjectParser.java:604)
	at org.elasticsearch.xcontent.ObjectParser.parseSub(ObjectParser.java:616)
	at org.elasticsearch.xcontent.ObjectParser.parse(ObjectParser.java:320)
	at org.elasticsearch.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:161)
	at org.elasticsearch.xcontent.AbstractObjectParser.lambda$declareFieldArray$20(AbstractObjectParser.java:336)
	at org.elasticsearch.xcontent.AbstractObjectParser.parseArray(AbstractObjectParser.java:418)
	at org.elasticsearch.xcontent.AbstractObjectParser.lambda$declareFieldArray$21(AbstractObjectParser.java:336)
	at org.elasticsearch.xcontent.ObjectParser.lambda$declareField$8(ObjectParser.java:424)
	at org.elasticsearch.xcontent.ObjectParser.parseValue(ObjectParser.java:602)
	... 40 more
Caused by: org.elasticsearch.xcontent.XContentParseException: [1:248] [index_template] failed to parse field [data_stream]
	at org.elasticsearch.xcontent.ObjectParser.throwFailedToParse(ObjectParser.java:609)
	at org.elasticsearch.xcontent.ObjectParser.parseValue(ObjectParser.java:604)
	at org.elasticsearch.xcontent.ObjectParser.parseSub(ObjectParser.java:616)
	at org.elasticsearch.xcontent.ObjectParser.parse(ObjectParser.java:320)
	at org.elasticsearch.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:161)
	at org.elasticsearch.xcontent.ObjectParser.lambda$declareField$8(ObjectParser.java:424)
	at org.elasticsearch.xcontent.ObjectParser.parseValue(ObjectParser.java:602)
	... 48 more
Caused by: org.elasticsearch.xcontent.XContentParseException: [1:225] [data_stream_template] unknown field [allow_custom_routing]
	at org.elasticsearch.xcontent.ObjectParser.lambda$errorOnUnknown$2(ObjectParser.java:100)
	at org.elasticsearch.xcontent.ObjectParser.parse(ObjectParser.java:304)
	at org.elasticsearch.xcontent.ConstructingObjectParser.parse(ConstructingObjectParser.java:161)
	at org.elasticsearch.xcontent.ObjectParser.lambda$declareField$8(ObjectParser.java:424)
	at org.elasticsearch.xcontent.ObjectParser.parseValue(ObjectParser.java:602)
	... 53 more

[oliverlockwood]

@swallez it's been 7 weeks since I flagged this issue with you in a comment on #203.

Perhaps this and the other issues I highlighted need to be tagged with a label such as "blocks use of new Java client"?

Also, surely this particular issur is trivial to resolve, so why not just get it fixed?

If you're really short staffed I'll consider raising a PR myself. Let me know.

Many thanks.

[swallez]

@oliverlockwood I'm working on a number of things both on the Java client and other projects, and prioritizing something will make some people happy but will obviously not be satisfying for others who don't care much about it. I understand your concerns and I'm currently actively working on fixing the data structure related issues (coming from the API specification) since they're more problematic that say missing features or a more extensive documentation, which are also highly requested. However data structure related issues also have to be prioritized according to the usage frequency of the affected APIs.

Now I heard you, and expect some news on these issues very soon. Thanks.

I also added a comment in #203 about the split package issue.

[oliverlockwood]

Thank you @swallez ♥️

[swallez]

@raynigon what is the content of your policy variable in the snippet you provided:

client.ilm().putLifecycle(it -> {
    it.name(policy.name())
    it.policy(policy)
});

I was able to reproduce the problem by setting the name property on the policy. This is a bug and that property should not be there.

However, when removing policy.name and with the name only set on the request as in it.name(policy.name()) above this works nicely.

Example:

Time tenDays = Time.of(t -> t.time("10d"));
JsonObject empty = Json.createObjectBuilder().build();

PutLifecycleRequest req = PutLifecycleRequest.of(it -> it
    .name("foo")     // <=== set this one
    .policy(pol -> pol
        .phases(ph -> ph
            .hot(h -> h
                .minAge(tenDays)
                .actions(JsonData.of(empty))
            )
        )
        .name("bar") // <=== do NOT set this one
    )
);

We'll remove policy.name in the next release.

[swallez]

Also:

I updated my cluster to 8.1.3 and tried to use the RHLC but this did not work

Have you enabled HLRC compatibility mode using .setApiCompatibilityMode(true)? See https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/migrate-hlrc.html

This allows the HLRC to work with Elasticsearch version 8.x.

[raynigon]

@swallez Thanks that helped a lot.

Not setting the name on the ILM Policy resolves the error.
I can now create/update the ILM Policy, but i suggest to let this issue stay open, until the name property was removed.

One of the next steps in my MigrationService handles component templates i found an error case there as well, see #282

[swallez]

i suggest to let this issue stay open

Absolutely. I've already fixed this in the API specification as part or a larger batch of fixes, and will close this issue once the Java code has been udpated to remove this property that should not exist.

[swallez]

This has been fixed in PR #292, backported to version 8.2 in PR #293 and to version 7.17 in PR #294.