Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CreateIndexRequest with raw JSON and "limit token count token filter" fails if optional parameter is missing #313

Closed
slovdahl opened this issue Jun 15, 2022 · 3 comments
Labels
Area: Specification Related to the API spec used to generate client code

Comments

@slovdahl
Copy link

slovdahl commented Jun 15, 2022

Java API client version

7.17.4

Java version

1.8.0_332

Elasticsearch Version

7.17.4

Problem description

We have index mappings stored as JSON files and have successfully used RestHighLevelClient like this:

CreateIndexRequest request = new CreateIndexRequest("test")
        .source(mappingJson, XContentType.JSON);

client.indices().create(request, RequestOptions.DEFAULT);

We're now trying to migrate away from RestHighLevelClient to something like this:

client.indices()
        .create(
                createIndexBuilder -> createIndexBuilder
                        .index("test")
                        .withJson(new ByteArrayInputStream(mappingJson.getBytes(StandardCharsets.UTF_8)))
        );

But with the JSON below, we get an exception like this:

Caused by:
        co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch._types.analysis.TokenFilterDefinition: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'LimitTokenCountTokenFilter.consumeAllTokens' (JSON path: settings.index.analysis.filter.one_token_limit) (line no=1, column no=10226, offset=10225)
            at co.elastic.clients.json.JsonpMappingException.from0(JsonpMappingException.java:134)
            at co.elastic.clients.json.JsonpMappingException.from(JsonpMappingException.java:121)
            at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:206)
            at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
            at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:53)
            at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:48)
            at co.elastic.clients.json.UnionDeserializer$SingleMemberHandler.deserialize(UnionDeserializer.java:74)
            at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:291)
            at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:258)
            at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
            at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:347)
            at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:331)
            at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
            at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
            at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
            at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
            at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
            at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
            at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
            at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
            at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
            at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
            at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
            at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
            at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
            at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
            at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
            at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
            at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
            at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
            at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
            at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
            at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
            at co.elastic.clients.util.WithJsonObjectBuilderBase.withJson(WithJsonObjectBuilderBase.java:66)
            at co.elastic.clients.json.WithJson.withJson(WithJson.java:43)
            at test.ElasticsearchMappingUtil.lambda$createIndices$5(ElasticsearchMappingUtil.java:121)
            at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesAsyncClient.create(ElasticsearchIndicesAsyncClient.java:284)
            at test.ElasticsearchMappingUtil.createIndices(ElasticsearchMappingUtil.java:118)
            ... 2 more

            Caused by:
            co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'LimitTokenCountTokenFilter.consumeAllTokens'
                at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76)
                at co.elastic.clients.elasticsearch._types.analysis.LimitTokenCountTokenFilter.<init>(LimitTokenCountTokenFilter.java:58)
                at co.elastic.clients.elasticsearch._types.analysis.LimitTokenCountTokenFilter.<init>(LimitTokenCountTokenFilter.java:48)
                at co.elastic.clients.elasticsearch._types.analysis.LimitTokenCountTokenFilter$Builder.build(LimitTokenCountTokenFilter.java:144)
                at co.elastic.clients.elasticsearch._types.analysis.LimitTokenCountTokenFilter$Builder.build(LimitTokenCountTokenFilter.java:107)
                at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:80)
                at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
                at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
                at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:197)
                ... 38 more

Even though the documentation on https://www.elastic.co/guide/en/elasticsearch/reference/7.17/analysis-limit-token-count-tokenfilter.html#analysis-limit-token-count-tokenfilter-configure-parms states that consume_all_tokens is optional. It works fine if "consume_all_tokens": false is added to the one_token_limit filter.

This is a stripped down mapping JSON that fails:

{
    "mappings": {
        "properties": {
            "id": {
                "type": "keyword"
            }
        },
        "dynamic_templates": [
            {
                "metadata_title_en": {
                    "path_match": "metadata.en.title",
                    "mapping": {
                        "type": "text",
                        "fields": {
                            "first_word": {
                                "type": "text",
                                "fielddata": true,
                                "analyzer": "first_word",
                                "search_analyzer": "standard"
                            }
                        }
                    }
                }
            }
        ]
    },
    "settings": {
        "index": {
            "analysis": {
                "analyzer": {
                    "first_word": {
                        "type": "custom",
                        "tokenizer": "standard",
                        "filter": [
                            "lowercase",
                            "one_token_limit"
                        ]
                    }
                },
                "normalizer": {
                    "lowercase": {
                        "type": "custom",
                        "filter": [
                            "lowercase"
                        ]
                    }
                },
                "filter": {
                    "one_token_limit": {
                        "type": "limit",
                        "max_token_count": 1
                    }
                }
            }
        }
    }
}
@swallez swallez added the Area: Specification Related to the API spec used to generate client code label Jun 16, 2022
@slovdahl
Copy link
Author

Working around the CreateIndexRequest issue is easy enough, just set max_token_count to the same as the default. But one thing that's harder to work around is that a client.indices().get(GetIndexRequest) request for an index created without max_token_count set fails in the same way. 😒

co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch._types.analysis.TokenFilterDefinition: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'LimitTokenCountTokenFilter.consumeAllTokens' (JSON path: ['test-index'].settings.index.analysis.filter.one_token_limit) (line no=1, column no=27809, offset=-1)
	at co.elastic.clients.json.JsonpMappingException.from0(JsonpMappingException.java:134)
	at co.elastic.clients.json.JsonpMappingException.from(JsonpMappingException.java:121)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:206)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.BuildFunctionDeserializer.deserialize(BuildFunctionDeserializer.java:53)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:48)
	at co.elastic.clients.json.UnionDeserializer$SingleMemberHandler.deserialize(UnionDeserializer.java:74)
	at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:291)
	at co.elastic.clients.json.UnionDeserializer.deserialize(UnionDeserializer.java:258)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:347)
	at co.elastic.clients.json.JsonpDeserializerBase$StringMapDeserializer.deserialize(JsonpDeserializerBase.java:331)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:180)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.transport.endpoints.DictionaryResponse.lambda$setupDictionaryResponseDeserializer$0(DictionaryResponse.java:153)
	at co.elastic.clients.json.ObjectDeserializer.parseUnknownField(ObjectDeserializer.java:214)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:178)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:136)
	at co.elastic.clients.json.JsonpDeserializer.deserialize(JsonpDeserializer.java:75)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:79)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.transport.rest_client.RestClientTransport.decodeResponse(RestClientTransport.java:328)
	at co.elastic.clients.transport.rest_client.RestClientTransport.getHighLevelResponse(RestClientTransport.java:294)
	at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:147)
	at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.get(ElasticsearchIndicesClient.java:934)
	at co.elastic.clients.elasticsearch.indices.ElasticsearchIndicesClient.get(ElasticsearchIndicesClient.java:950)
	at test.ElasticsearchService.lambda$getAllIndices$21(ElasticsearchService.java:1172)
Caused by: co.elastic.clients.util.MissingRequiredPropertyException: Missing required property 'LimitTokenCountTokenFilter.consumeAllTokens'
	at co.elastic.clients.util.ApiTypeHelper.requireNonNull(ApiTypeHelper.java:76)
	at co.elastic.clients.elasticsearch._types.analysis.LimitTokenCountTokenFilter.<init>(LimitTokenCountTokenFilter.java:58)
	at co.elastic.clients.elasticsearch._types.analysis.LimitTokenCountTokenFilter.<init>(LimitTokenCountTokenFilter.java:48)
	at co.elastic.clients.elasticsearch._types.analysis.LimitTokenCountTokenFilter$Builder.build(LimitTokenCountTokenFilter.java:144)
	at co.elastic.clients.elasticsearch._types.analysis.LimitTokenCountTokenFilter$Builder.build(LimitTokenCountTokenFilter.java:107)
	at co.elastic.clients.json.ObjectBuilderDeserializer.deserialize(ObjectBuilderDeserializer.java:80)
	at co.elastic.clients.json.DelegatingDeserializer$SameType.deserialize(DelegatingDeserializer.java:43)
	at co.elastic.clients.json.ObjectDeserializer$FieldObjectDeserializer.deserialize(ObjectDeserializer.java:71)
	at co.elastic.clients.json.ObjectDeserializer.deserialize(ObjectDeserializer.java:197)
	... 67 common frames omitted

@chriswhite199
Copy link

Facing the same issue, and my workaround was to:

  • Get the index settings (GET my-index/_settings)
  • Close the index (POST my-index/_close)
  • Update the analyzer settings (to include the consume_all_tokens field) (PUT my-index/_settings {...})
  • Open the index (POST my-index/_open)

Subsequent GetIndex request via Java client no longer errors.

@swallez
Copy link
Member

swallez commented Aug 23, 2022

This has been fixed in PR #344, which will be included in versions 8.4.0 and 7.17.6

@swallez swallez closed this as completed Aug 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area: Specification Related to the API spec used to generate client code
Projects
None yet
Development

No branches or pull requests

3 participants