Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] Document limits for JSON objects with ignore_malformed mapping setting #40976

Merged
merged 2 commits into from
Apr 10, 2019
Merged

[DOCS] Document limits for JSON objects with ignore_malformed mapping setting #40976

merged 2 commits into from
Apr 10, 2019

Conversation

jrodewig
Copy link
Contributor

@jrodewig jrodewig commented Apr 8, 2019

As of this writing,ignore_malformed has two undocumented limits for JSON objects:

  • ignore_malformed does not support datatypes intended for JSON objects, like range or object
  • Other field types support ignore_malformed for any JSON data format except JSON objects

This documents those limits. As #12366 notes, support for JSON objects may be added later. We can remove this section then.

Resolves #36464.

@jrodewig jrodewig added >docs General docs changes :Search Foundations/Mapping Index mappings, including merging and defining field types v7.0.0 v8.0.0 v7.2.0 labels Apr 8, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search

Copy link
Member

@bczifra bczifra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change says:

You can't use ignore_malformed to index JSON objects into field datatypes used
for other data formats, like text or integer.

ignore_malformed isn't used to index values. As the docs currently indicate:

The ignore_malformed parameter, if set to true, allows the exception to be ignored. The malformed field is not indexed, but other fields in the document are processed normally.

Thus, this should be something more like:

You can't use `ignore_malformed` to ignore JSON objects submitted for indexing against fields that were not mapped as objects, nested objects, or ranges. The field will not be ignored, the document will not be indexed, and instead an error will be returned for the indexing request.

I don't know whether this limitation is restricted to object, nested object, and range mappings, or whether there are any additional types for which this should be documented.

@jrodewig
Copy link
Contributor Author

jrodewig commented Apr 9, 2019

Thanks for your feedback @bczifra. I particularly appreciate you catching that use of index.

In 99f135d, I revised the section to highlight:

  • The list of unsupported datatypes
  • What we consider a JSON object (which includes the nested, object, and range mappings)
  • Outcomes for submitting a JSON object to an unsupported field

I did some testing in 6.7 and confirmed the following with @jimczi:

  • ignore_malformed does not support submitting any malformed data to nested, object, or range fields at all
  • For fields mapped to other datatypes, ignore_malformed will work with any JSON data format except a JSON object. Data using the nested, object, or range mapping would be considered JSON objects.

@bczifra
Copy link
Member

bczifra commented Apr 9, 2019

@jrodewig the changed text is great! Thanks for clarifying this!

@polyfractal polyfractal removed the v7.0.0 label Apr 9, 2019
@jrodewig jrodewig added v7.0.1 and removed v7.0.1 labels Apr 9, 2019
@jrodewig
Copy link
Contributor Author

@bczifra When convenient and if you approve, do you mind changing your review status from requested changes to approved? Thanks!

Copy link
Contributor

@jimczi jimczi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@bczifra bczifra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jrodewig jrodewig added v7.0.1 and removed v7.0.1 labels Apr 10, 2019
@jrodewig jrodewig merged commit 5a44cab into elastic:master Apr 10, 2019
@jrodewig jrodewig deleted the ignore-malformed-json-objects branch April 10, 2019 18:46
jrodewig added a commit that referenced this pull request Apr 10, 2019
jrodewig added a commit that referenced this pull request Apr 10, 2019
jrodewig added a commit that referenced this pull request Apr 10, 2019
jrodewig added a commit that referenced this pull request Apr 10, 2019
jrodewig added a commit that referenced this pull request Apr 10, 2019
jrodewig added a commit that referenced this pull request Apr 10, 2019
gurkankaymak pushed a commit to gurkankaymak/elasticsearch that referenced this pull request May 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>docs General docs changes :Search Foundations/Mapping Index mappings, including merging and defining field types v7.2.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[documentation] Document field type limitations of ignore_malformed
6 participants