Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow specify dynamic templates in bulk request #69948

Merged
merged 49 commits into from
Apr 8, 2021

Conversation

dnhatn
Copy link
Member

@dnhatn dnhatn commented Mar 4, 2021

This change allows users to specify dynamic templates in a bulk request.

PUT myindex
{
  "mappings": {
    "dynamic_templates": [{
      "time_histograms": {
        "mapping": {
          "type": "histogram",
          "meta": {
            "unit": "s"
          }
        }
      }
    }]
  }
}
// Tells ES to use the dynamic template named `time_histograms` for field `response_times`
// in the index request
POST myindex/_bulk
{ "index": { "dynamic_templates": { "response_times": "time_histograms" } } }
{ "@timestamp": "2020-08-12", "response_times": { "values": [1, 10], "counts": [5, 1] }}

Closes #61939

@dnhatn dnhatn force-pushed the dynamic-mapping-hint branch from 07aa9d1 to 6e80821 Compare March 4, 2021 16:31
@dnhatn dnhatn marked this pull request as ready for review March 4, 2021 17:08
@dnhatn dnhatn requested review from jimczi and jpountz March 4, 2021 17:08
@dnhatn dnhatn added the :Search Foundations/Mapping Index mappings, including merging and defining field types label Mar 4, 2021
@elasticmachine elasticmachine added the Team:Search Meta label for search team label Mar 4, 2021
@dnhatn dnhatn added >enhancement and removed Team:Search Meta label for search team labels Mar 4, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-search (Team:Search)

@dnhatn dnhatn added v8.0.0 v7.13.0 Team:Search Meta label for search team labels Mar 4, 2021
@dnhatn dnhatn force-pushed the dynamic-mapping-hint branch from e346e5a to 40e6a57 Compare March 4, 2021 20:09
@jimczi jimczi requested review from javanna and removed request for jpountz March 5, 2021 09:28
Copy link
Member

@javanna javanna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some questions. The approach looks good code-wise overall, my comments are mostly about the API.

Another one: does the matched "type" become the content of the dynamic_type variable? Or should it even? I am wondering myself if this should be considered and promoted as a type. To me it is more of an arbitrary name that defined dynamic templates can match against. Should we rather call it match_mapping_hint?

@dnhatn
Copy link
Member Author

dnhatn commented Mar 9, 2021

@javanna Thank you for the review. I tweaked the API using "match_mapping_hint" and I found it's simpler and less confusing. I have updated the PR. Can you take another look?

@dnhatn dnhatn requested a review from javanna March 9, 2021 21:26
@axw
Copy link
Member

axw commented Mar 10, 2021

@dnhatn apologies if this is obvious: will this be immediately usable through an Ingest script processor (and if so how?), or would that come later?

@dnhatn
Copy link
Member Author

dnhatn commented Mar 10, 2021

can users use one of these types as hint in their bulk requests, in which case they won't have to do define anything on the dynamic template side of things? Or should hints always be arbitrary and explicitly defined in the dynamic template?

That's a good question. Let's discuss it after this iteration.

Copy link
Contributor

@jimczi jimczi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change looks great @dnhatn ! I left some comments but I think the feature is ready.

Copy link
Member

@javanna javanna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some more comments, nothing major though. I like how simple the implementation, thanks for iterating on it. Would it make sense to add docs as part of this PR or do you plan to do it as a followup?

Copy link
Member

@javanna javanna left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some minor comments, LGTM otherwise. Thanks for all the iterations.

@dnhatn
Copy link
Member Author

dnhatn commented Apr 8, 2021

Thanks everyone :).

@dnhatn dnhatn merged commit 5c99692 into elastic:master Apr 8, 2021
@dnhatn dnhatn deleted the dynamic-mapping-hint branch April 8, 2021 16:44
dnhatn added a commit to dnhatn/elasticsearch that referenced this pull request Apr 10, 2021
@dnhatn dnhatn mentioned this pull request Apr 10, 2021
dnhatn added a commit that referenced this pull request Apr 10, 2021
dnhatn added a commit that referenced this pull request Apr 19, 2021
This change exposes the newly introduced parameter `dynamic_templates`
in ingest. This parameter can be set by a set processor or a script processor.

Relates #69948
dnhatn added a commit that referenced this pull request Apr 19, 2021
This change exposes the newly introduced parameter dynamic_templates
in ingest. This parameter can be set by a set processor or a script processor.

Relates #69948
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>enhancement :Search Foundations/Mapping Index mappings, including merging and defining field types Team:Search Meta label for search team v7.13.0 v8.0.0-alpha1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Give bulk requests the ability to give hints to dynamic mappings
7 participants