Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Discover] Fix filtering out custom meta fields of Elasticsearch plugins enhanced documents #137147

Merged
merged 9 commits into from
Jul 29, 2022
Merged

[Discover] Fix filtering out custom meta fields of Elasticsearch plugins enhanced documents #137147

merged 9 commits into from
Jul 29, 2022

Conversation

kertal
Copy link
Member

@kertal kertal commented Jul 26, 2022 •
edited by kibanamachine
Loading

Summary

In #114517 multiple flattening of documents implementations were unified. This caused a bug with meta fields. The implementation assumed meta fields would match a given set of meta fields, filtering out all other fields. However, there's a way to extend the set of meta fields in Kibana with Elasticsearch plugins like the Mapper Size Plugin
There was not way to view the _size meta field in Discover (apart from the JSON view). This fix changes the implementation to exclude the fields that should not be merged ('_source', '_type')

Testing

  • Install the Mapper Size Plugin to test this.
    https://www.elastic.co/guide/en/elasticsearch/plugins/current/mapper-size.html
  • You need to install a previously release ES version, since it doesn't work with snapshots
    yarn es archive https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-8.3.2-darwin-aarch64.tar.gz
  • Go to new downloaded archive cd .es/elasticsearch-8.3.2-darwin-aarch64
  • There you can install the plugin by ./bin/elasticsearch-plugin install mapper-size
  • You need to disable security by settingxpack.security.enabled: false in ./config/elasticsearch.yml
  • Then you can start Elasticsearch by executing ES_PATH_CONF=./config ./bin/elasticsearch in es/elasticsearch-8.3.2-darwin-aarch64
  • You need to configure Kibana to work with this previous version by setting elasticsearch.ignoreVersionMismatch: true in your config/kibana.dev.yml
  • Then you should be able to start Kibana
  • Add _size to metaFields of Advanced settings
  • Configure mapping to use _size and ingest data like it is described in https://www.elastic.co/guide/en/elasticsearch/plugins/current/mapper-size-usage.html
  • Go to Discover, create a data view for the new index with the ingested data
  • _size should be displayed with a value when expanding a document, and you can add the field to Discover's document explorer

Fixes #121394

Checklist

@kertal kertal self-assigned this Jul 27, 2022
@kertal kertal added the Feature:Discover Discover Application label Jul 27, 2022
@kertal kertal marked this pull request as ready for review July 27, 2022 13:52
@kertal kertal requested a review from a team as a code owner July 27, 2022 13:52
@kertal kertal changed the title [Discover] improve meta field flattening [Discover] Fix filtering out custom meta fields of Elasticsearch plugins Jul 27, 2022
@kertal kertal added backport:prev-major Backport to the previous major branch and all later branches still in development release_note:fix Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph) labels Jul 27, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-data-discovery (Team:DataDiscovery)

@kertal kertal requested a review from ppisljar July 28, 2022 06:07
ppisljar
ppisljar approved these changes Jul 28, 2022
View reviewed changes
Copy link
Member

@ppisljar ppisljar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@kertal
Copy link
Member Author

kertal commented Jul 28, 2022

@elasticmachine merge upstream

davismcphee
davismcphee approved these changes Jul 28, 2022
View reviewed changes
Copy link
Contributor

@davismcphee davismcphee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Followed your instructions to test and it works as expected, LGTM 👍

@kertal
Copy link
Member Author

kertal commented Jul 29, 2022

@elasticmachine merge upstream

@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id before after diff
data 428.1KB 428.0KB -102.0B

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kertal

@kertal kertal changed the title [Discover] Fix filtering out custom meta fields of Elasticsearch plugins [Discover] Fix filtering out custom meta fields of Elasticsearch plugins enhanced documents Jul 29, 2022
@kertal kertal merged commit 6323628 into elastic:main Jul 29, 2022
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jul 29, 2022
@kertal
[Discover] Fix filtering out custom meta fields of Elasticsearch plug…
f9c05ec 
…ins enhanced documents (elastic#137147)

* Restoring the ability of showing custom meta fields like _size

(cherry picked from commit 6323628)
@kibanamachine kibanamachine mentioned this pull request Jul 29, 2022
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jul 29, 2022
@kertal
[Discover] Fix filtering out custom meta fields of Elasticsearch plug…
da7c785 
…ins enhanced documents (elastic#137147)

* Restoring the ability of showing custom meta fields like _size

(cherry picked from commit 6323628)
@kibanamachine kibanamachine mentioned this pull request Jul 29, 2022
Merged
@kibanamachine
Copy link
Contributor

💔 Some backports could not be created

Status Branch Result
❌ 7.17 Backport failed because of merge conflicts
✅ 8.3
✅ 8.4

Note: Successful backport PRs will be merged automatically after passing CI.

Manual backport

To create the backport manually run:

node scripts/backport --pr 137147

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Jul 29, 2022
@kibanamachine @kertal
[Discover] Fix filtering out custom meta fields of Elasticsearch plug…
38aa093 
…ins enhanced documents (#137147) (#137555)

* Restoring the ability of showing custom meta fields like _size

(cherry picked from commit 6323628)

Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
kibanamachine added a commit that referenced this pull request Jul 29, 2022
@kibanamachine @kertal
[Discover] Fix filtering out custom meta fields of Elasticsearch plug…
2d20394 
…ins enhanced documents (#137147) (#137556)

* Restoring the ability of showing custom meta fields like _size

(cherry picked from commit 6323628)

Co-authored-by: Matthias Wilhelm <matthias.wilhelm@elastic.co>
@kertal kertal mentioned this pull request Sep 21, 2022
Closed
@kertal
Copy link
Member Author

kertal commented Sep 21, 2022

💚 All backports created successfully

Status Branch Result
✅ 7.17

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kertal added a commit to kertal/kibana that referenced this pull request Sep 21, 2022
@kertal
[Discover] Fix filtering out custom meta fields of Elasticsearch plug…
122c02c 
…ins enhanced documents (elastic#137147)

* Restoring the ability of showing custom meta fields like _size

(cherry picked from commit 6323628)

# Conflicts:
#	src/plugins/data/common/search/tabify/tabify_docs.ts
@jughosta jughosta mentioned this pull request Sep 22, 2022
Closed
@kertal kertal mentioned this pull request May 3, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ppisljar ppisljar ppisljar approved these changes

@davismcphee davismcphee davismcphee approved these changes

Assignees

@kertal kertal

Labels
backport:prev-major Backport to the previous major branch and all later branches still in development Feature:Discover Discover Application release_note:fix Team:DataDiscovery Discover App Team (Document Explorer, Saved Search, Surrounding documents, Graph) v8.3.4 v8.4.0 v8.5.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Discover] Using the mapper_size (ES plugin) _size is missing in the document table
6 participants
@kertal @elasticmachine @kibana-ci @kibanamachine @ppisljar @davismcphee