Enhanced segment files sizes information in Nodes Stats/Indices Stats APIs #71643
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tlrx
added
>enhancement
:Data Management/Stats
elasticmachine
added
the
Team:Data Management
Collaborator
|
Pinging @elastic/es-core-features (Team:Core/Features) |
ywelsch
approved these changes
Apr 14, 2021
Contributor
ywelsch
left a comment
ywelsch
left a comment
There was a problem hiding this comment.
I think that these information give a better view of the segment files and are useful in many cases, specially with searchable snapshots whose segment stats can now be introspected thanks to the include_unloaded_segments parameter.
agree. LGTM
Member
Author
|
Thanks Yannick and Adrien! |
tlrx
added a commit
to tlrx/elasticsearch
that referenced
this pull request
Apr 15, 2021
… APIs (elastic#71643) Since elastic#16661 it is possible to know the total sizes for some Lucene segment files by using the Node Stats or Indices Stats API with the include_segment_file_sizes parameter, and the list of file extensions has been extended in elastic#71416. This commit adds a bit more information about file sizes like the number of files (count), the min, max and average file sizes in bytes that share the same extension. Here is a sample: "cfs" : { "description" : "Compound Files", "size_in_bytes" : 2260, "min_size_in_bytes" : 2260, "max_size_in_bytes" : 2260, "average_size_in_bytes" : 2260, "count" : 1 } This commit also simplifies how compound file sizes were computed: before compound segment files were extracted and sizes aggregated with regular non-compound files sizes (which can be confusing and out of the scope of the original issue elastic#6728), now CFS/CFE files appears as distinct files. These new information are provided to give a better view of the segment files and are useful in many cases, specially with frozen searchable snapshots whose segment stats can now be introspected thanks to the include_unloaded_segments parameter.
Contributor
|
Is it breaking enough that it should only go in 8.0? |
Member
Author
I think it can go in 7.13.0. The REST response contains more information now but the previous fields are unchanged; the set of files has been already extended in #71416 but existing extensions and descriptions remain the same (just more extensions now); CFS/CFE is computed differently but the memory infos are still the same. |
Member
Author
|
@ywelsch do you have an opinion? |
Contributor
|
I think this is not breaking for the reasons that you've outlined, so 7.13 is ok |
Member
Author
Contributor
|
@tlrx I trust your judgement, I only wanted to make sure this question had been considered given the change to how CFS files are treated. |
tlrx
added a commit
that referenced
this pull request
Apr 15, 2021
… Stats APIs (#71725) Since #16661 it is possible to know the total sizes for some Lucene segment files by using the Node Stats or Indices Stats API with the include_segment_file_sizes parameter, and the list of file extensions has been extended in #71416. This commit adds a bit more information about file sizes like the number of files (count), the min, max and average file sizes in bytes that share the same extension. Here is a sample: "cfs" : { "description" : "Compound Files", "size_in_bytes" : 2260, "min_size_in_bytes" : 2260, "max_size_in_bytes" : 2260, "average_size_in_bytes" : 2260, "count" : 1 } This commit also simplifies how compound file sizes were computed: before compound segment files were extracted and sizes aggregated with regular non-compound files sizes (which can be confusing and out of the scope of the original issue #6728), now CFS/CFE files appears as distinct files. These new information are provided to give a better view of the segment files and are useful in many cases, specially with frozen searchable snapshots whose segment stats can now be introspected thanks to the include_unloaded_segments parameter. Backport of #71643
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Since #16661 it is possible to know the total sizes for some Lucene segment files by using the Node Stats or Indices Stats API with the
include_segment_file_sizesparameter, and the list of file extensions has been extended in #71416.This pull request adds a bit more information about file sizes like the number of files (
count), themin,maxand average file sizes in bytes that share the same extension. Here is a sample:This pull request also simplifies how compound file sizes were computed: before compound segment files were extracted and sizes aggregated with regular non-compound files sizes (which I find confusing and out of the scope of the original issue #6728), now CFS/CFE files appears as distinct files.
I think that these information give a better view of the segment files and are useful in many cases, specially with searchable snapshots whose segment stats can now be introspected thanks to the
include_unloaded_segmentsparameter.