API for listing index file sizes (revisited)

[ppf2]

It looks like #6728 is closed via #8832 - but #8832 reports on the memory usage of these items, eg. the doc_values_memory_in_bytes reports on the memory we use for efficient access to the doc values on disk. So I think 8832 only addresses part of this request here. The other part is that it will be helpful to report on the size on disk usage (eg. ls -l) of the Lucene files, in particular, the ones that tend to use the most storage like files associated to term dictionaries, norms, doc values. The use case is that one can use this information to help size the amount of OS cache space (probably not 1-1 since its unlikely that it will use all of the norms, doc values, terms, etc.. but it will be helpful to get an idea of the worst case scenario when it comes to sizing). Since this may end up being a separate API, I am filing this as a new ticket.

[jpountz]

+1

[mikemccand]

+1

[camilojd]

Hi, I'd like to add support for these stats as another member of CommonStats. Already did some tinkering and would like to go further :-)

[camilojd]

Please see my implementation at camilojd@f80b627. Some comments/questions follow.

I'm consolidating file sizes using the same criteria of SegmentsStats to expose disk usage of Lucene index files: Terms, TermVectors, StoredFields, Norms, DocValues (class IndexResources in SegmentsStats).

SegmentsStats currently exposes memory consumption as bytes of each of the aforementioned so it's possible to add these as part of a class (IndexResources) and expose it as two fields of SegmentsStats, one for memory resources (existing stats) and other for disk resources. This is what I implemented in my branch, but I'm not sure if it would be better to expose this as another "sister" class of SegmentsStats, member of CommonStats, and leave SegmentsStats as it is. I'm however aware that my toXContent serialization could break clients expecting certain fields that now are placed elsewhere.

Some other couple questions I've got:

1. Is it necessary to consolidate sizes by the same criteria of SegmentsStats or may be better doing it using the file extensions? This last option has the advantage that would include additional information about postings, i.e., separate size info for positions, payloads, etc.
2. The meat of the matter: Is the way I open the Directory from the SegmentInfo and the compound reader a sensible approach? I'm not sure if this could introduce issues with the Store.

[clintongormley]

@camilojd i'd suggest opening a PR to discuss this, even if it is a WIP

[camilojd]

Will do! thanks @clintongormley

[clintongormley]

Closed by #16661