Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hidden indices not exposed in monitoring index_stats documents #18639

Closed
chrisronline opened this issue May 19, 2020 · 5 comments · Fixed by #18706
Closed

Hidden indices not exposed in monitoring index_stats documents #18639

chrisronline opened this issue May 19, 2020 · 5 comments · Fixed by #18706
Assignees
@ycombinator
Labels
enhancement Feature:Stack Monitoring Team:Services (Deprecated) Label for the former Integrations-Services team

Comments

@chrisronline
Copy link
Contributor

With the introduction of hidden indices, it looks like they are not included in the monitoring index_stats documents collected by Metricbeat. It looks we need to add expand_wildcards=hidden to this api call.

We should also add an additional field for hidden indices that indicates they are hidden, so the UI can filter it. I'm suggesting this to be index_stats.hidden: <boolean>. The challenge here is the API doesn't return this state, but it's rather available in /*/_settings so I'm not sure the right solution moving forward. Happy to discuss

Related to elastic/elasticsearch#56949
@elasticmachine
Copy link
Collaborator

Pinging @elastic/stack-monitoring (Stack monitoring)

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 19, 2020
@andresrc andresrc added the Team:Services (Deprecated) Label for the former Integrations-Services team label May 22, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 22, 2020
@ycombinator ycombinator mentioned this issue May 22, 2020
Merged
8 tasks
@ycombinator
Copy link
Contributor

ycombinator commented May 22, 2020 •
edited
Loading

@chrisronline Thanks for bringing this up. For starters I've inquired with the ES team if it makes sense to make GET stats behave as before, given the intent of hidden indices was more about hiding their data from user's by default, not so much their metadata like stats IIUC. I've started that conversation here: elastic/elasticsearch#50452 (comment).

[EDIT] Just in case the Elasticsearch team decides not to modify the default GET stats behavior to include all indices' stats, I have a PR up to make the Metricbeat elasticsearch module collect all indices stats: #18703.

@ycombinator
Copy link
Contributor

I'm relabeling this issue as an enhancement because the hidden indices feature in Elasticsearch itself was an enhancement. We now need to enhance the Metricbeat elasticsearch module correspondingly.

@ycombinator ycombinator mentioned this issue May 22, 2020
Merged
6 tasks
@ycombinator
Copy link
Contributor

ycombinator commented May 22, 2020 •
edited
Loading

@chrisronline I'm planning to resolve this issue via two PRs:

@ycombinator ycombinator mentioned this issue May 22, 2020
Merged
6 tasks
@ycombinator ycombinator self-assigned this May 22, 2020
This was referenced Apr 23, 2024
Closed
Merged
@mergify mergify bot mentioned this issue Apr 25, 2024
Merged
6 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ycombinator ycombinator

Labels
enhancement Feature:Stack Monitoring Team:Services (Deprecated) Label for the former Integrations-Services team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Report if index is hidden in index stats ycombinator/beats
4 participants
@ycombinator @chrisronline @andresrc @elasticmachine