Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make use of IsAlias safeguard introduced in libbeat #7116

Open
simitt opened this issue Jan 24, 2022 · 4 comments
Open

Make use of IsAlias safeguard introduced in libbeat #7116

simitt opened this issue Jan 24, 2022 · 4 comments

Comments

@simitt
Copy link
Contributor

simitt commented Jan 24, 2022

Follow up from #7101. With elastic/beats@84bf434, libbeat introduced a safeguard check before ingesting data to ES. It adds the IsAlias() function to the IndexSelector Interface, enabling the output handler to require an alias instead of an index when ingesting data.

The APM Server currently always returns false for the IsAlias() check for two reasons.

  • The check for IsAlias is not straight forward when returning the autoSelector. A fist pass on this can be found in 5111d3e. This implementation does not yet handle whether an alias of index is set in the meta information.
  • When ILM is enabled, APM Server still needs to write to indices for sourcemaps and onboarding. Therefore the newly introduced check cannot simply be leveraged by APM at this point.

Investigate how this check can be leveraged for general event ingestion, while still allowing to send sourcemap and onboarding data to indices.

@simitt simitt added this to the 7.17 milestone Jan 24, 2022
@simitt
Copy link
Contributor Author

simitt commented Jan 27, 2022

Turns out that managing the meta information in the event is not only a problem for apm, but also for the original beats implementation. This is now handled in the output implementation https://github.com/elastic/beats/pull/30055/files.

This should reduce the issues to handling onboarding and sourcemaps events. Mayb we can leverage the meta field and just always overwrite the meta.index for these events before shipping them.

@simitt
Copy link
Contributor Author

simitt commented Jan 28, 2022

@lucabelluccini - APM doesn't make use of the new require_alias feature yet due to reasons explained above; but we look into how to support it.

@simitt
Copy link
Contributor Author

simitt commented Feb 2, 2022

Libbeat changes have been reverted in elastic/beats#30058. A new PR elastic/beats#30055 is up.

@simitt
Copy link
Contributor Author

simitt commented Mar 23, 2022

New libbeat issue to follow along with: elastic/beats#27874

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant