Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove default maxmind geoip databases from distribution #78362

Merged
merged 19 commits into from
Oct 13, 2021

Conversation

martijnvg
Copy link
Member

@martijnvg martijnvg commented Sep 28, 2021

  • Adjusted integration tests to use geoip test fixture or to use test databases provided via config dirs (for qa module / docs).
  • Kept the geolite2-databases dependency for most of the unit tests only.
  • Made fallback_to_default_databases parameter on geoip processor a noop and emit deprecation warning upon using it.
  • If no geoip databases are available yet to a node then the geoip processor factory returns a processor implementation that flags documents that databases are unavailable. This allows these documents to be reindex later with a pipeline. These documents will have a tag string array field, which contains a string _geoip_database_unavailable_{database_name} for each missing database in a pipeline.
  • Added reload pipeline capabilities is IngestService, so that when databases are available again on a node then pipelines with geoip processor definition can be reloaded.

Relates to #68920

@martijnvg martijnvg force-pushed the geoip_remove_default_databases branch 3 times, most recently from 197b5a5 to 6e1c111 Compare October 7, 2021 13:05
adjusted tests. Kept the `geolite2-databases` dependency for
tests only.

Relates to elastic#68920
…lementation.

When DatabaseRegistry detects that databases are available then all geoip processor
that failed loading will be reloaded.
@martijnvg martijnvg force-pushed the geoip_remove_default_databases branch from 6e1c111 to b44dcb7 Compare October 7, 2021 15:10
@martijnvg martijnvg added :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >breaking >bug labels Oct 7, 2021
@martijnvg martijnvg marked this pull request as ready for review October 11, 2021 07:51
@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Oct 11, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@martijnvg martijnvg requested a review from probakowski October 11, 2021 07:51
@martijnvg martijnvg changed the title Remove default maxmind geoip databases from distribution and Remove default maxmind geoip databases from distribution Oct 11, 2021
Copy link
Contributor

@probakowski probakowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks @martijnvg !
One super minor nit

/**
* Keeps track of the databases locally available to a node:
* 1) Default databases shipped with the default distribution via ingest-geoip module
* 2) User provided databases from the ES_HOME/config/ingest-geoip directory. This directory is monitored
* 1) User provided databases from the ES_HOME/config/ingest-geoip directory. This directory is monitored
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. doesn't make sense here as we have single point

@martijnvg
Copy link
Member Author

  1. doesn't make sense here as we have single point

Also I will rename this class to ConfigDatabases in a followup. To better describe the function of this class.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>breaking >bug :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP Team:Data Management Meta label for data/management team v8.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants