Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fetch a GeoIP Database for MaxMind GeoIP Processor #33510

Open
rogercoll opened this issue Jun 12, 2024 · 4 comments
Open

Fetch a GeoIP Database for MaxMind GeoIP Processor #33510

rogercoll opened this issue Jun 12, 2024 · 4 comments
Labels

Comments

@rogercoll
Copy link
Contributor

rogercoll commented Jun 12, 2024

Component(s)

processor/geoip

Is your feature request related to a problem? Please describe.

Initial issue: #32663

Currently, a local file path must be provided to the MaxMind GeoIP processor in order to retrieve the associated IP metadata. This approach requires users to manually download and manage the GeoIP database files, ensuring they are kept up-to-date and properly licensed. This manual process can be cumbersome and error-prone. Automating the retrieval of the GeoIP database would significantly streamline the setup and maintenance process.

Describe the solution you'd like

The idea would be to initially download the latest available database from MaxMind's Direct download link: https://dev.maxmind.com/geoip/updating-databases#directly-downloading-databases. The following configuration options will be needed:

  • permalink: url to fetch the database from.
  • account_id: account ID used to fetch.
  • license_key: license key used to fetch.
  • auto_update: Whether to frequently check for available updates.

Describe alternatives you've considered

TODO: Verify if it would be feasible to use the auth extension instead of the custom account_id and license_key configuration options.

Additional context

Automatic database fetch

If neither a local path nor a license key is specified, the processor should be able to fetch a default GeoIP database automatically, similar to Elasticsearch's geoip processor. This feature would eliminate the need to specify a custom configuration.

Elasticsearch automatically downloads updates for these databases from the Elastic GeoIP endpoint: https://geoip.elastic.co/v1/database. The processor could use the same endpoint to automatically download the latest available database.

Licensing

There are some concerns regarding the implementation of this feature if we use Elastic's https://geoip.elastic.co/v1/database as default download endpoint. Could there be any licensing implications by using the same URL? MaxMind databases are shared under the CC BY-SA 4.0 license.

@rogercoll rogercoll added enhancement New feature or request needs triage New item requiring triage labels Jun 12, 2024
Copy link
Contributor

Pinging code owners for processor/geoip: @andrzej-stencel @michalpristas @rogercoll. See Adding Labels via Comments if you do not have permissions to add labels yourself.

@rogercoll
Copy link
Contributor Author

MaxMind changed from releasing the GeoIP database under a Creative Commons (CC) license to a proprietary end-user license agreement (EULA). The MaxMind EULA requires any software bundling their databases to update them within 30 days of a database update.

The automatic database update strategy would be similar to what the GeoIP Logstash plugin does. By default, the Logstash filter bundles Creative Commons (CC) license databases and if the auto-update feature is enabled, it checks for database updates every day. It downloads the latest and replaces the old database while the plugin is running. After Logstash has switched to a EULA licensed database, the geoip filter will stop enriching events in order to maintain compliance if Logstash fails to check for database updates for 30 days. Events will be tagged with _geoip_expired_database tag to facilitate the handling of this situation.

My main concern when porting this functionality into Otel is the dynamic license change of the Logstash filter itself:

If the auto-update feature is enabled, Logstash upgrades from the CC database license to the EULA version on the first download.

@michalpristas
Copy link
Contributor

does this also apply when db is not bundled but just fetched instead?

Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants