BUG: maxmind update script: license key

MaxMind requires a free registration and license key for downloading the
GeoLite2 database

CHANGELOG.md

@@ -33,6 +33,7 @@ CHANGELOG
 
 #### Experts
 - `intelmq.bots.experts.national_cert_contact_certat`: Handle empty responses by server (#1467).
+- `intelmq.bots.experts.maxmind_geoip`: The script `update-geoip-data` now requires a license key as second parameter because of upstream changes.
 
 #### Outputs
 - `intelmq.bots.outputs.restapi.output`: Fix logging of response body if response status code was not ok.

NEWS.md

@@ -15,6 +15,10 @@ See the changelog for a full list of changes.
 
 ### Configuration
 
+#### MaxMind GeoIP
+MaxMind requires a registration before being able to download the GeoLite2 database starting with 2019-12-30: https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
+If the provided `update-geoip-data` script is used, the license key can now be set second parameter.
+
 ### Libraries
 
 ### Postgres databases

docs/Bots.md

@@ -1796,8 +1796,7 @@ Documentation about IDEA: https://idea.cesnet.cz/en/index
 
 The bot requires the maxmind's `geoip2` Python library, version 2.2.0 has been tested.
 
-The database is available at https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
-You need to unzip it.
+To download the database a free license key is required. More information can be found at https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
 
 You may want to use a shell script provided in the contrib directory to keep the database up to date: `contrib/cron-jobs/update-geoip-data`
 

intelmq/bots/experts/maxmind_geoip/update-geoip-data

@@ -12,20 +12,23 @@ PATH=/bin:/usr/bin
 # This product includes GeoLite2 data created by MaxMind, available from
 # <a href="http://www.maxmind.com">http://www.maxmind.com</a>.
 
+# requires a license key
+# https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
 
 set -e
 trap cleanup EXIT
 
 fail()
 {
-  echo >&2 "$@"
+  echo >&2 -e "$@"
   exit 23
 }
 
 setup()
 {
-  [ "$#" -eq 1 ] || fail "Exactly one argument DESTINATION-FILE must be given."
+  [ "$#" -eq 2 ] || fail "Exactly two arguments must be given:\n * DESTINATION-FILE (with absolute path, mmdb format)\n * GEOIP-LICENSE (See https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/ for details.)"
   DEST_FILE="$1"
+  GEOIP_LICENSE="$2"
   TMP_DIR=`mktemp -d`
   dest_dir=`dirname "$DEST_FILE"`
   [ -d "$dest_dir" ] || mkdir -p "$dest_dir"
@@ -39,9 +42,9 @@ cleanup()
 fetch_and_install()
 {
   cd "$TMP_DIR"
-  curl -s -S -O https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
-  gunzip -q GeoLite2-City.mmdb
-  mv -f GeoLite2-City.mmdb "$DEST_FILE"
+  curl -s -S -o GeoLite2-City.mmdb.tar.gz "https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&suffix=tar.gz&license_key=$GEOIP_LICENSE"
+  tar -mzxf GeoLite2-City.mmdb.tar.gz
+  mv -f GeoLite2-City_*/GeoLite2-City.mmdb "$DEST_FILE"
 }
 
 setup "$@"