Merge tag '2.2.0'

2.2.0 Feature release

Dropped support for Python 3.4.

 ### Core
- `__init__`: Changes to the path-handling, see [User Guide, section _/opt and LSB paths_](docs/User-Guide.md#opt-and-lsb-paths) for more information
  - The environment variable `INTELMQ_ROOT_DIR` can be used to set custom root directories instead of `/opt/intelmq/` (certtools#805) in case of non LSB-path installations.
  - The environment variable `ROOT_DIR` can be used to set custom root directories instead of `/` (certtools#805) in case of LSB-path installations.
- `intelmq.lib.exceptions`: Added `MissingDependencyError` for show error messages about a missing library and how to install it (certtools#1471).
  - Added optional parameter `installed` to show the installed version.
  - Added optional parameter `additional_text` to show arbitrary text.
- Adding more type annotations for core libraries.
- `intelmq.lib.pipeline.Pythonlist.sleep`: Drop deprecated method.
- `intelmq.lib.utils`: `write_configuration`: Append a newline at end of configuration/file to allow proper comparisons & diffs.
- `intelmq.lib.test`: `BotTestCase` drops privileges upon initialization (certtools#1489).
- `intelmq.lib.bot`:
  - New class `OutputBot`:
    - Method `export_event` to format/export events according to the parameters given by the user.
  - `ParserBot`: New methods `parse_json_stream` and `recover_line_json_stream`.
  - `ParserBot.recover_line_json`: Fix format by adding a list around the line data.
  - `Bot.send_message`: In debugging log level, the path to which the message is sent is now logged too.

 ### Bots
- Bots with dependencies: Use of `intelmq.lib.exceptions.MissingDependencyError`.

 #### Collectors
- `intelmq.bots.collectors.misp.collector`: Deprecate parameter `misp_verify` in favor of generic parameter `http_verify_cert`.
- `intelmq.bots.collectors.tcp.collector`: Drop compatibility with Python 3.4.
- `intelmq.bots.collectors.stomp.collector`:
  - Check the stomp.py version and show an error message if it does not match.
  - For stomp.py versions `>= 5.0.0` redirect the `stomp.PrintingListener` output to debug logging.
- `intelmq.bots.collectors.microsoft.collector_azure`: Support current Python library `azure-storage-blob>= 12.0.0`, configuration is incompatible and needs manual change. See NEWS file and bot's documentation for more details.
- `intelmq.bots.collectors.amqp.collector_amqp`: Require `pika` minimum version 1.0.
- `intelmq.bots.collectors.github_api.collector_github_contents_api`: Added (PR#1481).

 #### Parsers
- `intelmq.bots.parsers.autoshun.parser`: Drop compatibility with Python 3.4.
- `intelmq.bots.parsers.html_table.parser`: Drop compatibility with Python 3.4.
- `intelmq.bots.parsers.shadowserver.parser`: Add support for MQTT and Open-IPP feeds (PR#1512, PR#1544).
- `intelmq.bots.parsers.taichung.parser`:
  - Migrate to `ParserBot`.
  - Also parse geolocation information if available.
- `intelmq.bots.parsers.cymru.parser_full_bogons`:
  - Migrate to `ParserBot`.
  - Add last updated information in raw.
- `intelmq.bots.parsers.anubisnetworks.parser`: Add new parameter `use_malware_familiy_as_classification_identifier`.
- `intelmq.bots.parsers.microsoft.parser_ctip`: Compatibility for new CTIP data format used provided by the Azure interface.
- `intelmq.bots.parsers.cymru.parser_cap_program`: Support for `openresolver` type.
- `intelmq.bots.parsers.github_feed.parser`: Added (PR#1481).
- `intelmq.bots.parsers.urlvir.parser`: Removed, as the feed is discontinued (certtools#1537).

 #### Experts
- `intelmq.bots.experts.csv_converter`: Added as converter to CSV.
- `intelmq.bots.experts.misp`: Added (PR#1475).
- `intelmq.bots.experts.modify`: New parameter `maximum_matches`.

 #### Outputs
- `intelmq.bots.outputs.amqptopic`:
  - Use `OutputBot` and `export_event`.
  - Allow formatting the routing key with event data by the new parameter `format_routing_key` (boolean).
- `intelmq.bots.outputs.file`: Use `OutputBot` and `export_event`.
- `intelmq.bots.outputs.files`: Use `OutputBot` and `export_event`.
- `intelmq.bots.outputs.misp.output_feed`: Added, creates a MISP Feed (PR#1473).
- `intelmq.bots.outputs.misp.output_api`: Added, pushes to MISP via the API (PR#1506, PR#1536).
- `intelmq.bots.outputs.elasticsearch.output`: Dropped ElasticSearch version 5 compatibility, added version 7 compatibility (certtools#1513).

 ### Documentation
- Document usage of the `INTELMQ_ROOT_DIR` environment variable.
- Added document on MISP integration possibilities.
- Feeds:
  - Added "Full Bogons IPv6" feed.
  - Remove discontinued URLVir Feeds (certtools#1537).

 ### Packaging
- `setup.py` do not try to install any data to `/opt/intelmq/` as the behavior is inconsistent on various systems and with `intelmqsetup` we have a tool to create the structure and files anyway.
- `debian/rules`:
  - Provide a blank state file in the package.
- Patches:
  - Updated `fix-intelmq-paths.patch`.

 ### Tests
- Travis: Use `intelmqsetup` here too.
  - Install required build dependencies for the Debian package build test.
  - This version is no longer automatically tested on Python `<` 3.5.
  - Also run the tests on Python 3.8.
  - Run the Debian packaging tests on Python 3.5 and the code-style test on 3.8.
- Added tests for the new bot `intelmq.bots.outputs.misp.output_feed` (certtools#1473).
- Added tests for the new bot `intelmq.bots.experts.misp.expert` (certtools#1473).
- Added tests for `intelmq.lib.exceptions`.
- Added tests for `intelmq.lib.bot.OutputBot` and `intelmq.lib.bot.OutputBot.export_event`.
- Added IPv6 tests for `intelmq.bots.parsers.cymru.parser_full_bogons`.
- Added tests for `intelmq.lib.bot.ParserBot`'s new methods `parse_json_stream` and `recover_line_json_stream`.
- `intelmq.tests.test_conf`: Set encoding to UTF-8 for reading the `feeds.yaml` file.

 ### Tools
- `intelmqctl`:
  - `upgrade-config`:
    - Allow setting the state file location with the `--state-file` parameter.
    - Do not require a second run anymore, if the state file is newly created (certtools#1491).
    - New parameter `no_backup`/`--no-backup` to skip creation of `.bak` files for state and configuration files.
  - Only require `psutil` for the `IntelMQProcessManager`, not for process manager independent calls like `upgrade-config` or `check`.
  - Add new command `debug` to output some information for debugging. Currently implemented:
    - paths
    - environment variables
  - `IntelMQController`: New argument `--no-file-logging` to disable logging to file.
  - If dropping privileges does not work, `intelmqctl` will now abort (certtools#1489).
- `intelmqsetup`:
  - Add argument parsing and an option to skip setting file ownership, possibly not requiring root permissions.
  - Call `intelmqctl upgrade-config` and add argument for the state file path (certtools#1491).
- `intelmq_generate_misp_objects_templates.py`: Tool to create a MISP object template (certtools#1470).
- `intelmqdump`: New parameter `-t` or `--truncate` to optionally give the maximum length of `raw` data to show, 0 for no truncating.

 ### Contrib
- Added `development-tools`.
- ElasticSearch: Dropped version 5 compatibility, added version 7 compatibility (certtools#1513).
- Malware Name Mapping Downloader:
  - New parameter `--mwnmp-ignore-adware`.
  - The parameter `--add-default` supports an optional parameter to define the default value.

 ### Known issues
- Bots started with IntelMQ-Manager stop when the webserver is restarted. (certtools#952).
- Corrupt dump files when interrupted during writing (certtools#870).

.travis.yml

@@ -5,28 +5,34 @@ env:
   - requirements=true
   - requirements=false
 python:
-  - 3.4
   - 3.5
   - 3.6
   - 3.7
+  - 3.8
 matrix:
   include:
-  - python: 3.4
+  - python: 3.5
     env: mode=debian
-  - python: 3.6
+  - python: 3.8
     env: mode=codestyle
+before_install:
+  - if [[ -v requirements ]]; then curl -s -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.6.1-amd64.deb; fi
+  - if [[ -v requirements ]]; then sudo dpkg -i --force-confnew elasticsearch-7.6.1-amd64.deb; fi
+  - if [[ -v requirements ]]; then sudo sed -i.old 's/-Xms1g/-Xms128m/' /etc/elasticsearch/jvm.options; fi
+  - if [[ -v requirements ]]; then sudo sed -i.old 's/-Xmx1g/-Xmx128m/' /etc/elasticsearch/jvm.options; fi
+  - if [[ -v requirements ]]; then echo -e '-XX:+DisableExplicitGC\n-Djdk.io.permissionsUseCanonicalPath=true\n-Dlog4j.skipJansi=true\n-server\n' | sudo tee -a /etc/elasticsearch/jvm.options; fi
+  - if [[ -v requirements ]]; then sudo chown -R elasticsearch:elasticsearch /etc/default/elasticsearch; fi
+  - if [[ -v requirements ]]; then sudo systemctl start elasticsearch; fi
 install:
   - set -e
   - if [[ -v requirements ]]; then sudo apt-get install polipo lighttpd; fi
-  - if [[ $mode == debian ]]; then sudo apt-get install dpkg-dev dh-python python-setuptools python3-setuptools python3-all debhelper quilt fakeroot dh-systemd safe-rm; fi
+  - if [[ $mode == debian ]]; then sudo apt-get install dpkg-dev dh-python python-setuptools python3-setuptools python3-all debhelper quilt fakeroot dh-systemd safe-rm; pip3 install requests; pip3 install redis; pip3 install dnspython; pip3 install psutil; pip3 install python-dateutil; pip3 install termstyle; pip3 install pytz; pip3 install typing; fi
   - if [[ $requirements == true ]]; then for file in intelmq/bots/*/*/REQUIREMENTS.txt; do pip install -r $file; done; fi
-  - if [[ $TRAVIS_PYTHON_VERSION < '3.5' ]]; then sudo pip install typing; fi
-  - if [[ -v requirements ]]; then pip install Cerberus!=1.3 codecov pyyaml; fi
+  - if [[ -v requirements ]]; then pip install Cerberus!=1.3 codecov pyyaml requests_mock; fi
   - if [[ $mode == codestyle ]]; then pip install pycodestyle; fi
   - if [[ -v requirements ]]; then sudo sed -i '/^Defaults\tsecure_path.*$/ d' /etc/sudoers; fi
   - if [[ -v requirements ]]; then sudo pip install .; fi
-  - if [[ -v requirements ]]; then sudo cp /opt/intelmq/etc/examples/* /opt/intelmq/etc/; fi
-  - if [[ -v requirements ]]; then sudo chown -R $USER /opt/intelmq/; fi
+  - if [[ -v requirements ]]; then sudo intelmqsetup --skip-ownership; fi
 before_script:
   - if [[ $requirements == true ]] ; then psql -c "CREATE USER intelmq WITH SUPERUSER" -U postgres; fi
   - if [[ $requirements == true ]] ; then psql -c "CREATE DATABASE intelmq" -U intelmq template1; fi
@@ -44,6 +50,7 @@ before_script:
   - if [[ $mode == debian ]]; then tar -xzf ../intelmq_$debversion.debian.tar.gz; fi
   - if [[ $mode == debian ]]; then popd; fi
   - if [[ -v requirements ]]; then sudo cp intelmq/tests/assets/* /var/www/html/ && sudo touch /var/www/html/$(date +%Y).txt; fi
+  - if [[ $requirements == true ]]; then sudo bash -c 'echo "[rabbitmq_management]." > /etc/rabbitmq/enabled_plugins' && sudo systemctl restart rabbitmq-server; fi
 script:
   - if [[ $requirements == true ]]; then TZ=utc INTELMQ_TEST_DATABASES=1 INTELMQ_TEST_LOCAL_WEB=1 INTELMQ_TEST_EXOTIC=1 nosetests --with-coverage --cover-package=intelmq --cover-branches; find contrib/ -name "test*.py" -exec nosetests {} \+; elif [[ $requirements == false ]]; then INTELMQ_TEST_LOCAL_WEB=1 nosetests --with-coverage --cover-package=intelmq --cover-branches; fi
   - if [[ $mode == codestyle ]]; then pycodestyle intelmq/{bots,lib,bin}; fi
@@ -53,7 +60,6 @@ script:
 services:
   - redis-server
   - postgresql
-  - elasticsearch
   - mongodb
   - rabbitmq
 after_success:

NEWS.md

@@ -3,6 +3,84 @@ NEWS
 
 See the changelog for a full list of changes.
 
+2.2.0 Feature release (2020-06-17)
+----------------------------------
+
+### Requirements
+- IntelMQ no longer supports Python 3.4, Python `>=` 3.5 is required.
+  CentOS 7 (with EPEL) provides both Python 3.4 and Python 3.6. If IntelMQ was installed with Python 3.4, the code needs to be re-installed with Python 3.6 and removed for Python 3.4. Application data is compatible. To install needed packages: `yum install python36 python36-devel python36-requests`.
+- The *AMQP collector* requires the `pika` library minimum version 1.0.
+
+### Configuration
+
+#### ElasticSearch Output Bot
+The ElasticSearch Output bot does no longer support (only) ElasticSearch version 5, but only version 7 (#1513).
+
+#### Microsoft Azure Collector Bot
+The Bot has been majorly changed to support the current Azure Python library `azure-storage-blob>=12.0.0`.
+This also changes the required configuration parameters. The new required parameters are:
+
+* `redis_cache_db`: 5
+* `redis_cache_host`: `"127.0.0.1"`
+* `redis_cache_password`: null, depending on your Redis server configuration
+* `redis_cache_port`: 6379, depending on your Redis server configuration
+* `redis_cache_ttl`: 864000 (10 days), depending on how old fast the data you are fetching is removed from the storage
+* `connection_string`: connection string as given by Microsoft, includes endpoint and authentication information
+* `container_name`: name of the container to connect to
+
+The previous parameters `account_name`, `account_key` and `delete` are not supported anymore.
+
+#### URLVir Feeds and Parser
+All URLVir feeds have been discontinued. The URLVir Parser has been removed.
+The `intelmqctl upgrade-config` command warns if you have these feed and the bot in use.
+
+
+2.1.3 Bugfix release (2020-05-26)
+---------------------------------
+
+### Requirements
+The python library `requests` is required by the core.
+
+### Harmonization
+The regular expression of the field `protocol.transport` has been updated to accommodate the value `nvp-ii`.
+`intelmqctl upgrade-config` handles the change to automatically upgrade your configuration.
+
+#### Taichung feed
+The Taichung feed "Netflow (Recent 30)" with URL `https://www.tc.edu.tw/net/netflow/lkout/recent/30` is no longer available and gives an error 500.
+As a drop-in replacement the Parser as well as the Feed documentation are now adapted to the full feed available at `https://www.tc.edu.tw/net/netflow/lkout/recent/`.
+The `intelmqctl upgrade-config` command takes care of this change.
+
+#### Abuse.ch Zeus Tracker Feed
+The Abuse.ch Zeus Tracker has been discontinued on 2019-07-08. The `intelmqctl upgrade-config` command warns if you have this feed in use.
+
+#### Abuse.ch Ransomware Tracker Feed
+The Abuse.ch Ransomware Tracker has been discontinued on 2019-12-08. The `intelmqctl upgrade-config` command warns if you have this feed in use.
+
+#### Bitcash.cz Feed
+The Bitcash.cz Banned IPs / Blocklist feed previously available under `https://bitcash.cz/misc/log/blacklist` is no longer available. The `intelmqctl upgrade-config` command warns if you have this feed in use.
+
+#### Fraunhofer DDoS Attack Feed
+The Fraunhofer DDoS Attack feed previously available under `https://feed.caad.fkie.fraunhofer.de/ddosattackfeed/` is no longer available. The `intelmqctl upgrade-config` command warns if you have this feed in use.
+
+#### Bambenek Feeds
+Many Bambenek feeds require a license now and URLs have changed. See https://osint.bambenekconsulting.com/feeds/ for more information. The `intelmqctl upgrade-config` command also warns if you have previously documents feeds in use and migrates the URL for the DGA domain feed.
+
+#### Nothink Feeds and Parser
+All Nothink Honeypot feeds have been discontinued and current the data available covers the time until 2019. The Nothink Parser has been removed.
+The `intelmqctl upgrade-config` command warns if you have these feed and the bot in use.
+
+
+2.1.2 Bugfix release (2020-01-28)
+---------------------------------
+
+#### MaxMind GeoIP
+MaxMind requires a registration before being able to download the GeoLite2 database starting with 2019-12-30: https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/
+If the provided `update-geoip-data` script is used, the license key can now be set second parameter.
+
+### Libraries
+When using MISP integrations, make sure your currently installed PyMISP version supports the installed Python version. Any PyMISP version newer than 2.4.119.1 requires Python 3.6 or newer.
+
+
 2.1.1 Bugfix release (2019-11-11)
 ---------------------------------
 

README.rst

@@ -54,26 +54,19 @@ Table of Contents
 -----------------
 
 1. `How to Install <#how-to-install>`__
-2. `Developers Guide <#developers-guide>`__
-3. `User Guide <#user-guide>`__
-4. `IntelMQ Manager <#intelmq-manager>`__
+2. `User Guide <#user-guide>`__
+3. `IntelMQ Manager and more tools <#intelmq-manager-and-more-tools>`__
+4. `How to Participate <#how-to-participate>`__
 5. `Incident Handling Automation
    Project <#incident-handling-automation-project>`__
-6. `Data Harmonization <#data-harmonization>`__
-7. `How to Participate <#how-to-participate>`__
-8. `Licence <#licence>`__
-9. `Funded by <#funded-by>`__
+6. `Licence <#licence>`__
+7. `Funded by <#funded-by>`__
 
 How to Install
 --------------
 
 See `INSTALL <docs/INSTALL.md>`__.
 
-Developers Guide
-----------------
-
-See `Developers Guide <docs/Developers-Guide.md>`__.
-
 User Guide
 ----------
 
@@ -84,37 +77,38 @@ If you know additional feeds and how to parse them, please contribute your code
 For support questions please use the intelmq-users mailing list:
 https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-users
 
-IntelMQ Manager
----------------
+IntelMQ use the Data Harmonization. Please read `this
+document <docs/Data-Harmonization.md>`__ for more details.
+
+IntelMQ Manager and more tools
+------------------------------
 
 Check out this graphical
 `tool <https://github.com/certtools/intelmq-manager>`__ and easily
 manage an IntelMQ system.
 
+More tools can be found in the `ecosystem documentation <docs/Ecosystem.md>`__.
+
+How to participate
+------------------
+
+IntelMQ is a community project depending on your contributions. Please consider sharing your work.
+
+- Have a look at our `Developers Guide <docs/Developers-Guide.md>`__ for documentation.
+- Subscribe to the Intelmq-dev Mailing list to get answers to your development questions:
+  https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev
+- The `Github issues <github.com/certtools/intelmq/issues/>`__ lists all the open feature requests, bug reports and ideas.
+- Looking for ideas which additional feeds you could add support for? The `Feeds whishlist <docs/Feeds-wishlist.md>`__ is the list you are looking for.
+- Contribute code with pull requests (The `Github help <https://help.github.com/>`__ can be useful if you are not familiar with the system yet).
+- Some developers are also on IRC: `channel #intelmq on irc.freenode.net <ircs://chat.freenode.net:6697/intelmq>`__.
+
 Incident Handling Automation Project
 ------------------------------------
 
 - **URL:**
   http://www.enisa.europa.eu/activities/cert/support/incident-handling-automation
 - **Mailing-list:** ihap@lists.trusted-introducer.org
 
-Data Harmonization
-------------------
-
-IntelMQ use the Data Harmonization. Please read `this
-document <docs/Data-Harmonization.md>`__ for more details.
-
-How to participate
-------------------
-
-- Subscribe to the Intelmq-dev Mailing list:
-  https://lists.cert.at/cgi-bin/mailman/listinfo/intelmq-dev (for
-  developers)
-- Watch out for our regular developers conf call
-- IRC: server: irc.freenode.net, channel: #intelmq
-- Via github issues
-- Via Pull requests (please do read help.github.com first)
-
 Licence
 -------
 

contrib/README.md

@@ -4,11 +4,12 @@
 This directory contains contributed scripts which are helpful for maintaining an intelmq instance.
 
 * **cron-jobs**: cron job files for pulling in newer versions of supporting databases such as pyasn
-* **logcheck**: logcheck ruleset
+* **logcheck**: logcheck ruleset to filter logs for error messages
 * **prettyprint**: prints the json output for file-output bot prettyly
 * **config-backup**: simple Makefile for doing a `make backup` inside of /opt/intelmq in order to preserve the latest configs
 * **logrotate**: an example scrpt for Debian's /etc/logrotate.d/ directory.
 * **check_mk**: Scripts for monitoring an IntelMQ instance with Check_MK.
+* **development-tools**: Tools useful for development
 
 ## Outdated
 The following scripts are out of date but are left here for reference. TODO: adapt to current version

contrib/bash-completion/intelmqctl

@@ -18,7 +18,7 @@ _intelmqctl ()
     #echo "posice: $COMP_CWORD $COMP_WORDS";
     case $COMP_CWORD in
         1)
-            opts="start stop restart reload run status log clear list check enable disable upgrade-config";
+            opts="start stop restart reload run status log clear list check enable disable upgrade-config debug";
             COMPREPLY=($(compgen -W "${opts} ${generic_pre} ${generic_post}" -- ${cur}));
             return 0
         ;;

contrib/development-tools/bots-feeds.sh

@@ -0,0 +1,13 @@
+#!/bin/bash
+
+echo "Bots:"
+jq '.Collector | keys | length' intelmq/bots/BOTS
+jq '.Parser | keys | length' intelmq/bots/BOTS
+jq '.Expert | keys | length' intelmq/bots/BOTS
+jq '.Output | keys | length' intelmq/bots/BOTS
+
+echo "Feeds:"
+egrep -c '^    [^ ]' intelmq/etc/feeds.yaml
+echo "Shadowserver:"
+python3 -c "import intelmq.bots.parsers.shadowserver.config; print(len(intelmq.bots.parsers.shadowserver.config.mapping))"
+

contrib/elasticsearch/README.md

@@ -18,7 +18,7 @@ pip3 install elasticsearch
 ```
 usage: elasticmapper [-h] --harmonization-file <filepath>
                      [--harmonization-fallback] [--host <ip>] [--index INDEX]
-                     [--index-type INDEX_TYPE] [--output <filepath>]
+                     [--output <filepath>]
 
 Elastic Mapper tool
 
@@ -30,8 +30,6 @@ optional arguments:
                         harmonization fallback to `text` type
   --host <ip>           elasticsearch server IP
   --index INDEX         elasticsearch index
-  --index-type INDEX_TYPE
-                        elasticsearch index type
   --index-template      save the mapping as a template for newly-created indices
   --output <filepath>   write mapping to file
 ```
@@ -41,30 +39,30 @@ optional arguments:
 #### Send only to Elasticsearch
 
 ```
-elasticmapper --harmonization-file=intelmq/intelmq/etc/harmonization.conf --index=intelmq --index-type=events --host=127.0.0.1
+elasticmapper --harmonization-file=intelmq/intelmq/etc/harmonization.conf --index=intelmq --host=127.0.0.1
 ```
 
 #### Write only to output file
 
 ```
-elasticmapper --harmonization-file=intelmq/intelmq/etc/harmonization.conf --index=intelmq --index-type=events --output=/tmp/mapping.txt
+elasticmapper --harmonization-file=intelmq/intelmq/etc/harmonization.conf --index=intelmq --output=/tmp/mapping.txt
 ```
 
 #### Send to Elasticsearch and write to output file
 ```
-elasticmapper --harmonization-file=intelmq/intelmq/etc/harmonization.conf --index=intelmq --index-type=events --output=/tmp/mapping.txt --host=127.0.0.1
+elasticmapper --harmonization-file=intelmq/intelmq/etc/harmonization.conf --index=intelmq --output=/tmp/mapping.txt --host=127.0.0.1
 ```
 
 #### Send to Elasticsearch as a template (see https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html)
 
 ```
-elasticmapper --harmonization-file=intelmq/intelmq/etc/harmonization.conf --index=intelmq --index-type=events --host=127.0.0.1 --index-template
+elasticmapper --harmonization-file=intelmq/intelmq/etc/harmonization.conf --index=intelmq --host=127.0.0.1 --index-template
 ```
 
 #### Harmonization fallback
 
 Revert to the default 'text' type in the generated mapping for any fields which have unrecognizable field types.
 
 ```
-elasticmapper --harmonization-file=intelmq/intelmq/etc/harmonization.conf --index=intelmq --index-type=events --output=/tmp/mapping.txt --host=127.0.0.1 --harmonization-fallback
+elasticmapper --harmonization-file=intelmq/intelmq/etc/harmonization.conf --index=intelmq --output=/tmp/mapping.txt --host=127.0.0.1 --harmonization-fallback
 ```

contrib/elasticsearch/elasticmapper

@@ -62,18 +62,15 @@ def mapping_properties_from_harmonization(harmonization, replacement_char):
     return __mapping_properties_from_harmonization(properties), err
 
 
-def create_mapping(harmonization, index_type, replacement_char):
+def create_mapping(harmonization, replacement_char):
 
     config = {"enabled": False}
 
     properties, err = mapping_properties_from_harmonization(harmonization, replacement_char)
 
     data = {
         "mappings": {
-            index_type: {
-                "_all": config,
-                "properties": properties
-            }
+            "properties": properties
         }
     }
 
@@ -176,13 +173,6 @@ if __name__ == "__main__":
                         required=False,
                         help='elasticsearch index name, or template name if using a template')
 
-    parser.add_argument('--index-type',
-                        action="store",
-                        dest="index_type",
-                        default="events",
-                        required=False,
-                        help='elasticsearch index type')
-
     parser.add_argument('--index-template',
                         action="store_true",
                         dest="index_template",
@@ -209,7 +199,7 @@ if __name__ == "__main__":
     with open(arguments.harmonization_file) as fp:
         harmonization = json.load(fp)
 
-    data, err = create_mapping(harmonization, arguments.index_type, arguments.replacement_char)
+    data, err = create_mapping(harmonization, arguments.replacement_char)
 
     if err:
         if arguments.harmonization_fallback: