add ECS support

CHANGELOG.md

@@ -1,3 +1,6 @@
+## 3.1.0
+  - Add ECS support [#85](https://github.com/logstash-plugins/logstash-filter-csv/pull/85)
+
 ## 3.0.11
   - [DOC] Fixed formatting to improve readability [#84](https://github.com/logstash-plugins/logstash-filter-csv/pull/84)
 

VERSION

@@ -1 +1 @@
-3.0.11
+3.1.0

docs/index.asciidoc

@@ -24,6 +24,11 @@ The CSV filter takes an event field containing CSV data, parses it,
 and stores it as individual fields with optionally-specified field names.
 This filter can parse data with any separator, not just commas.
 
+[id="plugins-{type}s-{plugin}-ecs_metadata"]
+==== Event Metadata and the Elastic Common Schema (ECS)
+When ECS compatibility is disabled, the value is stored in the root level.
+When ECS is enabled, the value is stored in the `target` field.
+
 [id="plugins-{type}s-{plugin}-options"]
 ==== Csv Filter Configuration Options
 
@@ -36,6 +41,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 | <<plugins-{type}s-{plugin}-autogenerate_column_names>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-columns>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-convert>> |<<hash,hash>>|No
+| <<plugins-{type}s-{plugin}-ecs_compatibility>> | <<string,string>>|No
 | <<plugins-{type}s-{plugin}-quote_char>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-separator>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-skip_empty_columns>> |<<boolean,boolean>>|No
@@ -104,6 +110,18 @@ Example:
       }
     }
 
+[id="plugins-{type}s-{plugin}-ecs_compatibility"]
+===== `ecs_compatibility`
+
+* Value type is <<string,string>>
+* Supported values are:
+** `disabled`: does not use ECS-compatible field names
+** `v1`: uses the value in `target` as field name
+
+Controls this plugin's compatibility with the
+{ecs-ref}[Elastic Common Schema (ECS)].
+See <<plugins-{type}s-{plugin}-ecs_metadata>> for detailed information.
+
 [id="plugins-{type}s-{plugin}-quote_char"]
 ===== `quote_char` 
 
@@ -179,6 +197,10 @@ data structure.
 Define target field for placing the data.
 Defaults to writing to the root of the event.
 
+Without a `target`, events are created from each row column at the root level.
+When the `target` is set to a field reference, the column of each row is placed in the target field instead.
+
+This option can be useful to avoid populating unknown fields when a downstream schema such as ECS is enforced.
 
 
 [id="plugins-{type}s-{plugin}-common-options"]

lib/logstash/filters/csv.rb

@@ -1,13 +1,16 @@
 # encoding: utf-8
 require "logstash/filters/base"
 require "logstash/namespace"
+require "logstash/plugin_mixins/ecs_compatibility_support"
 
 require "csv"
 
 # The CSV filter takes an event field containing CSV data, parses it,
 # and stores it as individual fields (can optionally specify the names).
 # This filter can also parse data with any separator, not just commas.
 class LogStash::Filters::CSV < LogStash::Filters::Base
+  include LogStash::PluginMixins::ECSCompatibilitySupport(:disabled, :v1, :v8 => :v1)
+
   config_name "csv"
 
   # The CSV data in the value of the `source` field will be expanded into a
@@ -102,6 +105,15 @@ class LogStash::Filters::CSV < LogStash::Filters::Base
   CONVERTERS.default = lambda {|v| v}
   CONVERTERS.freeze
 
+  def initialize(params)
+    super
+    if ecs_compatibility != :disabled && @target.nil?
+      logger.info('ECS compatibility is enabled but no ``target`` option was specified, it is recommended'\
+                  ' to set the option to avoid potential schema conflicts (if your data is ECS compliant or'\
+                  ' non-conflicting feel free to ignore this message)')
+    end
+  end
+
   def register
     # validate conversion types to be the valid ones.
     bad_types = @convert.values.select do |type|

logstash-filter-csv.gemspec

@@ -23,7 +23,7 @@ Gem::Specification.new do |s|
 
   # Gem dependencies
   s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
-
   s.add_development_dependency 'logstash-devutils'
+  s.add_runtime_dependency 'logstash-mixin-ecs_compatibility_support', '~>1.2'
 end