Convert Filebeat system.auth to ECS (#9138)

* Space. The final frontier.

* Perform all the straightforward renames at once.

- system.auth.hostname => host.hostname
- system.auth.pid => process.pid
- system.auth.user => user.name
- system.auth.program => process.name
- system.auth.ssh.ip => source.ip
- system.auth.ssh.port => source.port
- system.auth.ssh.geoip.* => source.geo.*

* Rename `system.auth.message` to `message`...

Note that there's no log in this format being tested by the integration tests,
at this time.

* Perform a few more renames and type conversion on numeric fields.

* Changelog

* Move `group.*` definitions to a safer haven.

* List the fields that moved in ecs-migration.yml

* Replace previous definitions with aliases for all normalized fields

* Revert the mapping of ssh.signature to event.hash until discussed

* Move the fields not aliased to the top, to promote sanity

* Bring back `dropped_ip` and copy it to `source.ip` when it's there.

Damn, painless scripting is painful

* Fix test failure after introducing aliases.

`'Cannot write to a field alias [system.auth.timestamp].'`

* Add back missing log entry, likely lost during rebase

* Output to group.*, not to the alias

CHANGELOG.asciidoc

@@ -131,6 +131,7 @@ https://github.com/elastic/beats/compare/v6.5.0...v7.0.0-alpha1[View commits]
 - IIS module's user agent string is no longer encoded (`+` replaced with spaces). {pull}9084[9084]
 - Rename many `haproxy.*` fields to map to ECS. {pull}9117[9117]
 - Rename many `nginx.access.*` fields to map to ECS. {pull}9081[9081]
+- Rename many `system.auth.*` fields to map to ECS. {pull}9138[9138]
 
 *Metricbeat*
 

dev-tools/ecs-migration.yml

@@ -105,6 +105,56 @@
   alias: true
   copy_to: false
 
+- from: system.auth.hostname
+  to: host.hostname
+  alias: true
+  copy_to: false
+
+- from: system.auth.pid
+  to: process.pid
+  alias: true
+  copy_to: false
+
+- from: system.auth.groupadd.gid
+  to: group.id
+  alias: true
+  copy_to: false
+
+- from: system.auth.useradd.uid
+  to: user.id
+  alias: true
+  copy_to: false
+
+- from: system.auth.useradd.user
+  to: user.name
+  alias: true
+  copy_to: false
+
+- from: system.auth.ssh.event
+  to: event.action
+  alias: true
+  copy_to: false
+
+- from: system.auth.program
+  to: process.name
+  alias: true
+  copy_to: false
+
+- from: system.auth.ssh.ip
+  to: source.ip
+  alias: true
+  copy_to: false
+
+- from: system.auth.ssh.port
+  to: source.port
+  alias: true
+  copy_to: false
+
+- from: system.auth.ssh.geoip.*
+  to: source.geo.*
+  alias: true
+  copy_to: false
+
 # From Auditbeat's auditd module.
 - from: source.hostname
   to: source.domain

filebeat/docs/fields.asciidoc

@@ -5850,6 +5850,30 @@ Module for parsing system log files.
 --
 type: keyword
 
+--
+
+*`group.id`*::
++
+--
+type: keyword
+
+required: False
+
+ID of the group.
+
+
+--
+
+*`group.name`*::
++
+--
+type: keyword
+
+required: False
+
+Name of the group.
+
+
 --
 
 [float]
@@ -5866,73 +5890,49 @@ Fields from the Linux authorization logs.
 
 
 
-*`system.auth.timestamp`*::
+*`@timestamp`*::
 +
 --
-The timestamp as read from the auth message.
-
+type: alias
 
 --
 
-*`system.auth.hostname`*::
+*`host.hostname`*::
 +
 --
-The hostname as read from the auth message.
-
+type: alias
 
 --
 
-*`system.auth.program`*::
+*`process.name`*::
 +
 --
-The process name as read from the auth message.
-
+type: alias
 
 --
 
-*`system.auth.pid`*::
+*`process.pid`*::
 +
 --
-type: long
-
-The PID of the process that sent the auth message.
-
+type: alias
 
 --
 
-*`system.auth.message`*::
+*`message`*::
 +
 --
-type: text
-
-The message in the log line.
-
+type: alias
 
 --
 
-*`system.auth.user`*::
+*`user.name`*::
 +
 --
-The Unix user that this event refers to.
-
-
---
-
-[float]
-== ssh fields
-
-Fields specific to SSH login events.
-
-
+type: alias
 
-*`system.auth.ssh.event`*::
-+
 --
-The SSH login event. Can be one of "Accepted", "Failed", or "Invalid". "Accepted" means a successful login. "Invalid" means that the user is not configured on the system. "Failed" means that the SSH login attempt has failed.
 
 
---
-
 *`system.auth.ssh.method`*::
 +
 --
@@ -5941,12 +5941,10 @@ The SSH authentication method. Can be one of "password" or "publickey".
 
 --
 
-*`system.auth.ssh.ip`*::
+*`system.auth.ssh.signature`*::
 +
 --
-type: ip
-
-The client IP from where the login attempt was made.
+The signature of the client public key.
 
 
 --
@@ -5961,88 +5959,67 @@ The client IP from SSH connections that are open and immediately dropped.
 
 --
 
-*`system.auth.ssh.port`*::
+*`event.action`*::
 +
 --
-type: long
-
-The client port from where the login attempt was made.
-
+type: alias
 
 --
 
-*`system.auth.ssh.signature`*::
+*`source.ip`*::
 +
 --
-The signature of the client public key.
-
+type: alias
 
 --
 
-[float]
-== geoip fields
-
-Contains GeoIP information gathered based on the `system.auth.ip` field. Only present if the GeoIP Elasticsearch plugin is available and used.
+*`source.port`*::
++
+--
+type: alias
 
+--
 
 
-*`system.auth.ssh.geoip.continent_name`*::
+*`source.geo.continent_name`*::
 +
 --
-type: keyword
-
-The name of the continent.
-
+type: alias
 
 --
 
-*`system.auth.ssh.geoip.city_name`*::
+*`source.geo.country_iso_code`*::
 +
 --
-type: keyword
-
-The name of the city.
-
+type: alias
 
 --
 
-*`system.auth.ssh.geoip.region_name`*::
+*`source.geo.location`*::
 +
 --
-type: keyword
-
-The name of the region.
-
+type: alias
 
 --
 
-*`system.auth.ssh.geoip.country_iso_code`*::
+*`source.geo.region_name`*::
 +
 --
-type: keyword
-
-Country ISO code.
-
+type: alias
 
 --
 
-*`system.auth.ssh.geoip.location`*::
+*`source.geo.city_name`*::
 +
 --
-type: geo_point
-
-The longitude and latitude.
-
+type: alias
 
 --
 
-*`system.auth.ssh.geoip.region_iso_code`*::
+*`source.geo.region_iso_code`*::
 +
 --
-type: keyword
-
-Region ISO code.
-
+type: alias
 
 --
 
@@ -6104,43 +6081,38 @@ Fields specific to events created by the `useradd` command.
 
 
 
-*`system.auth.useradd.name`*::
+*`system.auth.useradd.home`*::
 +
 --
-The user name being added.
-
+The home folder for the new user.
 
 --
 
-*`system.auth.useradd.uid`*::
+*`system.auth.useradd.shell`*::
 +
 --
-type: long
-
-The user ID.
+The default shell for the new user.
 
 --
 
-*`system.auth.useradd.gid`*::
+*`user.name`*::
 +
 --
-type: long
-
-The group ID.
+type: alias
 
 --
 
-*`system.auth.useradd.home`*::
+*`user.id`*::
 +
 --
-The home folder for the new user.
+type: alias
 
 --
 
-*`system.auth.useradd.shell`*::
+*`group.id`*::
 +
 --
-The default shell for the new user.
+type: alias
 
 --
 
@@ -6151,21 +6123,17 @@ Fields specific to events created by the `groupadd` command.
 
 
 
-*`system.auth.groupadd.name`*::
+*`group.name`*::
 +
 --
-The name of the new group.
-
+type: alias
 
 --
 
-*`system.auth.groupadd.gid`*::
+*`group.id`*::
 +
 --
-type: long
-
-The ID of the new group.
-
+type: alias
 
 --