[ECS] Upgrade modules to 1.7

[marc-gr]

Required changes to upgrade beats modules to 1.7:

• After 1.7 is released, update ecs dependency to 1.7 e.g.: [ECS] Update ecs to 1.6.0 #20792

Using https://github.com/elastic/ecs-dev/issues/199 as reference:

Experimental

Use of basic types with wildcard elastic/dev#1508 elastic/ecs#970:
(important note: Going with option 2 will require each Beat to implement this fallback mechanism to replace wildcard to keyword, when posting an index template to Elasticsearch.)

• Auditbeat (Upgrade to ECS 1.7.0 #22571)
• Metricbeat (Upgrade to ECS 1.7.0 #22571)
• Heartbeat (Upgrade to ECS 1.7.0 #22571)
• Packetbeat (Upgrade to ECS 1.7.0 #22571)
• Winlogbeat (Upgrade to ECS 1.7.0 #22571)
• Auditbeat (Upgrade to ECS 1.7.0 #22571)
• Packetbeat (Upgrade to ECS 1.7.0 #22571)
• Winlogbeat (Upgrade to ECS 1.7.0 #22571)
• Filebeat auditd (Upgrade to ECS 1.7.0 #22571)
• Filebeat rsa2elk modules (Upgrade to ECS 1.7.0 #22571)
• Filebeat checkpoint firewall (Upgrade to ECS 1.7.0 #22571)
• Filebeat cisco asa (Upgrade to ECS 1.7.0 #22571)
• Filebeat cef (Upgrade to ECS 1.7.0 #22571)
• Filebeat cisco ftd (Upgrade to ECS 1.7.0 #22571)
• Filebeat cisco umbrella (Upgrade to ECS 1.7.0 #22571)
• Filebeat citrix netscaler (Upgrade to ECS 1.7.0 #22571)
• Filebeat f5 (Upgrade to ECS 1.7.0 #22571)
• Filebeat crowdstrike falcon (Upgrade to ECS 1.7.0 #22571)
• Filebeat fortinet (Upgrade to ECS 1.7.0 #22571)
• Filebeat googlecloud audit (Upgrade to ECS 1.7.0 #22571)
• Filebeat microsoft (Upgrade to ECS 1.7.0 #22571)
• Filebeat Gsuite (Upgrade to ECS 1.7.0 #22571)
• Filebeat o365 (Upgrade to ECS 1.7.0 #22571)
• Filebeat zoom (Upgrade to ECS 1.7.0 #22571)
• Filebeat okta (Upgrade to ECS 1.7.0 #22571)
• Filebeat aws cloudtrail (Upgrade to ECS 1.7.0 #22571)
• Filebeat barracuda (Upgrade to ECS 1.7.0 #22571)
• Filebeat s3 (Upgrade to ECS 1.7.0 #22571)
• Filebeat juniper (Upgrade to ECS 1.7.0 #22571)
• Filebeat netscout (Upgrade to ECS 1.7.0 #22571)
• Filebeat panw (Upgrade to ECS 1.7.0 #22571)
• Filebeat snort (Upgrade to ECS 1.7.0 #22571)
• Filebeat sonicwall (Upgrade to ECS 1.7.0 #22571)
• Filebeat sophos (Upgrade to ECS 1.7.0 #22571)
• Filebeat zeek (Upgrade to ECS 1.7.0 #22571)

Multiple users in an event elastic/ecs#914

• Auditbeat
• Packetbeat
• Winlogbeat
• Filebeat auditd
• Filebeat rsa2elk modules
• Filebeat checkpoint firewall
• Filebeat cisco asa
• Filebeat cef
• Filebeat cisco ftd
• Filebeat cisco umbrella
• Filebeat citrix netscaler
• Filebeat f5
• Filebeat crowdstrike falcon
• Filebeat fortinet clientendpoint
• Filebeat fortinet firewall
• Filebeat fortinet fortimail
• Filebeat fortinet fortimanager
• Filebeat googlecloud audit
• Filebeat microsoft
• Filebeat Gsuite
• Filebeat o365
• Filebeat zoom
• Filebeat okta
• Filebeat aws cloudtrail
• Filebeat barracuda
• Filebeat s3
• Filebeat juniper
• Filebeat netscout
• Filebeat panw
• Filebeat snort
• Filebeat sonicwall
• Filebeat sophos
• Filebeat zeek irc
• Filebeat zeek kerberos
• Filebeat zeek ntlm
• Filebeat zeek radius
• Filebeat zeek webhook

Additions in 1.7:

New ingress and egress allowed values for network.direction elastic/ecs#945:

• Auditbeat auditd (Add ingress/egress and configuration categorization go-libaudit#80) ([Auditbeat] Upgrade to latest go-libaudit to support ECS 1.7 #23000)
• Auditbeat system/socket ([Auditbeat] system/socket: Use ingress/egress for network direction #22991)
• Metricbeat ([Metricbeat] Use egress/ingress instead of inbound/outbound for system/socket metricset #22992)
• Packetbeat ([Packetbeat] Update ingress/egress traffic directionality #22996)
• Winlogbeat ([Winlogbeat] Use ingress/egress instead of inbound/outbound #22997)
• Filebeat auditd ([Filebeat] Use ingress/egress for crowdstrike and auditd modules #23041)
• Filebeat crowdstrike falcon ([Filebeat] Use ingress/egress for crowdstrike and auditd modules #23041)
• Filebeat netflow ([Filebeat] Add network.direction to netflow/log fileset #23052)
• Filebeat checkpoint firewall ([Filebeat] Allow cef and checkpoint modules to override network directionality based off of zones #23066)
• Filebeat cef ([Filebeat] Allow cef and checkpoint modules to override network directionality based off of zones #23066)
• Filebeat cisco asa ([Filebeat] Allow cisco/asa and cisco/ftd modules to override network directionality based off of zones #23068)
• Filebeat cisco ftd ([Filebeat] Allow cisco/asa and cisco/ftd modules to override network directionality based off of zones #23068)
• Filebeat fortinet firewall ([Filebeat] Add fortinet/firewall network direction override based on interface #23072)
• Filebeat cisco umbrella (waiting on CIDR matching processors/painless support in elasticsearch Painless convenience function for matching IP addresses elasticsearch#60668)
• Filebeat googlecloud firewall ([Filebeat] Add network.direction by specifying internal_networks to gcp module #23081)
• Filebeat googlecloud vpcflow ([Filebeat] Add network.direction by specifying internal_networks to gcp module #23081)
• Filebeat panw panos ([Filebeat] panos config option to set internal/external zones #22998)
• Filebeat sophos xg ([Filebeat] improve logic for network.direction in sophos xg fileset #22973)
• Filebeat zeek connection ([Filebeat] zeek ecs 1.7 updates for network.direction #22967)
• Filebeat rsa2elk modules (@adriansr) (need to add individual module configuration support, see Add network.direction classification to rsa2elk modules #23114)

HTTP request/response mime type elastic/ecs#944:
(important note: needs to analyze body)

• Packetbeat ([Processors] Mime-Type Detection #22940)
• Hearbeat ([Heartbeat] Add mime type detection #22976)
• Filebeat elasticsearch ([Filebeat] Add mime type detection for Elasticsearch module #22975)
• Filebeat o365 (doesn't capture request/response body)
• Filebeat gsuite (doesn't capture request/response body)
• Filebeat suricata eve (doesn't capture request/response body)
• Filebeat rsa2elk modules (doesn't capture request/response body)
• Filebeat checkpoint firewall (doesn't capture request/response body)
• Filebeat cef (doesn't capture request/response body)
• Filebeat cisco ftd (doesn't capture request/response body)
• Filebeat cisco umbrella (doesn't capture request/response body)
• Filebeat fortinet clientendpoint (doesn't capture request/response body)
• Filebeat fortinet firewall (doesn't capture request/response body)
• Filebeat fortinet fortimail (doesn't capture request/response body)
• Filebeat fortinet fortimanager (doesn't capture request/response body)
• Filebeat panw panos (doesn't capture request/response body)
• Filebeat sophos xg (doesn't capture request/response body)
• Filebeat zeek http (doesn't capture request/response body)

New allowed value configuration for event.category elastic/ecs#963:

• Winlogbeat ([Winlogbeat] Add configuration category and more security events #22988)
• Auditbeat system (no configuration events to be classified)
• Auditbeat auditd (Add ingress/egress and configuration categorization go-libaudit#80) ([Auditbeat] Upgrade to latest go-libaudit to support ECS 1.7 #23000)
• Filebeat auditd ([Filebeat] Update event categorization for configuration events for auditd, gsuite, o365, and zoom #23010)
• Filebeat GSuite ([Filebeat] Update event categorization for configuration events for auditd, gsuite, o365, and zoom #23010)
• Filebeat zoom ([Filebeat] Update event categorization for configuration events for auditd, gsuite, o365, and zoom #23010)
• Filebeat o365 ([Filebeat] Update event categorization for configuration events for auditd, gsuite, o365, and zoom #23010)
• Filebeat Okta (no configuration events to be classified)
• Filebeat microsoft (check if rsa2elk microsoft modules need updating)
• Filebeat rsa2elk modules (currently we don't do any event categorization)

Add subdomain domain breakdown across all domain breakdowns (currently only in dns) elastic/ecs#981:

• Packetbeat (seems done for dns)
• Winlogbeat ([Winlogbeat] Add subdomain value for sysmon module #22999)
• Filebeat suricata eve ([Filebeat] Enrich subdomain information for suricata and zeek #23011)
• Filebeat zeek dns ([Filebeat] Enrich subdomain information for suricata and zeek #23011)
• Filebeat rsa2elk modules (@adriansr)

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[webmat]

@marc-gr If the team would rather tackle only 1/2 of the experimental changes (e.g. only wildcard), you're welcome to do so.

Me or @ebeahan can help you get the artifacts you need, to only grab the relevant experimental changes, rather than both changes at once.

[marc-gr]

@jamiehynds maybe can give some input about what we want to do with the experimental changes

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[jamiehynds]

@jamiehynds maybe can give some input about what we want to do with the experimental changes

Discussed with @epixa and @andrewkroh yesterday. Adopting multi-user and wildcard is something we'd like to do. Once we have a clearer picture on affected modules and level of effort, we can prioritise which modules to focus on for 7.11.

[andrewstucki]

Closing this since we're done and moving on to 1.8 upgrade