[SIEM] Remove deprecated code from 8.0

[andrewkroh]

I looked through the Security Ingest modules/beats for comments and TODOs indicating that a change was needed for 8.0. This may not be an exhaustive list so if you know of change that should be made for 8.0 please add it hear.

Winlogbeat

• Remove hash.* fields
  

  beats/x-pack/winlogbeat/module/sysmon/config/winlogbeat-sysmon.js

  Lines 428 to 429 in d11d609

  	// TODO: remove in 8.0, see (https://github.com/elastic/beats/issues/18364).
  	evt.Put("hash." + key, value);

• Remove the eventlogging API reader implementation.
  

  beats/winlogbeat/eventlog/eventlogging.go

  Line 281 in d81ef73

  cfgwarn.Deprecate("8.0", "The eventlogging API reader is deprecated.")

Auditbeat

• Remove auditd module event categories [Auditbeat] Field cleanup for 8.0 #28378
  

  beats/auditbeat/module/auditd/audit_linux.go

  Line 579 in bd7414d

  // Remove this block in 8.x

• Remove FIM hash.* fields [Auditbeat] Field cleanup for 8.0 #28378
  

  beats/auditbeat/module/file_integrity/event.go

  Lines 308 to 309 in 56ba9d0

  	// Remove this for 8.x
  	out.MetricSetFields.Put("hash", hashes)

  
  

  beats/auditbeat/module/file_integrity/event_test.go

  Lines 318 to 320 in 56ba9d0

  	// Remove in 8.x
  	assertHasKey(t, fields, "hash.sha1")
  	assertHasKey(t, fields, "hash.sha256")

Packetbeat

• Remove TLS version and province fields Remove deprecated TLS fields #28487

beats/packetbeat/protos/tls/parse.go

Lines 574 to 575 in 00b41c3

	// remove this in 8.x
	"version": cert.Version,

beats/packetbeat/protos/tls/parse.go

Lines 607 to 608 in 00b41c3

	// remove this in 8.x
	{"province", name.Province},

• Remove TLS detailed.client_certificate and detailed.server_certificate in favor of x509 fields Remove deprecated TLS fields #28487

beats/packetbeat/protos/tls/tls.go

Lines 445 to 450 in 00b41c3

	fields.Put("tls.detailed", detailed)
	if cert, ok := detailed["client_certificate"]; ok {
	fields.Put("tls.client.x509", cert)
	}
	if cert, ok := detailed["server_certificate"]; ok {
	fields.Put("tls.server.x509", cert)

• Remove network_traffic ECS category:

beats/packetbeat/pb/event.go

Line 80 in 00b41c3

Category: []string{"network_traffic", "network"},

Filebeat

• Duplicated DNS fields in CoreDNS module

beats/x-pack/filebeat/module/coredns/log/ingest/pipeline-entry.yml

Lines 75 to 102 in 6e69b05

	# The following copies values from dns namespace (ECS) to the coredns
	# namespace to avoid introducing breaking change. This should be removed
	# for 8.0.0. Additionally coredns.dnssec_ok can be removed.
	- set:
	field: coredns.id
	value: '{{dns.id}}'
	ignore_empty_value: true
	- set:
	field: coredns.query.class
	value: '{{dns.question.class}}'
	ignore_empty_value: true
	- set:
	field: coredns.query.name
	value: '{{dns.question.name}}'
	ignore_empty_value: true
	- set:
	field: coredns.query.type
	value: '{{dns.question.type}}'
	ignore_empty_value: true
	- set:
	field: coredns.response.code
	value: '{{dns.response_code}}'
	ignore_empty_value: true
	- script:
	if: ctx.dns?.header_flags != null
	lang: painless
	source: >
	ctx.coredns.response.flags = ctx.dns.header_flags;

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[peasead]

An additional gap for Packetbeat is that event.type is blank.

My thought is that event.type (or event.dataset) would reflect what type currently is and that event.action would carry the start and end values?

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jamiehynds]

@andrewkroh is this issue still relevant for 8.0?

[andrewkroh]

Yes, this is relevant. I went through and checked the boxes for the ones that have been addressed, but the others still should be changed for 8.0.

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[efd6]

@andrewkroh Is this coredns deprecation relevant here? If so, what else is required?

[andrewkroh]

@efd6 Yes, that looks relevant. I've added a bullet point to the description for this. I think what needs done is to remove those processors from the pipeline, remove the related fields from the _meta/fields.yml file, and then run make update to regenerate some files and docs.

[efd6]

@andrewkroh It mentions removing coredns.dnssec_ok, so that should be hosed out too after using it to append the DO flag to dns.header_flags?

[andrewkroh]

Yes, coredns.dnssec_ok can be removed because we have dns.header_flags in the event (an ECS field) and users can check for the DO flag that indicates "DNSSEC answer OK".