Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable host fields for "cloud", panw, cef modules #18223

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,13 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Improve ECS field mappings in panw module. event.outcome now only contains success/failure per ECS specification. {issue}16025[16025] {pull}17910[17910]
- Improve ECS categorization field mappings for nginx module. http.request.referrer is now lowercase & http.request.referrer only populated when nginx sets a value {issue}16174[16174] {pull}17844[17844]
- Improve ECS field mappings in santa module. move hash.sha256 to process.hash.sha256 & move certificate fields to santa.certificate . {issue}16180[16180] {pull}17982[17982]
- With the default configuration the cloud modules (aws, azure, googlecloud, o365, okta)
will no longer send the `host` field that contains information about the host Filebeat is
running on. This is because the `host` field specifies the host on which the event
happened. {issue}13920[13920] {pull}18223[18223]
- With the default configuration the cef and panw modules will no longer send the `host`
field. You can revert this change by configuring tags for the module and omitting
`forwarded` from the list. {issue}13920[13920] {pull}18223[18223]

*Heartbeat*

Expand Down Expand Up @@ -325,6 +332,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Improve ECS categorization field mappings in system module. {issue}16031[16031] {pull}18065[18065]
- Change the `json.*` input settings implementation to merge parsed json objects with existing objects in the event instead of fully replacing them. {pull}17958[17958]
- Improve ECS categorization field mappings in osquery module. {issue}16176[16176] {pull}17881[17881]
- Added `observer.vendor`, `observer.product`, and `observer.type` to PANW module events. {pull}18223[18223]
- The `logstash` module can now automatically detect the log file format (JSON or plaintext) and process it accordingly. {issue}9964[9964] {pull}18095[18095]

*Heartbeat*
Expand Down
7 changes: 7 additions & 0 deletions filebeat/_meta/config/processors.yml.tmpl
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
{{header "Processors"}}
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
6 changes: 6 additions & 0 deletions filebeat/docs/modules/cef.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@ The UDP port to listen for syslog traffic. Defaults to `9003`

NOTE: Ports below 1024 require Filebeat to run as root.

*`var.tags`*::

A list of tags to include in events. Including `forwarded` indicates that the
events did not originate on this host and causes `host.name` to not be added to
events. Defaults to `[cef, forwarded]`.

[float]
==== Forcepoint NGFW Security Management Center

Expand Down
7 changes: 2 additions & 5 deletions filebeat/filebeat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -172,16 +172,13 @@ output.elasticsearch:
#ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
- add_host_metadata: ~
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~


# ================================== Logging ===================================

# Sets log level. The default log level is info.
Expand Down
13 changes: 13 additions & 0 deletions filebeat/fileset/fileset.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ import (
"io/ioutil"
"os"
"path/filepath"
"reflect"
"runtime"
"strings"
"text/template"
Expand Down Expand Up @@ -290,6 +291,18 @@ func getTemplateFunctions(vars map[string]interface{}) (template.FuncMap, error)
}

return template.FuncMap{
"inList": func(collection []interface{}, item string) bool {
for _, h := range collection {
if reflect.DeepEqual(item, h) {
return true
}
}
return false
},
"tojson": func(v interface{}) (string, error) {
bytes, err := json.Marshal(v)
return string(bytes), err
},
"IngestPipeline": func(shortID string) string {
return formatPipelineID(
builtinVars["prefix"].(string),
Expand Down
3 changes: 2 additions & 1 deletion filebeat/fileset/fileset_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,7 @@ func TestGetTemplateFunctions(t *testing.T) {
templateFunctions, err := getTemplateFunctions(vars)
assert.NoError(t, err)
assert.IsType(t, template.FuncMap{}, templateFunctions)
assert.Len(t, templateFunctions, 1)
assert.Contains(t, templateFunctions, "inList")
assert.Contains(t, templateFunctions, "tojson")
assert.Contains(t, templateFunctions, "IngestPipeline")
}
7 changes: 2 additions & 5 deletions x-pack/filebeat/filebeat.yml
Original file line number Diff line number Diff line change
Expand Up @@ -172,16 +172,13 @@ output.elasticsearch:
#ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
- add_host_metadata: ~
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~


# ================================== Logging ===================================

# Sets log level. The default log level is info.
Expand Down
3 changes: 3 additions & 0 deletions x-pack/filebeat/module/aws/cloudtrail/config/file.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,9 @@ paths:
- {{$path}}
{{ end }}
exclude_files: [".gz$"]
tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

processors:
- add_fields:
target: ''
Expand Down
3 changes: 3 additions & 0 deletions x-pack/filebeat/module/aws/cloudtrail/config/s3.yml
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,9 @@ session_token: {{ .session_token }}
role_arn: {{ .role_arn }}
{{ end }}

tags: {{.tags | tojson}}
publisher_pipeline.disable_host: {{ inList .tags "forwarded" }}

processors:
- add_fields:
target: ''
Expand Down
2 changes: 2 additions & 0 deletions x-pack/filebeat/module/aws/cloudtrail/manifest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ var:
- name: secret_access_key
- name: session_token
- name: role_arn
- name: tags
default: [forwarded]

ingest_pipeline: ingest/pipeline.yml
input: config/{{.input}}.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EX_PRINCIPAL_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@
"source.geo.region_iso_code": "CN-CQ",
"source.geo.region_name": "Chongqing",
"source.ip": "123.145.67.89",
"tags": [
"forwarded"
],
"user.id": "AROAIN5ATK5U7KEXAMPLE:JohnRole1",
"user_agent.device.name": "Spider",
"user_agent.name": "aws-cli",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "0123456789012",
"user.name": "Alice",
"user_agent.device.name": "Spider",
Expand Down Expand Up @@ -58,6 +61,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "0123456789012",
"user.name": "Alice",
"user_agent.device.name": "Spider",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
"service.type": "aws",
"source.address": "192.0.2.110",
"source.ip": "192.0.2.110",
"tags": [
"forwarded"
],
"user.id": "AIDACKCEVSQ6C2EXAMPLE",
"user.name": "JohnDoe",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -66,6 +69,9 @@
"service.type": "aws",
"source.address": "192.0.2.100",
"source.ip": "192.0.2.100",
"tags": [
"forwarded"
],
"user.id": "AIDACKCEVSQ6C2EXAMPLE",
"user.name": "JaneDoe",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -111,6 +117,9 @@
"service.type": "aws",
"source.address": "192.0.2.100",
"source.ip": "192.0.2.100",
"tags": [
"forwarded"
],
"user.id": "AROAIDPPEZS35WEXAMPLE:AssumedRoleSessionName",
"user.name": "RoleToBeAssumed",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "0123456789012",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -63,6 +66,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "0123456789012",
"user.name": "Alice",
"user_agent.device.name": "Spider",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@
"source.geo.region_iso_code": "US-VA",
"source.geo.region_name": "Virginia",
"source.ip": "72.21.198.64",
"tags": [
"forwarded"
],
"user.id": "EX_PRINCIPAL_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EX_PRINCIPAL_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@
"service.type": "aws",
"source.address": "192.0.2.1",
"source.ip": "192.0.2.1",
"tags": [
"forwarded"
],
"user.id": "AIDAQRSTUVWXYZEXAMPLE:devdsk",
"user_agent.device.name": "Spider",
"user_agent.name": "aws-cli",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "0123456789012",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down Expand Up @@ -62,6 +65,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_PRINCIPLE",
"user.name": "Alice",
"user_agent.device.name": "Spider",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Spider",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EX_PRINCIPAL_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,9 @@
"service.type": "aws",
"source.address": "127.0.0.1",
"source.ip": "127.0.0.1",
"tags": [
"forwarded"
],
"user.id": "EXAMPLE_ID",
"user.name": "Alice",
"user_agent.device.name": "Other",
Expand Down
Loading