Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add process.command_line #599

Merged
merged 4 commits into from
Nov 18, 2019
Merged

Add process.command_line #599

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d86686c
Add process.command_line
Oct 29, 2019
83ed5fc
Changelog
Oct 29, 2019
e10deea
Merge branch 'master' into process-cmdline
Nov 18, 2019
6a56a2e
Update generated files
Nov 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.next.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ Thanks, you're awesome :-) -->

### Added

* Added `process.command_line`. #599

### Improvements

### Deprecated
Expand Down
8 changes: 7 additions & 1 deletion code/go/ecs/process.go
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

17 changes: 15 additions & 2 deletions docs/field-details.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2912,13 +2912,26 @@ These fields can help you correlate metrics information with a process id/name f
// ===============================================================

| process.args
| Array of process arguments.
| Array of process arguments, starting with the absolute path to the executable.

May be filtered to protect sensitive information.

type: keyword

example: `['ssh', '-l', 'user', '10.0.0.16']`
example: `['/usr/bin/ssh', '-l', 'user', '10.0.0.16']`

| extended

// ===============================================================

| process.command_line
| Full command line that started the process, including the absolute path to the executable, and all arguments.

Some arguments may be filtered to protect sensitive information.

type: keyword

example: `/usr/bin/ssh -l user 10.0.0.16`

| extended

Expand Down
14 changes: 12 additions & 2 deletions generated/beats/fields.ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2166,14 +2166,24 @@
level: extended
type: keyword
ignore_above: 1024
description: 'Array of process arguments.
description: 'Array of process arguments, starting with the absolute path to
the executable.

May be filtered to protect sensitive information.'
example:
- ssh
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
- name: command_line
level: extended
type: keyword
ignore_above: 1024
description: 'Full command line that started the process, including the absolute
path to the executable, and all arguments.

Some arguments may be filtered to protect sensitive information.'
example: /usr/bin/ssh -l user 10.0.0.16
- name: executable
level: extended
type: keyword
Expand Down
3 changes: 2 additions & 1 deletion generated/csv/fields.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,8 @@ package.name,keyword,extended,go,1.2.0-dev
package.path,keyword,extended,/usr/local/Cellar/go/1.12.9/,1.2.0-dev
package.size,long,extended,62231,1.2.0-dev
package.version,keyword,extended,1.12.9,1.2.0-dev
process.args,keyword,extended,"['ssh', '-l', 'user', '10.0.0.16']",1.2.0-dev
process.args,keyword,extended,"['/usr/bin/ssh', '-l', 'user', '10.0.0.16']",1.2.0-dev
process.command_line,keyword,extended,/usr/bin/ssh -l user 10.0.0.16,1.2.0-dev
process.executable,keyword,extended,/usr/bin/ssh,1.2.0-dev
process.hash.md5,keyword,extended,,1.2.0-dev
process.hash.sha1,keyword,extended,,1.2.0-dev
Expand Down
34 changes: 24 additions & 10 deletions generated/ecs/ecs_flat.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3071,29 +3071,43 @@ package.version:
short: Package version
type: keyword
process.args:
description: 'Array of process arguments.
description: 'Array of process arguments, starting with the absolute path to the
executable.

May be filtered to protect sensitive information.'
example:
- ssh
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
flat_name: process.args
ignore_above: 1024
level: extended
name: args
order: 4
order: 5
short: Array of process arguments.
type: keyword
process.command_line:
description: 'Full command line that started the process, including the absolute
path to the executable, and all arguments.

Some arguments may be filtered to protect sensitive information.'
example: /usr/bin/ssh -l user 10.0.0.16
flat_name: process.command_line
ignore_above: 1024
level: extended
name: command_line
order: 4
short: Full command line that started the process.
type: keyword
process.executable:
description: Absolute path to the process executable.
example: /usr/bin/ssh
flat_name: process.executable
ignore_above: 1024
level: extended
name: executable
order: 5
order: 6
short: Absolute path to the process executable.
type: keyword
process.hash.md5:
Expand Down Expand Up @@ -3183,7 +3197,7 @@ process.start:
flat_name: process.start
level: extended
name: start
order: 9
order: 10
short: The time the process started.
type: date
process.thread.id:
Expand All @@ -3193,7 +3207,7 @@ process.thread.id:
format: string
level: extended
name: thread.id
order: 7
order: 8
short: Thread ID.
type: long
process.thread.name:
Expand All @@ -3203,7 +3217,7 @@ process.thread.name:
ignore_above: 1024
level: extended
name: thread.name
order: 8
order: 9
short: Thread name.
type: keyword
process.title:
Expand All @@ -3215,7 +3229,7 @@ process.title:
ignore_above: 1024
level: extended
name: title
order: 6
order: 7
short: Process title.
type: keyword
process.uptime:
Expand All @@ -3224,7 +3238,7 @@ process.uptime:
flat_name: process.uptime
level: extended
name: uptime
order: 10
order: 11
short: Seconds the process has been up.
type: long
process.working_directory:
Expand All @@ -3234,7 +3248,7 @@ process.working_directory:
ignore_above: 1024
level: extended
name: working_directory
order: 11
order: 12
short: The working directory of the process.
type: keyword
related.ip:
Expand Down
34 changes: 24 additions & 10 deletions generated/ecs/ecs_nested.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3459,29 +3459,43 @@ process:
copied to the global field for correlation.'
fields:
args:
description: 'Array of process arguments.
description: 'Array of process arguments, starting with the absolute path to
the executable.

May be filtered to protect sensitive information.'
example:
- ssh
- /usr/bin/ssh
- -l
- user
- 10.0.0.16
flat_name: process.args
ignore_above: 1024
level: extended
name: args
order: 4
order: 5
short: Array of process arguments.
type: keyword
command_line:
description: 'Full command line that started the process, including the absolute
path to the executable, and all arguments.

Some arguments may be filtered to protect sensitive information.'
example: /usr/bin/ssh -l user 10.0.0.16
flat_name: process.command_line
ignore_above: 1024
level: extended
name: command_line
order: 4
short: Full command line that started the process.
type: keyword
executable:
description: Absolute path to the process executable.
example: /usr/bin/ssh
flat_name: process.executable
ignore_above: 1024
level: extended
name: executable
order: 5
order: 6
short: Absolute path to the process executable.
type: keyword
hash.md5:
Expand Down Expand Up @@ -3571,7 +3585,7 @@ process:
flat_name: process.start
level: extended
name: start
order: 9
order: 10
short: The time the process started.
type: date
thread.id:
Expand All @@ -3581,7 +3595,7 @@ process:
format: string
level: extended
name: thread.id
order: 7
order: 8
short: Thread ID.
type: long
thread.name:
Expand All @@ -3591,7 +3605,7 @@ process:
ignore_above: 1024
level: extended
name: thread.name
order: 8
order: 9
short: Thread name.
type: keyword
title:
Expand All @@ -3603,7 +3617,7 @@ process:
ignore_above: 1024
level: extended
name: title
order: 6
order: 7
short: Process title.
type: keyword
uptime:
Expand All @@ -3612,7 +3626,7 @@ process:
flat_name: process.uptime
level: extended
name: uptime
order: 10
order: 11
short: Seconds the process has been up.
type: long
working_directory:
Expand All @@ -3622,7 +3636,7 @@ process:
ignore_above: 1024
level: extended
name: working_directory
order: 11
order: 12
short: The working directory of the process.
type: keyword
group: 2
Expand Down
4 changes: 4 additions & 0 deletions generated/elasticsearch/6/template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1277,6 +1277,10 @@
"ignore_above": 1024,
"type": "keyword"
},
"command_line": {
"ignore_above": 1024,
"type": "keyword"
},
"executable": {
"ignore_above": 1024,
"type": "keyword"
Expand Down
4 changes: 4 additions & 0 deletions generated/elasticsearch/7/template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1276,6 +1276,10 @@
"ignore_above": 1024,
"type": "keyword"
},
"command_line": {
"ignore_above": 1024,
"type": "keyword"
},
"executable": {
"ignore_above": 1024,
"type": "keyword"
Expand Down
4 changes: 4 additions & 0 deletions generated/legacy/template.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -900,6 +900,10 @@
"ignore_above": 1024,
"type": "keyword"
},
"command_line": {
"ignore_above": 1024,
"type": "keyword"
},
"executable": {
"ignore_above": 1024,
"type": "keyword"
Expand Down
14 changes: 12 additions & 2 deletions schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2130,15 +2130,25 @@
"description": "These fields contain information about a process.\nThese fields can help you correlate metrics information with a process id/name from a log message. The `process.pid` often stays in the metric itself and is copied to the global field for correlation.\n",
"fields": {
"process.args": {
"description": "Array of process arguments.\nMay be filtered to protect sensitive information.",
"example": "['ssh', '-l', 'user', '10.0.0.16']",
"description": "Array of process arguments, starting with the absolute path to the executable.\nMay be filtered to protect sensitive information.",
"example": "['/usr/bin/ssh', '-l', 'user', '10.0.0.16']",
"footnote": "",
"group": 2,
"level": "extended",
"name": "process.args",
"required": false,
"type": "keyword"
},
"process.command_line": {
"description": "Full command line that started the process, including the absolute path to the executable, and all arguments.\nSome arguments may be filtered to protect sensitive information.",
"example": "/usr/bin/ssh -l user 10.0.0.16",
"footnote": "",
"group": 2,
"level": "extended",
"name": "process.command_line",
"required": false,
"type": "keyword"
},
"process.executable": {
"description": "Absolute path to the process executable.",
"example": "/usr/bin/ssh",
Expand Down
19 changes: 15 additions & 4 deletions schemas/process.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,15 +45,26 @@
description: >
Identifier of the group of processes the process belongs to.

- name: command_line
level: extended
type: keyword
short: Full command line that started the process.
description: >
Full command line that started the process, including the absolute path
to the executable, and all arguments.

Some arguments may be filtered to protect sensitive information.
example: "/usr/bin/ssh -l user 10.0.0.16"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When do we use process.command_line vs process.args? They seem a little redundant.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I totally agree they are redundant.

Having the args as an array is exceptionally powerful (more than I initially realized) when detections are looking for an exact term in a command.

However especially in the context of endpoint detections, many of them will still have to rely on fuzzier searches e.g. with wildcards, to detect some attack patterns such as obfuscated commands, base64 decoding, webshells & so on.

So both fields are actually needed, in order to perform each class of detections.

Also unrelated to detections, displaying process.command_line in Kibana will be much nicer :-)

cc @rw-access @randomuserid

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah exactly.
I think process.args will have more utility for search than process.command_line, but process.command_line gives us full recoverability of the original command line as presented to the OS. For one odd edge case, cmd.exe essentially puts 2 spaces between arg[0] and arg[1], but arg parsing is lossy, so we lose that artifact.


- name: args
level: extended
type: keyword
short: Array of process arguments.
description: >
Array of process arguments.
Array of process arguments, starting with the absolute path to the executable.

May be filtered to protect sensitive information.
example: ["ssh", "-l", "user", "10.0.0.16"]
example: ["/usr/bin/ssh", "-l", "user", "10.0.0.16"]

- name: executable
level: extended
Expand Down Expand Up @@ -82,8 +93,8 @@

- name: thread.name
level: extended
type: keyword
example: 'thread-0'
type: keyword
example: 'thread-0'
description: >
Thread name.

Expand Down