auditbeat/module/file_integrity: add file parsing processor #28802

efd6 · 2021-11-04T01:22:24Z

What does this PR do?

This adds a file parser and analyser framework along the lines of the approach
described in #18048. The processor allows inspection of files visited
during a scan or seen in events based on a document path configuration.

A file analyser is also included that performs an analysis of executable or library
object files for a variety of platforms, providing details of imports and other
potential threat indicators.

The details shows an example event with a configuration examining elf headers.

{
  "@timestamp": "2021-11-04T10:51:27.776Z",
  "@metadata": {
    "beat": "auditbeat",
    "type": "_doc",
    "version": "8.1.0"
  },
  "file": {
    "elf": {
      "go_imports": [
        "github.com/elastic/beats/v7/auditbeat/module/file_integrity/testdata/b.Used",
        "github.com/elastic/beats/v7/auditbeat/module/file_integrity/testdata/b.hash"
      ],
      "go_imports_names_entropy": 4.156563879566413,
      "go_stripped": false,
      "sections": [
        {
          "name": "",
          "virtual_size": 0,
          "entropy": 0
        },
        {
          "name": ".text",
          "virtual_size": 524246,
          "entropy": 6.171439471921469
        },
        {
          "name": ".rodata",
          "virtual_size": 219456,
          "entropy": 4.35581679174804
        },
        {
          "name": ".shstrtab",
          "virtual_size": 378,
          "entropy": 4.332514286812164
        },
        {
          "name": ".typelink",
          "virtual_size": 1264,
          "entropy": 3.7700952245237285
        },
        {
          "name": ".itablink",
          "virtual_size": 96,
          "entropy": 2.149135857994785
        },
        {
          "virtual_size": 0,
          "entropy": 0,
          "name": ".gosymtab"
        },
        {
          "entropy": 5.48601456601009,
          "name": ".gopclntab",
          "virtual_size": 370120
        },
        {
          "name": ".go.buildinfo",
          "virtual_size": 32,
          "entropy": 3.560820381093429
        },
        {
          "name": ".noptrdata",
          "virtual_size": 67360,
          "entropy": 5.6078976935228955
        },
        {
          "name": ".data",
          "virtual_size": 30736,
          "entropy": 1.6046396762408546
        },
        {
          "name": ".bss",
          "virtual_size": 192328,
          "entropy": 7.993887986364125
        },
        {
          "virtual_size": 21344,
          "entropy": 7.975434273036139,
          "name": ".noptrbss"
        },
        {
          "name": ".zdebug_abbrev",
          "virtual_size": 281,
          "entropy": 7.186678878967747
        },
        {
          "name": ".zdebug_line",
          "virtual_size": 112916,
          "entropy": 7.990554045986755
        },
        {
          "name": ".zdebug_frame",
          "virtual_size": 21787,
          "entropy": 7.925008509003898
        },
        {
          "name": ".debug_gdb_scripts",
          "virtual_size": 49,
          "entropy": 4.249529170858451
        },
        {
          "entropy": 7.995385192057325,
          "name": ".zdebug_info",
          "virtual_size": 203308
        },
        {
          "name": ".zdebug_loc",
          "virtual_size": 104665,
          "entropy": 7.988800696836627
        },
        {
          "name": ".zdebug_ranges",
          "virtual_size": 36796,
          "entropy": 7.7864300204494885
        },
        {
          "virtual_size": 100,
          "entropy": 5.293528005004741,
          "name": ".note.go.buildid"
        },
        {
          "name": ".symtab",
          "virtual_size": 50664,
          "entropy": 3.2101068454851
        },
        {
          "name": ".strtab",
          "virtual_size": 45782,
          "entropy": 4.811971045761911
        }
      ],
      "import_hash": "d41d8cd98f00b204e9800998ecf8427e",
      "go_import_hash": "10bddcb4cee42080f76c88d9ff964491"
    },
    "type": "file",
    "gid": "100",
    "mtime": "2021-11-04T10:51:27.769Z",
    "size": 1796862,
    "ctime": "2021-11-04T10:51:27.769Z",
    "uid": "100",
    "inode": "259681",
    "group": "user",
    "path": ".../github.com/elastic/beats/auditbeat/module/file_integrity/testdata/go_elf_executable",
    "mime_type": "application/x-executable",
    "owner": "user",
    "mode": "0664",
    "hash": {
      "sha1": "1ae774a9f4d48ac29d53d6401407d9305d7a3fb5"
    }
  },
  "event": {
    "module": "file_integrity",
    "dataset": "file",
    "kind": "event",
    "category": [
      "file"
    ],
    "type": [
      "change"
    ],
    "action": [
      "attributes_modified"
    ]
  },
  "service": {
    "type": "file_integrity"
  },
  "ecs": {
    "version": "8.0.0"
  },
  "host": {
    "id": "f66b78414f9867166d794365855dd289",
    "containerized": false,
    "name": "mach",
    "hostname": "mach",
    "architecture": "x86_64",
    "os": {
      "type": "linux",
      "platform": "os",
      "version": "9000+",
      "family": "debian",
      "name": "osOS",
      "kernel": "5.15.0",
      "codename": "aybabtu"
    }
  },
  "agent": {
    "ephemeral_id": "464ae374-3dc8-11ec-9bbc-0242ac130002",
    "id": "5b6c99dc-3dc8-11ec-9bbc-0242ac130002",
    "name": "mach",
    "type": "auditbeat",
    "version": "8.1.0"
  }
}

Why is it important?

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Confirm new fields area appropriately typed
Check golden values for reasonableness

How to test this PR locally

Run go test in github.com/elastic/beats/auditbeat/module/file_integrity

Related issues

Relates File Format Parsing Processor #18048

Use cases

See related issue.

Screenshots

N/A

Logs

N/A

elasticmachine · 2021-11-04T01:29:46Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-11-02T01:21:10.251+0000
Duration: 128 min 48 sec

Test stats 🧪

Test	Results
Failed	0
Passed	23727
Skipped	1951
Total	25678

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

efd6 · 2021-11-04T05:11:46Z

Query whether large test assets are OK.

efd6 · 2021-11-04T20:34:54Z

Note that due to CVE-2021-41771 this may be unsafe to use until we are on go1.17.3. See https://groups.google.com/g/golang-dev/c/BC9Nelav6LU.

efd6 · 2021-11-05T01:02:51Z

Given Plan9 is not a supported platform I can remove the tests and logic for this GOOS, instead just exiting the parser early when the obj type is plan9. This will also save test asset size.

efd6 · 2021-11-05T03:28:25Z

/test

elasticmachine · 2021-11-05T06:38:05Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

efd6 · 2021-11-05T20:23:08Z

auditbeat/module/file_integrity/event.go

+	if len(fileParsers) != 0 && event.ParserResults == nil {
+		event.ParserResults = make(common.MapStr)
+	}
+	for _, p := range fileParsers {
+		err = p.Parse(event.ParserResults, path)
+		if err != nil {
+			event.errors = append(event.errors, err)
+		}
+	}
+


This should probably have INFO level total work time logging and per processor DEBUG level work time logging.

efd6 · 2021-11-05T23:42:34Z

auditbeat/module/file_integrity/file_parsers.go

+	// TODO: Implement globbing paths so that the user can write things like
+	// "file.*.imports".


Probably the best way to do this is to expand a RE2 pattern over the parser-available fields rather than inventing a new incomplete globbing syntax. The world already has enough of those.

andrewkroh

Really nice enhancement. I think it would be good to explain how this feature works in the module docs. As a user I'd probably want to know:

When are executables read? On every file change? Only when a file content hash changes?
Does it stop reading after determining the file is not an exe format?

Can you transfer toutoumomoma into the Elastic GH org and get it setup into Jenkins.

efd6 · 2021-11-16T11:41:10Z

/test

efd6 · 2021-11-16T20:26:20Z

/test

andrewkroh

LGTM. Thanks for adding to the docs.

efd6 · 2021-11-30T01:40:41Z

This is not for merging until go1.17.3 is our build tool.

mergify · 2021-12-09T15:33:22Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fileinterity/parsers upstream/fileinterity/parsers
git merge upstream/master
git push upstream fileinterity/parsers

botelastic · 2022-01-08T15:47:12Z

Hi!
We just realized that we haven't looked into this PR in a while. We're sorry!

We're labeling this issue as Stale to make it hit our filters and make sure we get back to it in as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1.
Thank you for your contribution!

efd6 · 2022-09-21T12:26:09Z

/test

andrewkroh

This is an awesome new feature. The code and docs look great. I know there is a lot of work hidden behind github.com/elastic/toutoumomoma. Leaving a few questions.

auditbeat/module/file_integrity/_meta/fields.yml

auditbeat/module/file_integrity/schema.fbs

Leave improvement in code gen directives.

rdner

Looks like the E2E test failure is unrelated to the change.

andrewkroh

Removal of flatbuffer changes LGTM

mergify · 2022-10-25T00:04:23Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fileinterity/parsers upstream/fileinterity/parsers
git merge upstream/main
git push upstream fileinterity/parsers

mergify · 2022-10-26T03:14:47Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fileinterity/parsers upstream/fileinterity/parsers
git merge upstream/main
git push upstream fileinterity/parsers

This adds a file parser and analyser framework along the lines of the approach described in #18048. The processor allows inspection of files visited during a scan or seen in events based on a document path configuration. A file analyser is also included that performs an analysis of executable or library object files for a variety of platforms, providing details of imports and other potential threat indicators.

efd6 added enhancement Team:Security-External Integrations backport-v8.0.0 Automated backport with mergify 8.1 candidate labels Nov 4, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Nov 4, 2021

efd6 force-pushed the fileinterity/parsers branch from 4d78099 to f95c439 Compare November 4, 2021 01:26

efd6 force-pushed the fileinterity/parsers branch from f95c439 to 3c7ee44 Compare November 4, 2021 03:03

efd6 force-pushed the fileinterity/parsers branch 2 times, most recently from 7bc083e to 8584d29 Compare November 4, 2021 10:03

efd6 force-pushed the fileinterity/parsers branch 2 times, most recently from 46f8d2d to 3b85d54 Compare November 5, 2021 00:54

efd6 marked this pull request as ready for review November 5, 2021 06:38

efd6 requested a review from a team as a code owner November 5, 2021 06:38

efd6 requested a review from andrewkroh November 5, 2021 06:38

efd6 commented Nov 5, 2021

View reviewed changes

andrewkroh reviewed Nov 15, 2021

View reviewed changes

andrewkroh approved these changes Nov 30, 2021

View reviewed changes

efd6 added 5 commits September 21, 2022 21:36

use elastic fork of toutoumomoma

d5b36e5

remove unneeded nolint directive

93a1514

recover panics during executable parse analysis

905c874

remove redundant lint comment and don't panic

f62cd61

add physical size field for sections

975b7ee

efd6 force-pushed the fileinterity/parsers branch from 1ca155d to 975b7ee Compare September 21, 2022 12:09

efd6 requested a review from andrewkroh September 21, 2022 20:42

efd6 marked this pull request as ready for review September 21, 2022 20:42

efd6 requested a review from a team as a code owner September 21, 2022 20:42

efd6 requested review from belimawr and rdner and removed request for a team September 21, 2022 20:42

andrewkroh reviewed Oct 7, 2022

View reviewed changes

auditbeat/module/file_integrity/_meta/fields.yml Show resolved Hide resolved

auditbeat/module/file_integrity/schema.fbs Outdated Show resolved Hide resolved

revert changes to flatbuffers definitions

05a8e2a

Leave improvement in code gen directives.

efd6 force-pushed the fileinterity/parsers branch from 31ba7c6 to 05a8e2a Compare October 9, 2022 21:22

rdner approved these changes Oct 11, 2022

View reviewed changes

andrewkroh approved these changes Oct 24, 2022

View reviewed changes

efd6 force-pushed the fileinterity/parsers branch from 05a8e2a to a8b5dfb Compare October 26, 2022 03:10

update toutoumomoma

fbebfc5

efd6 force-pushed the fileinterity/parsers branch from a8b5dfb to fbebfc5 Compare October 26, 2022 03:14

Merge branch 'master' into fileinterity/parsers

b65687c

This was referenced Oct 26, 2022

Add fields for executable object format metadata for ELF, Mach-O and PE elastic/ecs#2083

Merged

proposal: add package watcher to auditbeat file_integrity module #33481

Closed

use field definitions from ecs main

8322b72

efd6 merged commit 1fb2484 into elastic:main Nov 2, 2022

scottgeary mentioned this pull request Aug 21, 2024

Build 2577 for main with status FAILURE - open ./testdata/garble_macho_executable: Operation did not complete successfully because the file contains a virus or potentially unwanted software. #39090

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

auditbeat/module/file_integrity: add file parsing processor #28802

auditbeat/module/file_integrity: add file parsing processor #28802

efd6 commented Nov 4, 2021 •

edited

Loading

elasticmachine commented Nov 4, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

efd6 commented Nov 4, 2021

efd6 commented Nov 4, 2021

efd6 commented Nov 5, 2021

efd6 commented Nov 5, 2021

elasticmachine commented Nov 5, 2021

efd6 Nov 5, 2021

efd6 Nov 5, 2021

andrewkroh left a comment

efd6 commented Nov 16, 2021

efd6 commented Nov 16, 2021

andrewkroh left a comment

efd6 commented Nov 30, 2021

mergify bot commented Dec 9, 2021

botelastic bot commented Jan 8, 2022

efd6 commented Sep 21, 2022

andrewkroh left a comment •

edited

Loading

rdner left a comment

andrewkroh left a comment

mergify bot commented Oct 25, 2022

mergify bot commented Oct 26, 2022

		// TODO: Implement globbing paths so that the user can write things like
		// "file.*.imports".

auditbeat/module/file_integrity: add file parsing processor #28802

auditbeat/module/file_integrity: add file parsing processor #28802

Conversation

efd6 commented Nov 4, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Nov 4, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

efd6 commented Nov 4, 2021

efd6 commented Nov 4, 2021

efd6 commented Nov 5, 2021

efd6 commented Nov 5, 2021

elasticmachine commented Nov 5, 2021

efd6 Nov 5, 2021

Choose a reason for hiding this comment

efd6 Nov 5, 2021

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

efd6 commented Nov 16, 2021

efd6 commented Nov 16, 2021

andrewkroh left a comment

Choose a reason for hiding this comment

efd6 commented Nov 30, 2021

mergify bot commented Dec 9, 2021

botelastic bot commented Jan 8, 2022

efd6 commented Sep 21, 2022

andrewkroh left a comment • edited Loading

Choose a reason for hiding this comment

rdner left a comment

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

mergify bot commented Oct 25, 2022

mergify bot commented Oct 26, 2022

efd6 commented Nov 4, 2021 •

edited

Loading

elasticmachine commented Nov 4, 2021 •

edited by jenkins-beats-ci bot

Loading

andrewkroh left a comment •

edited

Loading