[Filebeat Enhancement] Pattern for Cisco Message 734001. #16612

MarcusCaepio · 2020-02-26T15:49:03Z

Fixes #16212
The split part is needed, because one has to be able to search for an
explicit dap_record. As the records order and number can vary a lot,
just saving the whole string makes no sense. I choosed "cisco.user" and
"cisco.connection_type", as looking to the syslog messages docs,
they also call it like that. I made an "make update" both in /beats/xpack/filebeat/ and /beats/filebeat/

Hopefully the pipeline runs sucessfully now.

My code follows the style guidelines of this project
~~- [ ] I have commented my code, particularly in hard-to-understand areas~~
~~- [ ] I have made corresponding changes to the documentation~~
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

Testfiles are included

Related issues

Closes [Filebeat - Module Cisco-ASA] Parsing of Cisco Event Message 734001 #16212

Use cases

Parsing VPN Logs for Cisco ASA and FTD

elasticmachine · 2020-02-26T15:49:05Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-02-26T15:49:06Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-03-02T14:29:39Z

Pinging @elastic/siem (Team:SIEM)

adriansr

Thanks for your contribution!

There's a few issues to address. Make sure to fix the pipeline per my suggestions and then generate the test data.

x-pack/filebeat/module/cisco/asa/_meta/fields.yml

adriansr · 2020-03-02T19:35:47Z

x-pack/filebeat/module/cisco/asa/test/dap_records-expected.json

+{
+    "source": {
+      "ip": "1.2.3.4"
+    },
+    "user": {
+      "email": "firstname.lastname@domain.net"
+    },
+    "cisco": {
+      "connection_type": "AnyConnect",
+      "dap_records": [
+        "dap_1",
+        "dap_2"
+      ],
+      "asa": {
+        "message_id": "734001"
+      }
+    }
+}


This file should be named dap_records.log-expected.json, and the content must be generated by the system tests. Otherwise is ignored.

To generate the content of this file, you must run the module's tests in generate mode (first fix the issues with the pipeline I pointed out in other comments):

Have an Elasticsearch in localhost:9200.

Compile a testing binary and create a virtualenv:

cd {beats_repo}/x-pack/filebeat go test -c virtualenv -p path_to_python_3 venv . venv/bin/activate pip install -r ../../libbeat/tests/system/requirements.txt

Then run the cisco/asa tests in generate mode:

touch module/cisco/asa/test/dap_records.log-expected.json GENERATE=1 MODULES_PATH=$PWD/module INTEGRATION_TESTS=1 TESTING_FILEBEAT_MODULES=cisco TESTING_FILEBEAT_FILESETS=asa nosetests -v -s tests/system/test_xpack_modules.py

Run the cisco tests in testing mode and it should pass:

MODULES_PATH=$PWD/module INTEGRATION_TESTS=1 TESTING_FILEBEAT_MODULES=cisco nosetests -v -s tests/system/test_xpack_modules.py

adriansr · 2020-03-02T19:37:14Z

x-pack/filebeat/module/cisco/ftd/_meta/fields.yml

+  - name: dap_records
+    type: keyword
+    description: >
+      The assigned DAP records


Also add default_field to this ones.

adriansr · 2020-03-02T19:39:26Z

x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml

+ - dissect:
+     if: "ctx._temp_.cisco.message_id == '734001'"
+     field: "message"
+     pattern: "DAP: User %{user.email}, Addr %{source.ip}, Connection %{cisco.connection_type}: The following DAP records were selected for this connection: %{cisco.dap_records->}"


A few issues with this pattern:

Should set source.address instead of source.ip. A later processor will set .ip from .address if it's a valid IP.

Should set _temp_.cisco.xxx instead of cisco.xxx fields. A later processor creates cisco.asa.xxx or cisco.ftd.xxx based on _temp_.cisco.

adriansr · 2020-03-02T19:40:52Z

x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml

+     field: "message"
+     pattern: "DAP: User %{user.email}, Addr %{source.ip}, Connection %{cisco.connection_type}: The following DAP records were selected for this connection: %{cisco.dap_records->}"
+ - split:
+     field: "cisco.dap_records"


Same here. The field to split is _temp_.cisco.dap_records.

Also this processor needs a ignore_missing: true because it runs for all cisco events, and only 734001 has the field. It can also do with a condition but it's better to use ignore_missing in this case.

MarcusCaepio · 2020-03-04T15:29:02Z

@adriansr Unfortunately, I am very busy right now, so I don't have time to change the PR. As you know, I already created this PR for the third time and every time there was sth. additionally I had to change. To avoid a neverending story, could you make the needed changes so I know directly, how to do it right for the next time?
Thanks a lot!

MarcusCaepio · 2020-03-09T07:54:08Z

Thanks a lot @adriansr
When will this be available? I see, in 7.6.1 it's not yet integrated.

adriansr · 2020-03-18T18:38:26Z

@MarcusCaepio I'll merge this into 7.7.0. As it is an enhancement, it can't go into a bugfix release.

I just rebased it and will try to merge it soon.

Fixes elastic#16212 The split part is needed, because one has to be able to search for an explicit dap_record. As the records order and number can vary a lot, just saving the whole string makes no sense. I chose "user.email", "source.ip" as ECS fields and "cisco.connection_type", "cisco.dap_records", as looking to the syslog messages docs,they also call it like that. I made "make update" in /beats/x.pack/filebeat and /beats/filebeat. Hopefully the pipeline succeeds now.

adriansr · 2020-03-18T23:11:57Z

jenkins, test this

adriansr · 2020-03-19T10:44:47Z

jenkins, test this

jsoriano

LGTM, only a possible suggestion.

x-pack/filebeat/module/cisco/ftd/_meta/fields.yml

The split part is needed, because one has to be able to search for an explicit dap_record. As the records order and number can vary a lot, just saving the whole string makes no sense. I chose "user.email", "source.ip" as ECS fields and "cisco.connection_type", "cisco.dap_records", as looking to the syslog messages docs,they also call it like that. I made "make update" in /beats/x.pack/filebeat and /beats/filebeat. Hopefully the pipeline succeeds now. Fixes elastic#16212 Co-authored-by: MarcusCaepio <7324088+MarcusCaepio@users.noreply.github.com> (cherry picked from commit ac2b333)

…essage 734001. (#17128) The split part is needed, because one has to be able to search for an explicit dap_record. As the records order and number can vary a lot, just saving the whole string makes no sense. I chose "user.email", "source.ip" as ECS fields and "cisco.connection_type", "cisco.dap_records", as looking to the syslog messages docs,they also call it like that. I made "make update" in /beats/x.pack/filebeat and /beats/filebeat. Hopefully the pipeline succeeds now. Fixes #16212 (cherry picked from commit ac2b333) Co-authored-by: MarcusCaepio <7324088+MarcusCaepio@users.noreply.github.com>

MarcusCaepio requested a review from a team as a code owner February 26, 2020 15:49

MarcusCaepio force-pushed the master branch from 7557987 to a00c466 Compare February 26, 2020 15:55

adriansr added enhancement review Team:SIEM labels Mar 2, 2020

adriansr suggested changes Mar 2, 2020

View reviewed changes

adriansr added the needs_backport PR is waiting to be backported to other branches. label Mar 18, 2020

adriansr force-pushed the master branch from 24dc826 to 59efb09 Compare March 18, 2020 18:36

MarcusCaepio and others added 3 commits March 19, 2020 00:10

fixes

5e01828

Set default_field: false

2ff9408

adriansr force-pushed the master branch from 59efb09 to 2ff9408 Compare March 18, 2020 23:11

adriansr added 2 commits March 19, 2020 10:38

Missing bits

fbc6dc5

Remove unneeded file

5a87a22

jsoriano reviewed Mar 19, 2020

View reviewed changes

x-pack/filebeat/module/cisco/ftd/_meta/fields.yml Show resolved Hide resolved

jsoriano approved these changes Mar 19, 2020

View reviewed changes

adriansr approved these changes Mar 19, 2020

View reviewed changes

adriansr merged commit ac2b333 into elastic:master Mar 19, 2020

adriansr mentioned this pull request Mar 19, 2020

Cherry-pick #16612 to 7.x: [Filebeat Enhancement] Pattern for Cisco Message 734001. #17128

Merged

3 tasks

adriansr added v7.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Mar 19, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat Enhancement] Pattern for Cisco Message 734001. #16612

[Filebeat Enhancement] Pattern for Cisco Message 734001. #16612

MarcusCaepio commented Feb 26, 2020

elasticmachine commented Feb 26, 2020

elasticmachine commented Feb 26, 2020

elasticmachine commented Mar 2, 2020

adriansr left a comment

adriansr Mar 2, 2020 •

edited

Loading

adriansr Mar 2, 2020

adriansr Mar 2, 2020 •

edited

Loading

adriansr Mar 2, 2020

MarcusCaepio commented Mar 4, 2020

MarcusCaepio commented Mar 9, 2020 •

edited

Loading

adriansr commented Mar 18, 2020

adriansr commented Mar 18, 2020

adriansr commented Mar 19, 2020

jsoriano left a comment

[Filebeat Enhancement] Pattern for Cisco Message 734001. #16612

[Filebeat Enhancement] Pattern for Cisco Message 734001. #16612

Conversation

MarcusCaepio commented Feb 26, 2020

How to test this PR locally

Related issues

Use cases

elasticmachine commented Feb 26, 2020

elasticmachine commented Feb 26, 2020

elasticmachine commented Mar 2, 2020

adriansr left a comment

Choose a reason for hiding this comment

adriansr Mar 2, 2020 • edited Loading

Choose a reason for hiding this comment

adriansr Mar 2, 2020

Choose a reason for hiding this comment

adriansr Mar 2, 2020 • edited Loading

Choose a reason for hiding this comment

adriansr Mar 2, 2020

Choose a reason for hiding this comment

MarcusCaepio commented Mar 4, 2020

MarcusCaepio commented Mar 9, 2020 • edited Loading

adriansr commented Mar 18, 2020

adriansr commented Mar 18, 2020

adriansr commented Mar 19, 2020

jsoriano left a comment

Choose a reason for hiding this comment

adriansr Mar 2, 2020 •

edited

Loading

adriansr Mar 2, 2020 •

edited

Loading

MarcusCaepio commented Mar 9, 2020 •

edited

Loading