Add 21 autogenerated filesets from rsa2elk devices #19713
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pinging @elastic/siem (Team:SIEM) |
botelastic
bot
added
needs_team
💔 Tests Failed
Expand to view the summary
Build stats
Test stats 🧪
Test errorsExpand to view the tests failures
Steps errorsExpand to view the steps failures
Log outputExpand to view the last 100 lines of log output
|
andrewstucki
previously requested changes
Jul 8, 2020
There was a problem hiding this comment.
Did an initial scan through the first 1/3 or so of mappings. Awesome work @adriansr! This might take awhile to get through though :(
Made some comments on the field mappings, but most important changes IMO right now are:
- Drop the
event.categoryfields for now since they're not currently mapping to ECS-allowed values user.nameshould be treated as a string rather than an array--dump everything underrelated.userthat's currently underuser.nameand then, if we have time we can figure out which of the referenced user belongs underuser.name.
x-pack/filebeat/module/cylance/protect/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/cylance/protect/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/imperva/securesphere/config/liblogparser.js
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/imperva/securesphere/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/imperva/securesphere/test/generated.log-expected.json
Show resolved
Hide resolved
x-pack/filebeat/module/fortinet/clientendpoint/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/infoblox/nios/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/infoblox/nios/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
andrewstucki
reviewed
Jul 8, 2020
x-pack/filebeat/module/microsoft/dhcp/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/sonicwall/firewall/test/general.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/sonicwall/firewall/test/general.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/squid/log/test/access1.log-expected.json
Outdated
Show resolved
Hide resolved
| "event.module": "squid", | ||
| "event.original": "1035368729.430 371 210.8.79.228 TCP_MISS/200 2136 GET http://www.fas.harvard.edu/~hpcws/nav/navbar_r3_c6.jpg - FIRST_PARENT_MISS/proxy2.syd.connect.com.au image/jpeg", | ||
| "fileset.name": "log", | ||
| "http.response.body.content": "navbar_r3_c6.jpg", |
There was a problem hiding this comment.
this doesn't look right, the content field would be the raw bytes of the jpeg in this case, so not something we'd fill in since it's binary data
There was a problem hiding this comment.
This is the field webpage, mapped to http.response.body.content. I don't think any parser is going to capture the full body of a request, so we might as well not map this field to ECS, as there is no field for the document name. Maybe file.name? @webmat WDYT?
x-pack/filebeat/module/squid/log/test/access2.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/squid/log/test/access2.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/zscaler/zia/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
leehinman
requested changes
Jul 8, 2020
x-pack/filebeat/module/cylance/protect/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
| "event.code": "https", | ||
| "event.dataset": "fortinet.clientendpoint", | ||
| "event.module": "fortinet", | ||
| "event.original": "March 26 2016/03/26 aqui4726.mail.localhost proto=icmp service=https status=deny src=10.85.66.161 dst=10.131.115.96 src_port=2638 dst_port=1890 server_app=eum pid=654 app_name=rmagni traff_direct=sit block_count=5509 logon_user=onev@tenima1073.local msg=success", |
There was a problem hiding this comment.
should the ips, ports, proto & service be parsed out? seems odd that it isn't.
| "event.dataset": "imperva.securesphere", | ||
| "event.module": "imperva", | ||
| "event.original": "%IMPERVA-Imperva,dstIP=10.70.155.35,dstPort=892,dbUsername=tatno,srcIP=10.81.122.126,srcPort=4141,creatTime=29 January 2016 06:09:59,srvGroup=uam,service=untutl,appName=rad,event#=taliqu,eventType=Login,usrGroup=ommod,usrAuth=True,application=\"scivel\",osUsername=aqui,srcHost=radipis5408.mail.local,dbName=enatuse,schemaName=magn,bindVar=equuntu,sqlError=failure,respSize=5910,respTime=10.347000,affRows=sum,action=\"cancel\",rawQuery=\"sit\"", | ||
| "event.outcome": "Success", |
There was a problem hiding this comment.
event.outcome should be lowercase "success"
| "observer.product": "Nexpose", | ||
| "observer.type": "Vulnerability", | ||
| "observer.vendor": "Rapid7", | ||
| "rsa.internal.messageid": "[Site:", |
There was a problem hiding this comment.
Is this right?, looks off with the [
There was a problem hiding this comment.
There's something off in this parser or the log generator. I will check
| "fileset.name": "firewall", | ||
| "input.type": "log", | ||
| "log.flags": [ | ||
| "dissect_parsing_error" |
There was a problem hiding this comment.
is the dissect failing? seems odd we aren't getting src & dst fields out of this.
| "rsa.network.domain": "login.yahoo.com", | ||
| "rsa.time.duration_time": 5006, | ||
| "rsa.time.event_time": "2006-09-08T04:21:52.000Z", | ||
| "rsa.time.event_time_str": "1157689312", |
There was a problem hiding this comment.
seems odd that the event_time_str field is populated with a number.
There was a problem hiding this comment.
Yep, that's how this parser is defined. squid uses UNIX timestamps, which is captured it in event_time_str (raw). That is used to populate event_time and @timestamp.
| "url.domain": "www.fas.harvard.edu", | ||
| "url.original": "http://www.fas.harvard.edu/~hpcws/nav/navbar_r3_c6.jpg", | ||
| "user.name": [ | ||
| "-" |
There was a problem hiding this comment.
very minor. I'm assuming "-" means no user name. can we skip if "-"?
Comment on lines
29
to
33
| "rsa.web.alias_host": "https://example.com/illumqui/ventore.html?min=ite#utl", | ||
| "rsa.web.fqdn": "https://example.com/illumqui/ventore.html?min=ite#utl", |
There was a problem hiding this comment.
seems odd that these fields have full url
There was a problem hiding this comment.
It's a bit odd. This parser is capturing the full URL and setting fqdn to that. There's a method to extract the FQDN from an URL, but it's not using it for some reason.
andrewstucki
reviewed
Jul 8, 2020
x-pack/filebeat/module/fortinet/clientendpoint/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/fortinet/clientendpoint/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
x-pack/filebeat/module/fortinet/clientendpoint/test/generated.log-expected.json
Outdated
Show resolved
Hide resolved
|
@leehinman @andrewstucki I think I've addressed all the issues I could. Thanks a lot for your feedback, it's helping to add a lot of improvement. |
tsg
approved these changes
Jul 9, 2020
leehinman
reviewed
Jul 9, 2020
| "10.232.59.7" | ||
| ], | ||
| "rsa.internal.messageid": "ntpdate", | ||
| "rsa.time.duration_str": "tur", |
There was a problem hiding this comment.
Might be a problem with the original parsers. I think "tur" is the timezone, not a duration.
This is caused by the log generator not being able to add valid timestamps to the logs.
This adds the following experimental filesets based on Apache 2 license device parsers: - tomcat.log - netscout.sightline - barracuda.waf - f5.bigipapm - bluecoat.director - cisco.nexus - citrix.virtualapps - cylance.protect - f5.firepass - fortinet.clientendpoint - imperva.securesphere - infoblox.nios - juniper.junos - kaspersky.av - microsoft.dhcp - tenable.nessus_security - rapid7.nexpose - radware.defensepro - sonicwall.firewall - squid.log - zscaler.zia
Some pipelines were failing due to trailing space at the end of messages (which the original XML format ignores). Updated the generator to strip those spaces.
|
Merged using admin privileges due to CI flakiness. All tests passing in my computer. |
6 tasks
v1v
added a commit
to v1v/beats
that referenced
this pull request
Jul 14, 2020
* upstream/master: (25 commits) [Elastic Agent] Send checkin payload to Fleet (elastic#19857) [Ingest Manager] Fixed tests across agent elastic#19877 [Ingest Manager] Fix serialization test elastic#19876 Fix service start type mapping in windows/service metricset (elastic#19551) ci: Change comment trigger detection method (elastic#19827) Add 21 autogenerated filesets from rsa2elk devices (elastic#19713) [Ingest Manager] Agent config cleanup (elastic#19848) libbeat/publisher/pipeline: fix data races (elastic#19821) Update monitoring-internal-collection.asciidoc (elastic#19422) (elastic#19697) [Elastic Agent] Trust exchange endpoint must bind to 127.0.0.1 (elastic#19861) Specify an ECS version in Auditbeat/Packetbeat/Winlogbeat (elastic#19159) Add azure billing metricset (elastic#19207) Add support for appinsights in the metricbeat azure module (elastic#18940) Add MySQL query metricset with lightweight module and SQL helper (elastic#18955) [Ingest Manager] Refuse invalid stream values in configuration (elastic#19587) Do not use vendor during integration tests (elastic#19839) LIBBEAT: Enhancement Convert dissected values from String to other basic data types and IP (elastic#18683) [Elastic Agent] Remove support for "logs" and only support logfile (elastic#19761) [CI] support windows-2012 (elastic#19773) Do not update go.mod during packaging and testing (elastic#19823) ...
adriansr
added a commit
to adriansr/beats
that referenced
this pull request
Jul 14, 2020
This adds the following experimental filesets based on Apache 2 license device parsers: - tomcat.log - netscout.sightline - barracuda.waf - f5.bigipapm - bluecoat.director - cisco.nexus - citrix.virtualapps - cylance.protect - f5.firepass - fortinet.clientendpoint - imperva.securesphere - infoblox.nios - juniper.junos - kaspersky.av - microsoft.dhcp - tenable.nessus_security - rapid7.nexpose - radware.defensepro - sonicwall.firewall - squid.log - zscaler.zia (cherry picked from commit 6d0dc62)
adriansr
added a commit
that referenced
this pull request
Jul 14, 2020
This adds the following experimental filesets based on Apache 2 license device parsers: - tomcat.log - netscout.sightline - barracuda.waf - f5.bigipapm - bluecoat.director - cisco.nexus - citrix.virtualapps - cylance.protect - f5.firepass - fortinet.clientendpoint - imperva.securesphere - infoblox.nios - juniper.junos - kaspersky.av - microsoft.dhcp - tenable.nessus_security - rapid7.nexpose - radware.defensepro - sonicwall.firewall - squid.log - zscaler.zia (cherry picked from commit 6d0dc62)
3 tasks
andrewkroh
added a commit
to andrewkroh/beats
that referenced
this pull request
Jul 22, 2020
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965
6 tasks
andrewkroh
added a commit
that referenced
this pull request
Jul 23, 2020
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in #19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes #19965
andrewkroh
added a commit
to andrewkroh/beats
that referenced
this pull request
Jul 23, 2020
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965 (cherry picked from commit ea7c05f)
6 tasks
andrewkroh
added a commit
to andrewkroh/beats
that referenced
this pull request
Jul 23, 2020
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965 (cherry picked from commit ea7c05f)
6 tasks
This was referenced Jul 23, 2020
andrewkroh
added a commit
that referenced
this pull request
Jul 23, 2020
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in #19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes #19965 (cherry picked from commit ea7c05f)
andrewkroh
added a commit
that referenced
this pull request
Jul 23, 2020
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in #19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes #19965 (cherry picked from commit ea7c05f)
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
This adds the following experimental filesets based on Apache 2 license device parsers: - tomcat.log - netscout.sightline - barracuda.waf - f5.bigipapm - bluecoat.director - cisco.nexus - citrix.virtualapps - cylance.protect - f5.firepass - fortinet.clientendpoint - imperva.securesphere - infoblox.nios - juniper.junos - kaspersky.av - microsoft.dhcp - tenable.nessus_security - rapid7.nexpose - radware.defensepro - sonicwall.firewall - squid.log - zscaler.zia
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
When the fields.yml file is constructed it is done by appending files together and adding some indenting. In the case of Filebeat, a fileset's fields.yml is appended with an indent of 8 spaces to the module's fields.yml. This generally allows for all of the filesets fields to become children of the module. The problem we had was that the new filesets added in elastic#19713 expected that their fields would be root fields (not children to the module namespace). In cases where the module already existed and had declared a module namespace field in its fields.yml this resulted in unexpectedly namespaced fieldset fields (e.g. microsoft.rsa.* instead of rsa.*). The size of the x-pack/filebeat index-pattern is still large (915885 bytes), but not so large that it goes beyond the Kibana request payload limit. Fixes elastic#19965 (cherry picked from commit 84a227a)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This adds the following experimental filesets based on Apache 2 license device parsers:
Why is it important?
This is an effort to generate as many as possible experimental input sources from a set of 300 Apache2-licensed log parsers.
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Notes to reviewers
The modules in this PR are autogenerated. All follow the same format:
The
README.md,config.yml,docs.asciidocandfields.ymlare the same, basically replacing the module/vendor/product name.Same for
input.ymlandmanifest.yml. They define the same variables and inputs (tcp,udpandfile).The
generated.logfiles are autogenerated by the same program that converts the original XML files to Javascript. Generating logs using parser's patterns (some with overlap) and user-defined field names is hard. Some generated logs make more sense than others.A few selected modules contain real logs that we were able to obtain from other sources. Currently:
The
liblogparser.jsis the helper Javascript library for the parser. It's important to review this file. It's the same for all filesets.The
pipeline.jsis the autogenerated pipeline. Contains all the parsers and actions defined in the source XML. We're aware that some of these parsers are outdated and some partly broken regarding extra whitespace in patterns. There will be an ongoing effort to fix them.A couple of modules already existed:
In this case the
_meta/config.ymland_meta/docs.asciidochave been merged automatically.Some parsers are currently broken (have
tags:dissect_parsing_error). I'm working on fixing those.The code for the generator is in https://github.com/adriansr/nwdevice2filebeat