-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Filebeat] Add URI Parts Processor to multiple modules #24699
[Filebeat] Add URI Parts Processor to multiple modules #24699
Conversation
08dda87
to
800299e
Compare
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
Trends 🧪💚 Flaky test reportTests succeeded. Expand to view the summary
Test stats 🧪
|
800299e
to
dcd06c2
Compare
5c94728
to
50cc14b
Compare
Pinging @elastic/security-external-integrations (Team:Security-External Integrations) |
00493ed
to
4d466fe
Compare
@andrewstucki This should be ready review and CI tests |
4d466fe
to
dd1fdc8
Compare
jenkins run tests |
@legoguy1000 so it looks to me like the new log entries that were generated for the nginx module are missing
and then run it with:
prior to re-running the regeneration for |
Not a problem |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Question about the backslash behavior. Not entirely sure if this is behavior based off of a combination of using urldecode
+ uri_parts
and ordering or if this is just some strange behavior/potentially a bug in the processor itself. I'll try and check this out locally and play around with it to see what's going on.
Ran the pipelines to update the |
dd1fdc8
to
8afd9c3
Compare
@andrewstucki @andrewkroh If you guys are good, can u run the pipeline? |
jenkins run tests |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So, based off of #19088 (comment) I want to see if @ebeahan can chime in about whether the best course is to keep the url.original
field in url-encoded form or if it should be decoded.
filebeat/module/apache/access/test/ssl-request.log-expected.json
Outdated
Show resolved
Hide resolved
e930be4
to
aabbfa3
Compare
jenkins run tests |
@legoguy1000 In general, this looks fine to me, I'll try and get it merged assuming that the tests pass |
I suspect its going to fail since they made the changes to the geo IP database, i'm re-generating the data now. I'll hold off pushing to see if it passes. |
@andrewstucki I update the generated data files to account for the changes in the pipeline. If you rerun the tests, we should be good. |
jenkins run tests |
@andrewstucki all passed |
@legoguy1000 thanks for the additions, I'll backport this for the 7.14 release |
Ya, that was definitely a long one. Definitely good conversations with the ECS team clarifying the fields. |
* Update Nginx pipelines * Update Apache, Nginx, IIS, Traefik pipelines * Update AWS S3 * Update Cisco * Update F5 * Update Fortinet * Update Imperva, Netscout, O365, Sophos, Squid, Suricata, Zscaler * additional fixes * update pipelines * unescape \ * remove urldecodes for url.original * updates after rebase * update zeek SIP * update changelog as requested by @andrewstucki * remove `url_decode` for `http.request.referrer` * update generated data (cherry picked from commit f1fea95)
@legoguy1000 backport opened at #25353 |
* Update Nginx pipelines * Update Apache, Nginx, IIS, Traefik pipelines * Update AWS S3 * Update Cisco * Update F5 * Update Fortinet * Update Imperva, Netscout, O365, Sophos, Squid, Suricata, Zscaler * additional fixes * update pipelines * unescape \ * remove urldecodes for url.original * updates after rebase * update zeek SIP * update changelog as requested by @andrewstucki * remove `url_decode` for `http.request.referrer` * update generated data (cherry picked from commit f1fea95) Co-authored-by: Alex Resnick <adr8292@gmail.com>
What does this PR do?
Updates Ingest Pipelines for the below modules:
Apache, Nginx, IIS, Traefik, S3Access, Cisco, F5, Fortinet, Google Workspace, Imperva, Microsoft, Netscout, O365, Sophos, Squid, Suricata, Zeek, Zia, Zoom, ZScaler
With the below changes
uri_parts
processor to parse URIs (includes URL decoding) to addurl.path
,url.extension
,url.query
....http.request.referrer
(when applicable) to make them human readableWhy is it important?
Parses URLs to break up the URL into the different parts and URL decodes them.
Checklist
CHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.Author's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs