-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Graylog 3.0 grok extractors broken #5704
Comments
remove the underscore from the capture group name |
I rewrote all my grok patterns using the workaround in my original post. |
@leftorbit23 I already looked into the problem (#5563). Can you please confirm that the patterns worked prior 3.0. |
@kmerz yes it was working before 3.0, we had a lot of patterns with underscore in capture group too |
The example I provided worked prior to the upgrade
The following still works in 3.0:
|
@leftorbit23 I found the issue. We updated a library we use and they dropped the underscore support (unintentionally). I opened a issue there and we discuss now internally how to handle that. |
So I just upgraded from 2.5.x to 3.0, and I believe I'm having the same issue. I finally sorted out all the other warnings/errors on the graylog server.log file. I believe that I only have two remaining issues related to pfsense log extraction. I deleted the two extractors, both gave errors in the webUI and when accessing the UI produced the able error in the log. Unfortunately, the errors are still there even after a server restart. It looks like all my other data sources are fine, but my pfSense source isn't. I'm not sure how to fix any of this, or if I can. I thought I had it figured out and that I'd be able to just remove those two extractors and then the data would then correctly be parsed again. That is apparently not the case though. Is there anything I can do myself? Can someone point me in the right direction? I don't know how it can still be throwing the errors after I removed those two extractors. If it helps, I used this guide to add the information to grafana: https://github.com/opc40772/pfsense-graylog |
For help you are better of asking in the community forum: https://community.graylog.org/ But as I said in the community forum, since here is a place to discuss issues and how to fix them! |
Thanks @kmerz. I just posted there, but I think I'm going to be screwed as I don't know what I need to fix and it looks like it's an upstream issue. |
To support underscores ("_") in Grok match group names, we had to modify the java-grok library to use the old regexp engine again. See: graylog-labs/java-grok#2 This also adds a test for the Grok extractor to make sure that using underscores works. Fixes #5704 Fixes #5563
* Switch back to a repackaged and fixed version of java-grok To support underscores ("_") in Grok match group names, we had to modify the java-grok library to use the old regexp engine again. See: graylog-labs/java-grok#2 This also adds a test for the Grok extractor to make sure that using underscores works. Fixes #5704 Fixes #5563 * Fix GrokPatternService#extractPatternNames and add a test for it * Add missing license header to GrokPatternServiceTest * Add test for named group with underscore Prior to this change, there was no test for named groups with underscores in the FunctionSnippetsTest This change enhances the grok() test to run with a named group with underscore.
@leftorbit23 @Hetann @JSylvia007 This has been fixed in master and will be backported into the upcoming 3.0.1 release. That means in 3.0.1 you will be able to use underscores again. |
* Switch back to a repackaged and fixed version of java-grok To support underscores ("_") in Grok match group names, we had to modify the java-grok library to use the old regexp engine again. See: graylog-labs/java-grok#2 This also adds a test for the Grok extractor to make sure that using underscores works. Fixes #5704 Fixes #5563 * Fix GrokPatternService#extractPatternNames and add a test for it * Add missing license header to GrokPatternServiceTest * Add test for named group with underscore Prior to this change, there was no test for named groups with underscores in the FunctionSnippetsTest This change enhances the grok() test to run with a named group with underscore. (cherry picked from commit e642a41)
…5807) * Switch back to a repackaged and fixed version of java-grok To support underscores ("_") in Grok match group names, we had to modify the java-grok library to use the old regexp engine again. See: graylog-labs/java-grok#2 This also adds a test for the Grok extractor to make sure that using underscores works. Fixes #5704 Fixes #5563 * Fix GrokPatternService#extractPatternNames and add a test for it * Add missing license header to GrokPatternServiceTest * Add test for named group with underscore Prior to this change, there was no test for named groups with underscores in the FunctionSnippetsTest This change enhances the grok() test to run with a named group with underscore. (cherry picked from commit e642a41)
After upgrading to Graylog 3.0, I noticed that many of my grok extractors didn't load.
server.log errors
Graylog 3.0 unable to process the following grok pattern:
(?<asa_proto>UDP)
Data sample:
Feb 20 2019 07:44:35: %ASA-6-302016: Teardown UDP connection 43191210 for outside:1.1.1.1/123 to inside:2.2.2.2/123 duration 0:04:01 bytes 985
Error:
We were not able to run the grok extraction because of the following error: named capturing group is missing trailing '>' near index 6 (?<asa_proto>UDP) ^
I was able to rewrite one of the grok patterns to get it to work
Before:
ASA-\\d-(?<asa_messageid>302016): (?<asa_action>Teardown) (?<asa_proto>UDP) connection %{BASE10NUM:asa_conn_id} for %{NOTSPACE:asa_interface_in}:%{IPV4:asa_src_ip}/%{BASE10NUM:asa_src_port} to %{NOTSPACE:asa_interface_out}:%{IPV4:asa_dst_ip}/%{BASE10NUM:asa_dst_port} duration %{TIME:asa_conn_durration} bytes %{BASE10NUM:asa_conn_bytes;long}
After:
ASA-\d-%{WORD:asa_messageid:int}: %{WORD:asa_action} %{WORD:asa_proto} connection %{BASE10NUM:asa_conn_id} for %{NOTSPACE:asa_interface_in}:%{IPV4:asa_src_ip}/%{BASE10NUM:asa_src_port} to %{NOTSPACE:asa_interface_out}:%{IPV4:asa_dst_ip}/%{BASE10NUM:asa_dst_port} duration %{TIME:asa_conn_durration} bytes %{BASE10NUM:asa_conn_bytes;long}
The text was updated successfully, but these errors were encountered: