[Feature Request] Build Grok Search Response Processor

[dbwiddis]

Is your feature request related to a problem? Please describe

Many search users want to manipulate field data which is not stored in the index. For example, an index may contain a log message "127.0.0.1 198.126.12 10/Oct/2000:13:55:36 -0700 200".

The Grok Ingest Processor does this parsing to create new fields, e.g.,

        "_source": {
          "message": "127.0.0.1 198.126.12 10/Oct/2000:13:55:36 -0700 200",
          "response_status": 200,
          "clientip": "198.126.12",
          "timestamp": "10/Oct/2000:13:55:36 -0700"
        },

However, if these fields are not populated on ingesting (for example, to save storage space) they are not available on search.

While search functionality does allow matching via regex, one still needs to manually post-parse the response to get useful data. This post-parsing should be automated.

Describe the solution you'd like

Replicate the functionality of the Grok ingest processor in a Search Response Processor

Related component

Search

Describe alternatives you've considered

More general regex parsing, but Grok is regex-based and well known/documented.

Additional context

• This is a similar request to [Feature Request] Build Sort Search Processor and Split Search Processor #14758 and I'll be happy to do the work to implement it if maintainers think it's a good idea. (I'm looking at you, triage meeting attendees.)

[dbwiddis]

Closing this request in favor of #16627 and its related implementations

[songkant-aws]

Dan, thanks for mentioning the log pattern support RFC! The goal of log pattern is to recover log statement's constant and variable words from log message.

I think the example you mentioned in this feature request is also valuable and could still be achieved by Grok. If the field is following strict regex pattern and user wants to split to fine-grained subfields, looks like Grok is still a good choice.