Filebeat AWS ELB ingest fields differ from Kibana Observability

[stephank]

Previous discussion: https://discuss.elastic.co/t/aws-elb-ingest-pipeline-bug/276356
All Elastic Stack components are on version 7.13.2.

It looks like the AWS ALB support in the Filebeat AWS module produces http.request.referrer in its ingest pipeline, while Kibana Observability (such as the Logs section) expects url.* fields to be set (probably via uri_parts).

Currently, the Kibana Logs section displays access logs as follows, with a missing request path: (This display is apparently produced by generic_webserver.ts.)

[aws][access] 1.2.3.4  "GET  HTTP/2.0" 200 152966

I believe the fix should be something like:

--- a/x-pack/filebeat/module/aws/elb/ingest/pipeline.yml
+++ b/x-pack/filebeat/module/aws/elb/ingest/pipeline.yml
@@ -77,7 +77,7 @@ processors:
           (?:-|%{NUMBER:aws.elb.backend.http.response.status_code:long})
           %{NUMBER:http.request.body.bytes:long}
           %{NUMBER:http.response.body.bytes:long}
-          \"(?:-|%{WORD:http.request.method}) (?:-|%{NOTSPACE:http.request.referrer}) (?:-|HTTP/%{NOTSPACE:http.version})\"
+          \"(?:-|%{WORD:http.request.method}) (?:-|%{NOTSPACE:_tmp.url_orig}) (?:-|HTTP/%{NOTSPACE:http.version})\"
           \"%{DATA:user_agent.original}\"
           %{ELBSSL}
         ELBTCPLOG: >-
@@ -110,6 +110,11 @@ processors:
       field: event.category
       value: web

+  - uri_parts:
+      if: 'ctx.http != null'
+      field: _tmp.url_orig
+      ignore_failure: true
+
   - set:
       if: 'ctx.http == null'
       field: 'aws.elb.protocol'

Which results in correct display in Kibana Logs:

[aws][access] 1.2.3.4  "GET /blogs? HTTP/2.0" 200 6815

(I'm not sure why it always displays a ? despite there not being a query string, but I believe that's a separate Kibana issue.)

[kaiyan-sheng]

Hi @stephank , thanks for creating the issue here. Do you have a sample log from ELB (something like https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/aws/elb/test/elb-http.log) that we can use to test the new grok pattern?

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[legoguy1000]

Hi @stephank , thanks for creating the issue here. Do you have a sample log from ELB (something like https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/aws/elb/test/elb-http.log) that we can use to test the new grok pattern?

@kaiyan-sheng My interpretation of this from the forum was that the current sample logs are the same that is what we already have, just that the URL that's being parsed shouldn't be the referrer but the actual request. I referenced, https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#access-log-entry-syntax, see the "request" row.

[legoguy1000]

PR submitted

[ramgurufh]

I think these changes are not yet pushed to 7.13.4.

[legoguy1000]

I think these changes are not yet pushed to 7.13.4.

Ya it looks like it was only backported to 7.14.