Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Filebeat] Update AWS ELB ingest pipeline #26441

Merged
merged 3 commits into from
Jun 24, 2021
Merged

[Filebeat] Update AWS ELB ingest pipeline #26441

merged 3 commits into from
Jun 24, 2021

Conversation

legoguy1000
Copy link
Contributor

@legoguy1000 legoguy1000 commented Jun 23, 2021 •
edited
Loading

What does this PR do?

Updates the Filebeat aws.elb module ingest pipeline to properly parse the request URL and user agent string.

Why is it important?

Currently the request URL is being recorded as the http.request.referer when per https://docs.aws.amazon.com/elasticloadbalancing/latest/application/load-balancer-access-logs.html#access-log-entry-syntax, its the actual request, not referer. This fixes that and parses the URL via the uri_parts processor.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • [ ]

How to test this PR locally

cd beats/x-pack/filebeat
TESTING_FILEBEAT_MODULES=aws TESTING_FILEBEAT_FILESETS=elb mage -v pythonIntegTest

Related issues

Use cases

Screenshots

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jun 23, 2021
@legoguy1000 legoguy1000 marked this pull request as ready for review June 23, 2021 12:32
@legoguy1000
Copy link
Contributor Author

@kaiyan-sheng For the changelog IDK if the removal of the field should be a bug or breaking change. Please let me know and I will update it.

@elasticmachine
Copy link
Collaborator

elasticmachine commented Jun 23, 2021 •
edited by jenkins-beats-ci bot
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Started by user kaiyan-sheng

  • Start Time: 2021-06-24T19:53:37.049+0000

  • Duration: 111 min 33 sec

  • Commit: 0871665

Test stats 🧪

Test Results
Failed 0
Passed 7395
Skipped 1201
Total 8596

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 7395
Skipped 1201
Total 8596

@kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng For the changelog IDK if the removal of the field should be a bug or breaking change. Please let me know and I will update it.

I would consider this as a bug. Thank you!

@kaiyan-sheng kaiyan-sheng self-assigned this Jun 23, 2021
@kaiyan-sheng kaiyan-sheng added needs_backport PR is waiting to be backported to other branches. needs_integration_sync Changes in this PR need synced to elastic/integrations. Team:Integrations Label for the Integrations team labels Jun 23, 2021
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jun 23, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@mergify
Copy link
Contributor

mergify bot commented Jun 23, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26435-aws-elb upstream/26435-aws-elb
git merge upstream/master
git push upstream 26435-aws-elb

@legoguy1000
Copy link
Contributor Author

@kaiyan-sheng can you run the CI tests when ur able.

@kaiyan-sheng kaiyan-sheng requested a review from leehinman June 24, 2021 02:50
@jsoriano
Copy link
Member

/test

@mergify
Copy link
Contributor

mergify bot commented Jun 24, 2021

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26435-aws-elb upstream/26435-aws-elb
git merge upstream/master
git push upstream 26435-aws-elb

leehinman
leehinman approved these changes Jun 24, 2021
View reviewed changes
Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code changes approved. please remove extra newline in CHANGELOG.next.asciidoc.

@legoguy1000
Copy link
Contributor Author

@leehinman any idea why the Windows 2019 job failed??

@kaiyan-sheng
Copy link
Contributor

@legoguy1000 It's probably a flaky test. Let me restart the test.

@kaiyan-sheng kaiyan-sheng merged commit d7724cd into elastic:master Jun 24, 2021
@kaiyan-sheng
Copy link
Contributor

@legoguy1000 Thank you for your contribution!!!

@kaiyan-sheng kaiyan-sheng mentioned this pull request Jun 24, 2021
Merged
6 tasks
@kaiyan-sheng kaiyan-sheng added v7.14.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jun 24, 2021
@kaiyan-sheng kaiyan-sheng mentioned this pull request Jun 24, 2021
Merged
4 tasks
@legoguy1000 legoguy1000 deleted the 26435-aws-elb branch June 24, 2021 23:07
@kaiyan-sheng kaiyan-sheng removed the needs_integration_sync Changes in this PR need synced to elastic/integrations. label Jun 24, 2021
kaiyan-sheng added a commit that referenced this pull request Jun 25, 2021
@kaiyan-sheng @legoguy1000
[Filebeat] Update AWS ELB ingest pipeline (#26441) (#26493)
dd0b565 
(cherry picked from commit d7724cd)

Co-authored-by: Alex Resnick <adr8292@gmail.com>
mdelapenya added a commit to mdelapenya/beats that referenced this pull request Jun 28, 2021
@mdelapenya
Merge branch 'master' into archive-system-tests
b6dbec3 
* master: (32 commits)
  [Metricbeat] Change Account ID to Project ID in `gcp.billing` module (elastic#26412)
  update libbeat fields.ecs.yml file and ecsVersion to 1.10.0 (elastic#26121)
  [Filebeat] Update AWS ELB ingest pipeline (elastic#26441)
  [FIlebeat] add strict_date_optional_time_nanos date format to PanOS module (elastic#26158)
  Fix the irregular and typo on prometheus module. (elastic#25726)
  [Filebeat] Parse additonal debug data fields for Okta module (elastic#25818)
  fix: update MSSQL Server linux image's Docker registry (elastic#26440)
  Update indexing.go godocs (elastic#26408)
  Do not close filestream harvester if an unexpected error is returned when close.on_state_change.* is enabled (elastic#26411)
  Add support for copytruncate method when rotating input logs with an external tool in `filestream` input (elastic#23457)
  Allow fields with ip_range datatype (elastic#26444)
  Add Anomali ThreatStream support to threatintel module (elastic#26350)
  fix: use the right param type (elastic#26469)
  [Automation] Update elastic stack version to 8.0.0-7640093f for testing (elastic#26460)
  Set SM Filebeat modules as GA (elastic#26226)
  Fix rfc5464 date parsing in the syslog input (elastic#26419)
  Add linked account information into billing metricset (elastic#26285)
  [Filebeat] Update HA Proxy log grok patterns (elastic#25835)
  disable metricbeat logstash test_node_stats (elastic#26436)
  chore: pass BEAT_VERSION when running E2E tests (elastic#26291)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@leehinman leehinman leehinman approved these changes

Assignees

@kaiyan-sheng kaiyan-sheng

Labels
Team:Integrations Label for the Integrations team v7.14.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Filebeat AWS ELB ingest fields differ from Kibana Observability
5 participants
@legoguy1000 @elasticmachine @kaiyan-sheng @jsoriano @leehinman