[filebeat][httpjson] Initial implementation of httpjson v2 #22320

marc-gr · 2020-10-30T15:53:34Z

What does this PR do?

Creates an initial implementation of the new Httpjson input.

Why is it important?

HTTPJSON is too specific to our particular modules use cases and is hard to extend or reuse out of them. For this we wanted to do an effort to generalize the input so it can be used in a more flexible way.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

Set a mock endpoint to respond to http requests
Set up a config that uses the new input

- type: httpjson
  is_v2: true
  interval: 10s
  request.method: post
  request.url: https://example.com
  request.transforms:
    - set:
        target: body.foo
        value: bazz
  response.transforms:
    - delete:
        target: body.this_is_private
  response.pagination:
    - set:
        target: url.value
        value: "https://example.com/page"
    - set:
        target: url.params.p
        value: "{{.last_response.body.page}}"

output.console:
  pretty: true

Then in x-pack/filebeat run mage build && ./filebeat -e -v and check the output.

Related issues

marc-gr · 2020-10-30T16:05:39Z

cc @P1llus @epixa

elasticmachine · 2020-10-30T16:41:30Z

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	24
Passed	1270
Skipped	14
Total	1308

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / x-pack/filebeat-build / TestNewAppend – v2
Name: Build&Test / x-pack/filebeat-build / TestNewAppend/newAppendResponse_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-build / TestNewAppend/newAppendRequest_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-build / TestNewAppend/newAppendPagination_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-build / TestNewDelete – v2
Name: Build&Test / x-pack/filebeat-build / TestNewDelete/newDeleteResponse_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-build / TestNewDelete/newDeleteRequest_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-build / TestNewDelete/newDeletePagination_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-build / TestNewSet – v2
Name: Build&Test / x-pack/filebeat-build / TestNewSet/newSetResponse_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-build / TestNewSet/newSetRequest_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-build / TestNewSet/newSetPagination_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-windows-windows-2019 / TestNewAppend – v2
Name: Build&Test / x-pack/filebeat-windows-windows-2019 / TestNewAppend/newAppendResponse_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-windows-windows-2019 / TestNewAppend/newAppendRequest_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-windows-windows-2019 / TestNewAppend/newAppendPagination_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-windows-windows-2019 / TestNewDelete – v2
Name: Build&Test / x-pack/filebeat-windows-windows-2019 / TestNewDelete/newDeleteResponse_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-windows-windows-2019 / TestNewDelete/newDeleteRequest_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-windows-windows-2019 / TestNewDelete/newDeletePagination_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-windows-windows-2019 / TestNewSet – v2
Name: Build&Test / x-pack/filebeat-windows-windows-2019 / TestNewSet/newSetResponse_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-windows-windows-2019 / TestNewSet/newSetRequest_targets_something_else – v2
Name: Build&Test / x-pack/filebeat-windows-windows-2019 / TestNewSet/newSetPagination_targets_something_else – v2

elasticmachine · 2020-10-30T16:41:32Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #22320 updated
Start Time: 2020-12-04T08:47:00.374+0000
Duration: 46 min 37 sec

Test stats 🧪

Test	Results
Failed	0
Passed	2420
Skipped	259
Total	2679

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	2420
Skipped	259
Total	2679

elasticmachine · 2020-11-30T15:55:00Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

- Modify context done error log - Change is_v2 config bool for string config_version - Change retryable logger to be created from input logger

…2320) * Add oauth config * Add stateless input * Add request and basic config * Create v2 input and basic transforms * Refactor and set up basic requester structure * Run fmt * Set tests and fix tpl * basic processing structure * Add pagination mechanism * Publish the events * Add check redirect and oauth tests * Deprecate old httpjson and include both * Return default value if execution is empty * wip * Add split * Handle request errors and pagination end * Add date functions for templates and tests * Add getRFC5988Link functionality to templates * Switch between v1/v2 based on config flag * Add cursor * Add rate limit * Put back original filebeat.yml * Add deprecation warning for old httpjson version * Add transform tests * Initial doc changes * Add more debug logs and split tests * Ignore empty values on set and append * Allow content type and accept override and change redirect default * Add add and toInt functions * Support array responses * Add tests cases and minor changes * Add changelog entry * Do not copy mutex on delete transform * Add PR suggestions: - Modify context done error log - Change is_v2 config bool for string config_version - Change retryable logger to be created from input logger * Documentation fixes (cherry picked from commit 47d8941)

…22920) * Add oauth config * Add stateless input * Add request and basic config * Create v2 input and basic transforms * Refactor and set up basic requester structure * Run fmt * Set tests and fix tpl * basic processing structure * Add pagination mechanism * Publish the events * Add check redirect and oauth tests * Deprecate old httpjson and include both * Return default value if execution is empty * wip * Add split * Handle request errors and pagination end * Add date functions for templates and tests * Add getRFC5988Link functionality to templates * Switch between v1/v2 based on config flag * Add cursor * Add rate limit * Put back original filebeat.yml * Add deprecation warning for old httpjson version * Add transform tests * Initial doc changes * Add more debug logs and split tests * Ignore empty values on set and append * Allow content type and accept override and change redirect default * Add add and toInt functions * Support array responses * Add tests cases and minor changes * Add changelog entry * Do not copy mutex on delete transform * Add PR suggestions: - Modify context done error log - Change is_v2 config bool for string config_version - Change retryable logger to be created from input logger * Documentation fixes (cherry picked from commit 47d8941)

marc-gr added enhancement in progress Pull request is currently in progress. labels Oct 30, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Oct 30, 2020

marc-gr added the Team:Security-External Integrations label Oct 30, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Oct 30, 2020

marc-gr force-pushed the httpjson_v2 branch from df63b7b to 62135f7 Compare November 11, 2020 11:41

marc-gr force-pushed the httpjson_v2 branch 13 times, most recently from 1b23270 to 89c280f Compare November 25, 2020 14:30

P1llus mentioned this pull request Nov 26, 2020

[Filebeat][New Fileset] Cisco AMP #22768

Merged

6 tasks

marc-gr force-pushed the httpjson_v2 branch from b205821 to 6d56d41 Compare November 30, 2020 15:50

marc-gr marked this pull request as ready for review November 30, 2020 15:54

marc-gr added 17 commits December 4, 2020 09:44

Switch between v1/v2 based on config flag

6759001

Add cursor

c410215

Add rate limit

12857e1

Put back original filebeat.yml

bedf52d

Add deprecation warning for old httpjson version

02dcf8a

Add transform tests

afb68a7

Initial doc changes

6789ae3

Add more debug logs and split tests

bdbb877

Ignore empty values on set and append

ce233a9

Allow content type and accept override and change redirect default

5f685fa

Add add and toInt functions

55c0275

Support array responses

9408239

Add tests cases and minor changes

3a427c9

Add changelog entry

92fadf8

Do not copy mutex on delete transform

42fb485

Add PR suggestions:

f807f73

- Modify context done error log - Change is_v2 config bool for string config_version - Change retryable logger to be created from input logger

Documentation fixes

d4afbda

marc-gr force-pushed the httpjson_v2 branch from 0b36844 to d4afbda Compare December 4, 2020 08:46

marc-gr merged commit 47d8941 into elastic:master Dec 4, 2020

marc-gr deleted the httpjson_v2 branch December 4, 2020 10:33

marc-gr mentioned this pull request Dec 4, 2020

Cherry-pick #22320 to 7.x: [filebeat][httpjson] Initial implementation of httpjson v2 #22920

Merged

43 tasks

marc-gr added the v7.11.0 label Dec 4, 2020

jsoriano mentioned this pull request Dec 7, 2020

Add support into the httpjson input to collect values of maps in different events #21465

Closed

This was referenced Dec 14, 2020

Cherry-pick #22677 to 7.x: [Filebeat][New Module] Snyk vulnerabilities and audit logs #23102

Merged

Filebeat - httpjson missing "content-type" #22807

Closed

P1llus mentioned this pull request Feb 15, 2021

Cherry-pick #22768 to 7.x: [Filebeat][New Fileset] Cisco AMP #24038

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[filebeat][httpjson] Initial implementation of httpjson v2 #22320

[filebeat][httpjson] Initial implementation of httpjson v2 #22320

marc-gr commented Oct 30, 2020 •

edited

Loading

marc-gr commented Oct 30, 2020

elasticmachine commented Oct 30, 2020 •

edited by jenkins-beats-ci bot

Loading

Test stats 🧪

Genuine test errors

elasticmachine commented Oct 30, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Test stats 🧪

elasticmachine commented Nov 30, 2020

[filebeat][httpjson] Initial implementation of httpjson v2 #22320

[filebeat][httpjson] Initial implementation of httpjson v2 #22320

Conversation

marc-gr commented Oct 30, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

marc-gr commented Oct 30, 2020

elasticmachine commented Oct 30, 2020 • edited by jenkins-beats-ci bot Loading

🐛 Flaky test report

Test stats 🧪

Genuine test errors

elasticmachine commented Oct 30, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

Test stats 🧪

elasticmachine commented Nov 30, 2020

marc-gr commented Oct 30, 2020 •

edited

Loading

elasticmachine commented Oct 30, 2020 •

edited by jenkins-beats-ci bot

Loading

elasticmachine commented Oct 30, 2020 •

edited by jenkins-beats-ci bot

Loading