[Filebeat][New Input] Http Input

[P1llus]

What does this PR do?

This filebeat input configures a HTTP port listener, accepting JSON formatted POST requests, which again is formatted into a event, initially the event is created with the "json." prefix and expects the ingest pipeline to mutate the event during ingestion.

The initial set of features is based on the Logstash input plugin, but implemented differently: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http.html

Why is it important?

This idea is based on a few different scenarios:

• The user already has a large beats installation and no Logstash, and do not want to install Logstash solely for a single feature.
• HTTP Input allows applications to be directly integrated with Elastic, without needing connectivity to Elasticsearch directly (or Logstash).
• Allows us to integrate and create modules for any product that supports HTTP POST events like SOAR, cloud applications, ticketing systems etc etc.

Features currently implemented

HTTP Basic Auth On/Off
HTTP/HTTPS configurable
Listening interface and port configurable
Response code on success configurable
Response body on success configurable
Response header on success configurable
Proper HTTP codes on both success and error responses
Message prefix configurable
URL to post to is configurable
SSL path to cert, key and CA is configurable.

TODO

• Clean up code
• Import configuration options from common
• Write documentation
• Add TLS mutual auth support
• Config validation
• Tests

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

Pinging @elastic/siem (Team:SIEM)

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request [Filebeat][New Input] Http Input #18298 updated]

• Start Time: 2020-05-19T09:00:32.844+0000

• Duration: 52 min 44 sec (3104325)

Test stats 🧪

Test	Results
Failed	0
Passed	3811
Skipped	672
Total	4483

[P1llus]

@kvch Added first iteration of docs, let me know if it looks okay :) Do I need to add it anywhere else as well? I don't think it will be automatically included by just creating a new asciidoc in the inputs docs folder.

[andrewstucki]

Some more thoughts as I incrementally work through this 🙂

[andrewstucki]

So, my previous comment about overriding headers mainly stems from this line. The Accept header checking seems to indicate that we're always going to respond with a Content-Type: application/json header--but the fact that we allow the response headers to be overridden in the config actually breaks this. Nothing currently prevents the user from doing something like:

{beatname_lc}.inputs:
- type: http_endpoint
  ...
  response_body: '<message>success</message>'
  response_header: '{"Content-Type": "application/xml"}'

If we're actually wanting to make sure we always hand back json I'd do two things:

1. Always set the Content-Type: application/json header and ignore what the user may specify in the config
2. Validate the response value that the user specifies in response_body is actually valid JSON.

[P1llus]

@andrewstucki
I thought more about the response header for now, and it makes sense to me to remove it for now.
There is a few reasons for that.

1. If we decide later on to add support for more than JSON, it would make more sense.
2. Currently the custom response headers was more to add other headers than content-type, though I am still having a hard time finding good use-cases for when that would be needed.
   So if it is okay with you, I have hardcoded the response header (upon success), but still allowing anyone to set the response code and will revisit this later if there is requests open for this feature.

[kvch]

LGTM. Once all questions are resolved, I am OK with merging it.

[andrewstucki]

Last comment and then I think I'm good on this

[andrewstucki]

If we're going to hardcode that this is a JSON-based response, I think we should likely still sanity check that the user actually entered JSON. You could just make a Validate method for the config structure and inside do something like

func (c *config) Validate() error {
  if !json.Valid(c.ResponseBody) {
    return errors.New("response_body must be valid JSON")
  }
  // other validations here
  return nil 
}

[P1llus]

Added the validation, just had to convert it from a string to a byteslice as well.

New return upon wrong response body is:

2020-05-25T08:43:01.744+0200	ERROR	instance/beat.go:931	Exiting: Failed to start crawler: starting input failed: Error while initializing input: response_body must be valid JSON accessing 'filebeat.inputs.0' (source:'filebeat.yml')
Exiting: Failed to start crawler: starting input failed: Error while initializing input: response_body must be valid JSON accessing 'filebeat.inputs.0' (source:'filebeat.yml')

[P1llus]

Should now be ready for merge and backport :)

[elasticmachine]

💚 Build Succeeded

Expand to view the summary

Build stats

• Build Cause: [Pull request #18298 updated]

• Start Time: 2020-05-25T11:32:33.266+0000

• Duration: 53 min 15 sec

Test stats 🧪

Test	Results
Failed	0
Passed	3811
Skipped	672
Total	4483

[P1llus]

jenkins, test this please

[kvch]

jenkins run tests