Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat][New Input] Http Input #18298

Merged
merged 29 commits into from
May 25, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
0a7972c
MVP for http input
P1llus May 5, 2020
c985208
adding temp error message on event send failure
P1llus May 5, 2020
5102356
updated comment that was there from old input
P1llus May 5, 2020
7c824c9
modify config
P1llus May 5, 2020
ebb8ed1
Merge branch 'master' into http_input_module
P1llus May 6, 2020
c94b6b9
changing default config
P1llus May 6, 2020
aaafd65
cleaning up the code and refactor checks to its own functions
P1llus May 7, 2020
458dcb5
mage fmt
P1llus May 7, 2020
f51160e
Changed name from httpinput to http_endpoint, moved http server to it…
P1llus May 9, 2020
1ccb3c2
updated code based on PR comments. Changed clientauth after confirmin…
P1llus May 12, 2020
c185b48
change packagename
P1llus May 12, 2020
72a8652
change packagename
P1llus May 12, 2020
7db39a9
updated code to be more idiomatic with comments from noemi
P1llus May 14, 2020
04be474
removing variable declaration to be more idiomatic
P1llus May 14, 2020
aa27ac9
adding basic test, currently not working
P1llus May 15, 2020
4d2cc8b
forgot to add method validation
P1llus May 15, 2020
07c00d6
make ALLL the tests!
P1llus May 15, 2020
a59503e
added changes from PR comments and mage fmt
P1llus May 17, 2020
23a5c52
small change on response header and adding documentation
P1llus May 17, 2020
219eb08
including new input docs
P1llus May 18, 2020
7f6abed
changing to older string formatting to support python version in nose…
P1llus May 18, 2020
f6032a4
wrong doc reference?
P1llus May 18, 2020
e188040
removing response header, updating docs and modify based on PR comments
P1llus May 19, 2020
94d2e27
needed to remove responseheader from defaultconf as well
P1llus May 19, 2020
111b1e3
mage fmt update
P1llus May 19, 2020
fce4e8e
validation for response body added
P1llus May 25, 2020
c9b9a81
Mage fmt update
P1llus May 25, 2020
d17e5d0
updated changelog
P1llus May 25, 2020
cffcade
Merge branch 'master' into http_input_module
P1llus May 25, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -341,6 +341,7 @@ field. You can revert this change by configuring tags for the module and omittin
- Improve ECS categorization field mappings in system module. {issue}16031[16031] {pull}18065[18065]
- Change the `json.*` input settings implementation to merge parsed json objects with existing objects in the event instead of fully replacing them. {pull}17958[17958]
- Improve ECS categorization field mappings in osquery module. {issue}16176[16176] {pull}17881[17881]
- Added http_endpoint input{pull}18298[18298]
- Add support for array parsing in azure-eventhub input. {pull}18585[18585]
- Added `observer.vendor`, `observer.product`, and `observer.type` to PANW module events. {pull}18223[18223]
- The `logstash` module can now automatically detect the log file format (JSON or plaintext) and process it accordingly. {issue}9964[9964] {pull}18095[18095]
Expand Down
3 changes: 3 additions & 0 deletions filebeat/docs/filebeat-options.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ You can configure {beatname_uc} to use the following inputs:
* <<{beatname_lc}-input-container>>
* <<{beatname_lc}-input-docker>>
* <<{beatname_lc}-input-google-pubsub>>
* <<{beatname_lc}-input-http_endpoint>>
* <<{beatname_lc}-input-httpjson>>
* <<{beatname_lc}-input-kafka>>
* <<{beatname_lc}-input-log>>
Expand All @@ -73,6 +74,8 @@ include::inputs/input-docker.asciidoc[]

include::../../x-pack/filebeat/docs/inputs/input-google-pubsub.asciidoc[]

include::../../x-pack/filebeat/docs/inputs/input-http-endpoint.asciidoc[]

include::../../x-pack/filebeat/docs/inputs/input-httpjson.asciidoc[]

include::inputs/input-kafka.asciidoc[]
Expand Down
116 changes: 116 additions & 0 deletions x-pack/filebeat/docs/inputs/input-http-endpoint.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
[role="xpack"]

:type: http_endpoint

[id="{beatname_lc}-input-{type}"]
=== HTTP Endpoint input

++++
<titleabbrev>HTTP Endpoint</titleabbrev>
++++

beta[]

Use the `http_endpoint` input to create a HTTP listener that can receive incoming HTTP POST requests.

This input can for example be used to receive incoming webhooks from a third-party application or service.

Example configurations:

Basic example:
["source","yaml",subs="attributes"]
----
{beatname_lc}.inputs:
- type: http_endpoint
enabled: true
listen_address: 192.168.1.1
listen_port: 8080
----

Custom response example:
["source","yaml",subs="attributes"]
----
{beatname_lc}.inputs:
- type: http_endpoint
enabled: true
listen_address: 192.168.1.1
listen_port: 8080
response_code: 200
response_body: '{"message": "success"}'
url: "/"
prefix: "json"
----

Basic auth and SSL example:
["source","yaml",subs="attributes"]
----
{beatname_lc}.inputs:
- type: http_endpoint
enabled: true
listen_address: 192.168.1.1
listen_port: 8080
ssl.enabled: true
ssl.certificate: "/home/user/server.pem"
ssl.key: "/home/user/server.key"
ssl.verification_mode: "none"
ssl.certificate_authority: "/home/user/ca.pem"
basic_auth: true
username: someuser
password: somepassword
----


==== Configuration options

The `http_endpoint` input supports the following configuration options plus the
<<{beatname_lc}-input-{type}-common-options>> described later.

[float]
==== `basic_auth`

Enables or disables HTTP basic auth for each incoming request. If enabled then `username` and `password` will also need to be configured.

[float]
==== `username`

If `basic_auth` is enabled, this is the username used for authentication against the HTTP listener. Requires `password` to also be set.

[float]
==== `password`

If `basic_auth` is eanbled, this is the password used for authentication against the HTTP listener. Requires `username` to also be set.

[float]
==== `response_code`

The HTTP response code returned upon success. Should be in the 2XX range.

[float]
==== `response_body`

The response body returned upon success.

[float]
==== `listen_address`

If multiple interfaces is present the `listen_address` can be set to control which IP address the listener binds to. Defaults to `127.0.0.1`.

[float]
==== `listen_port`

Which port the listener binds to. Defaults to 8000

[float]
==== `url`

This options specific which URL path to accept requests on. Defaults to `/`

[float]
==== `prefix`

This option specifies which prefix the incoming request will be mapped to.

[id="{beatname_lc}-input-{type}-common-options"]
include::../../../../filebeat/docs/inputs/input-common-options.asciidoc[]

:type!:
1 change: 1 addition & 0 deletions x-pack/filebeat/include/list.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

48 changes: 48 additions & 0 deletions x-pack/filebeat/input/http_endpoint/config.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

package http_endpoint

import (
"encoding/json"
"errors"

"github.com/elastic/beats/v7/libbeat/common/transport/tlscommon"
)

// Config contains information about httpjson configuration
type config struct {
TLS *tlscommon.ServerConfig `config:"ssl"`
BasicAuth bool `config:"basic_auth"`
Username string `config:"username"`
Password string `config:"password"`
ResponseCode int `config:"response_code" validate:"positive"`
ResponseBody string `config:"response_body"`

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we're going to hardcode that this is a JSON-based response, I think we should likely still sanity check that the user actually entered JSON. You could just make a Validate method for the config structure and inside do something like

func (c *config) Validate() error {
  if !json.Valid(c.ResponseBody) {
    return errors.New("response_body must be valid JSON")
  }
  // other validations here
  return nil 
}

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added the validation, just had to convert it from a string to a byteslice as well.

New return upon wrong response body is:

2020-05-25T08:43:01.744+0200	ERROR	instance/beat.go:931	Exiting: Failed to start crawler: starting input failed: Error while initializing input: response_body must be valid JSON accessing 'filebeat.inputs.0' (source:'filebeat.yml')
Exiting: Failed to start crawler: starting input failed: Error while initializing input: response_body must be valid JSON accessing 'filebeat.inputs.0' (source:'filebeat.yml')

ListenAddress string `config:"listen_address"`
ListenPort string `config:"listen_port"`
URL string `config:"url"`
Prefix string `config:"prefix"`
}

func defaultConfig() config {
return config{
BasicAuth: false,
Username: "",
Password: "",
ResponseCode: 200,
ResponseBody: `{"message": "success"}`,
ListenAddress: "127.0.0.1",
ListenPort: "8000",
URL: "/",
Prefix: "json",
}
}

func (c *config) Validate() error {
if !json.Valid([]byte(c.ResponseBody)) {
return errors.New("response_body must be valid JSON")
}

return nil
}
77 changes: 77 additions & 0 deletions x-pack/filebeat/input/http_endpoint/httpserver.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
// or more contributor license agreements. Licensed under the Elastic License;
// you may not use this file except in compliance with the Elastic License.

package http_endpoint

import (
"context"
"net/http"
"time"

"github.com/elastic/beats/v7/libbeat/common/transport/tlscommon"
"github.com/elastic/beats/v7/libbeat/logp"
)

type HttpServer struct {
log *logp.Logger
server *http.Server
ctx context.Context
stop context.CancelFunc
}

func (h *HttpServer) Start() {
go func() {
if h.server.TLSConfig != nil {
h.log.Infof("Starting HTTPS server on %s", h.server.Addr)
//certificate is already loaded. That's why the parameters are empty
err := h.server.ListenAndServeTLS("", "")
if err != nil && err != http.ErrServerClosed {
h.log.Fatalf("Unable to start HTTPS server due to error: %v", err)
}
} else {
h.log.Infof("Starting HTTP server on %s", h.server.Addr)
err := h.server.ListenAndServe()
if err != nil && err != http.ErrServerClosed {
h.log.Fatalf("Unable to start HTTP server due to error: %v", err)
}
}
}()
}

func (h *HttpServer) Stop() {
h.log.Info("Stopping HTTP server")
h.stop()
if err := h.server.Shutdown(h.ctx); err != nil {
h.log.Fatalf("Unable to stop HTTP server due to error: %v", err)
}
}

func createServer(in *HttpEndpoint) (*HttpServer, error) {
mux := http.NewServeMux()
responseHandler := http.HandlerFunc(in.apiResponse)
mux.Handle(in.config.URL, in.validateRequest(responseHandler))
server := &http.Server{
Addr: in.config.ListenAddress + ":" + in.config.ListenPort,
Handler: mux,
}

tlsConfig, err := tlscommon.LoadTLSServerConfig(in.config.TLS)
if err != nil {
return nil, err
}

if tlsConfig != nil {
server.TLSConfig = tlsConfig.BuildModuleConfig(in.config.ListenAddress)
}

ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
h := &HttpServer{
ctx: ctx,
stop: cancel,
log: logp.NewLogger("http_server"),
}
h.server = server

return h, nil
}
Loading