-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Filebeat][New Input] Http Input #18298
Merged
Merged
Changes from all commits
Commits
Show all changes
29 commits
Select commit
Hold shift + click to select a range
0a7972c
MVP for http input
P1llus c985208
adding temp error message on event send failure
P1llus 5102356
updated comment that was there from old input
P1llus 7c824c9
modify config
P1llus ebb8ed1
Merge branch 'master' into http_input_module
P1llus c94b6b9
changing default config
P1llus aaafd65
cleaning up the code and refactor checks to its own functions
P1llus 458dcb5
mage fmt
P1llus f51160e
Changed name from httpinput to http_endpoint, moved http server to it…
P1llus 1ccb3c2
updated code based on PR comments. Changed clientauth after confirmin…
P1llus c185b48
change packagename
P1llus 72a8652
change packagename
P1llus 7db39a9
updated code to be more idiomatic with comments from noemi
P1llus 04be474
removing variable declaration to be more idiomatic
P1llus aa27ac9
adding basic test, currently not working
P1llus 4d2cc8b
forgot to add method validation
P1llus 07c00d6
make ALLL the tests!
P1llus a59503e
added changes from PR comments and mage fmt
P1llus 23a5c52
small change on response header and adding documentation
P1llus 219eb08
including new input docs
P1llus 7f6abed
changing to older string formatting to support python version in nose…
P1llus f6032a4
wrong doc reference?
P1llus e188040
removing response header, updating docs and modify based on PR comments
P1llus 94d2e27
needed to remove responseheader from defaultconf as well
P1llus 111b1e3
mage fmt update
P1llus fce4e8e
validation for response body added
P1llus c9b9a81
Mage fmt update
P1llus d17e5d0
updated changelog
P1llus cffcade
Merge branch 'master' into http_input_module
P1llus File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
116 changes: 116 additions & 0 deletions
116
x-pack/filebeat/docs/inputs/input-http-endpoint.asciidoc
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,116 @@ | ||
[role="xpack"] | ||
|
||
:type: http_endpoint | ||
|
||
[id="{beatname_lc}-input-{type}"] | ||
=== HTTP Endpoint input | ||
|
||
++++ | ||
<titleabbrev>HTTP Endpoint</titleabbrev> | ||
++++ | ||
|
||
beta[] | ||
|
||
Use the `http_endpoint` input to create a HTTP listener that can receive incoming HTTP POST requests. | ||
|
||
This input can for example be used to receive incoming webhooks from a third-party application or service. | ||
|
||
Example configurations: | ||
|
||
Basic example: | ||
["source","yaml",subs="attributes"] | ||
---- | ||
{beatname_lc}.inputs: | ||
- type: http_endpoint | ||
enabled: true | ||
listen_address: 192.168.1.1 | ||
listen_port: 8080 | ||
---- | ||
|
||
Custom response example: | ||
["source","yaml",subs="attributes"] | ||
---- | ||
{beatname_lc}.inputs: | ||
- type: http_endpoint | ||
enabled: true | ||
listen_address: 192.168.1.1 | ||
listen_port: 8080 | ||
response_code: 200 | ||
response_body: '{"message": "success"}' | ||
url: "/" | ||
prefix: "json" | ||
---- | ||
|
||
Basic auth and SSL example: | ||
["source","yaml",subs="attributes"] | ||
---- | ||
{beatname_lc}.inputs: | ||
- type: http_endpoint | ||
enabled: true | ||
listen_address: 192.168.1.1 | ||
listen_port: 8080 | ||
ssl.enabled: true | ||
ssl.certificate: "/home/user/server.pem" | ||
ssl.key: "/home/user/server.key" | ||
ssl.verification_mode: "none" | ||
ssl.certificate_authority: "/home/user/ca.pem" | ||
basic_auth: true | ||
username: someuser | ||
password: somepassword | ||
---- | ||
|
||
|
||
==== Configuration options | ||
|
||
The `http_endpoint` input supports the following configuration options plus the | ||
<<{beatname_lc}-input-{type}-common-options>> described later. | ||
|
||
[float] | ||
==== `basic_auth` | ||
|
||
Enables or disables HTTP basic auth for each incoming request. If enabled then `username` and `password` will also need to be configured. | ||
|
||
[float] | ||
==== `username` | ||
|
||
If `basic_auth` is enabled, this is the username used for authentication against the HTTP listener. Requires `password` to also be set. | ||
|
||
[float] | ||
==== `password` | ||
|
||
If `basic_auth` is eanbled, this is the password used for authentication against the HTTP listener. Requires `username` to also be set. | ||
|
||
[float] | ||
==== `response_code` | ||
|
||
The HTTP response code returned upon success. Should be in the 2XX range. | ||
|
||
[float] | ||
==== `response_body` | ||
|
||
The response body returned upon success. | ||
|
||
[float] | ||
==== `listen_address` | ||
|
||
If multiple interfaces is present the `listen_address` can be set to control which IP address the listener binds to. Defaults to `127.0.0.1`. | ||
|
||
[float] | ||
==== `listen_port` | ||
|
||
Which port the listener binds to. Defaults to 8000 | ||
|
||
[float] | ||
==== `url` | ||
|
||
This options specific which URL path to accept requests on. Defaults to `/` | ||
|
||
[float] | ||
==== `prefix` | ||
|
||
This option specifies which prefix the incoming request will be mapped to. | ||
|
||
[id="{beatname_lc}-input-{type}-common-options"] | ||
include::../../../../filebeat/docs/inputs/input-common-options.asciidoc[] | ||
|
||
:type!: |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,48 @@ | ||
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
// or more contributor license agreements. Licensed under the Elastic License; | ||
// you may not use this file except in compliance with the Elastic License. | ||
|
||
package http_endpoint | ||
|
||
import ( | ||
"encoding/json" | ||
"errors" | ||
|
||
"github.com/elastic/beats/v7/libbeat/common/transport/tlscommon" | ||
) | ||
|
||
// Config contains information about httpjson configuration | ||
type config struct { | ||
TLS *tlscommon.ServerConfig `config:"ssl"` | ||
BasicAuth bool `config:"basic_auth"` | ||
Username string `config:"username"` | ||
Password string `config:"password"` | ||
ResponseCode int `config:"response_code" validate:"positive"` | ||
ResponseBody string `config:"response_body"` | ||
ListenAddress string `config:"listen_address"` | ||
ListenPort string `config:"listen_port"` | ||
URL string `config:"url"` | ||
Prefix string `config:"prefix"` | ||
} | ||
|
||
func defaultConfig() config { | ||
return config{ | ||
BasicAuth: false, | ||
Username: "", | ||
Password: "", | ||
ResponseCode: 200, | ||
ResponseBody: `{"message": "success"}`, | ||
ListenAddress: "127.0.0.1", | ||
ListenPort: "8000", | ||
URL: "/", | ||
Prefix: "json", | ||
} | ||
} | ||
|
||
func (c *config) Validate() error { | ||
if !json.Valid([]byte(c.ResponseBody)) { | ||
return errors.New("response_body must be valid JSON") | ||
} | ||
|
||
return nil | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,77 @@ | ||
// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
// or more contributor license agreements. Licensed under the Elastic License; | ||
// you may not use this file except in compliance with the Elastic License. | ||
|
||
package http_endpoint | ||
|
||
import ( | ||
"context" | ||
"net/http" | ||
"time" | ||
|
||
"github.com/elastic/beats/v7/libbeat/common/transport/tlscommon" | ||
"github.com/elastic/beats/v7/libbeat/logp" | ||
) | ||
|
||
type HttpServer struct { | ||
log *logp.Logger | ||
server *http.Server | ||
ctx context.Context | ||
stop context.CancelFunc | ||
} | ||
|
||
func (h *HttpServer) Start() { | ||
go func() { | ||
if h.server.TLSConfig != nil { | ||
h.log.Infof("Starting HTTPS server on %s", h.server.Addr) | ||
//certificate is already loaded. That's why the parameters are empty | ||
err := h.server.ListenAndServeTLS("", "") | ||
if err != nil && err != http.ErrServerClosed { | ||
h.log.Fatalf("Unable to start HTTPS server due to error: %v", err) | ||
} | ||
} else { | ||
h.log.Infof("Starting HTTP server on %s", h.server.Addr) | ||
err := h.server.ListenAndServe() | ||
if err != nil && err != http.ErrServerClosed { | ||
h.log.Fatalf("Unable to start HTTP server due to error: %v", err) | ||
} | ||
} | ||
}() | ||
} | ||
|
||
func (h *HttpServer) Stop() { | ||
h.log.Info("Stopping HTTP server") | ||
h.stop() | ||
if err := h.server.Shutdown(h.ctx); err != nil { | ||
h.log.Fatalf("Unable to stop HTTP server due to error: %v", err) | ||
} | ||
} | ||
|
||
func createServer(in *HttpEndpoint) (*HttpServer, error) { | ||
mux := http.NewServeMux() | ||
responseHandler := http.HandlerFunc(in.apiResponse) | ||
mux.Handle(in.config.URL, in.validateRequest(responseHandler)) | ||
server := &http.Server{ | ||
Addr: in.config.ListenAddress + ":" + in.config.ListenPort, | ||
Handler: mux, | ||
} | ||
|
||
tlsConfig, err := tlscommon.LoadTLSServerConfig(in.config.TLS) | ||
if err != nil { | ||
return nil, err | ||
} | ||
|
||
if tlsConfig != nil { | ||
server.TLSConfig = tlsConfig.BuildModuleConfig(in.config.ListenAddress) | ||
} | ||
|
||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) | ||
h := &HttpServer{ | ||
ctx: ctx, | ||
stop: cancel, | ||
log: logp.NewLogger("http_server"), | ||
} | ||
h.server = server | ||
|
||
return h, nil | ||
} |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we're going to hardcode that this is a JSON-based response, I think we should likely still sanity check that the user actually entered JSON. You could just make a
Validate
method for theconfig
structure and inside do something likeThere was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added the validation, just had to convert it from a string to a byteslice as well.
New return upon wrong response body is: