Add HTTP listener service input plugin #1407

bagelswitch · 2016-06-23T00:49:58Z

The HTTP listener is a service input plugin that listens for messages sent via HTTP POST.
The plugin expects messages in the InfluxDB line-protocol ONLY, other Telegraf input data formats are not supported.
The intent of the plugin is to allow Telegraf to serve as a proxy/router for the /write endpoint of the InfluxDB HTTP API.

Example usage:

curl -i -X POST 'http://localhost:8186/write' --data-binary 'cpu_load_short,host=server01,region=us-west value=0.64 1434055562000000000'

Required for all PRs:

CHANGELOG.md updated
Sign CLA (if not already signed)
README.md updated (if adding a new plugin)

sparrc · 2016-06-23T07:45:10Z

Godeps

@@ -25,6 +25,7 @@ github.com/gorilla/mux c9e326e2bdec29039a3761c07bece13133863e1e
 github.com/hailocab/go-hostpool e80d13ce29ede4452c43dea11e79b9bc8a15b478
 github.com/hashicorp/consul 5aa90455ce78d4d41578bafc86305e6e6b28d7d2
 github.com/hpcloud/tail b2940955ab8b26e19d43a43c4da0475dd81bdb56
+github.com/hydrogen18/stoppableListener dadc9ccc400c712e5a316107a5c462863919e579


this is only ~40 lines of code, you should just copy it in-repo instead of as a dependency

Makes sense, done!

sparrc · 2016-06-23T15:49:37Z

plugins/inputs/http_listener/http_listener.go

+				res.WriteHeader(500)
+				res.Write([]byte("ERROR parsing metrics"))
+			}
+			res.WriteHeader(204)


for these numbers you should use the http package constants, http.StatusNoContent, http.StatusOK, etc.

kaey · 2016-07-06T11:18:59Z

There is https://github.com/influxdata/influxdb-relay
Is this plugin really needed?

bagelswitch · 2016-07-06T18:07:16Z

@kaey, in our use case the telegraf agent is already co-located with, for example, a containerized application instance in a PaaS - it sits either w/in the instance container, or on the container host, and we use multiple collectors to gather host, middleware, and application metrics from the instance - with this plugin, influx metrics emitters w/in the applications are also just pointed at the local listener - so we get a single stream of metrics covering the complete stack for this application instance that we can direct somewhere.

So in our view, this is an agent/forward-proxy use case, not an HA/reverse-proxy case where influxdb-relay would be appropriate (in fact we will use both, pointing these telegraf instances at balanced influxdb-relay instances for HA).

kaey · 2016-07-07T12:47:10Z

plugins/inputs/http_listener/http_listener.go

+		WriteTimeout: time.Duration(writeTimeout) * time.Second,
+	}
+
+	err = server.Serve(t.listener)


return server.Serve(t.listener)

bagelswitch · 2016-07-08T17:06:11Z

Addressed your comments, @kaey

kaey · 2016-07-11T10:21:02Z

plugins/inputs/http_listener/http_listener.go

+
+type HttpListener struct {
+	ServiceAddress string
+	ReadTimeout    string


Type of timeout fields should be time.Duration

bagelswitch · 2016-07-12T17:12:19Z

Addressed your further comments, @kaey

jackzampolin · 2016-07-20T23:49:20Z

I was able to quickly get this up and running to forward points from influx_stress through telegraf to influxdb.

FWIW I was able to get throughput on the plugin of 65k points a second locally while breaking the 10k point batches I was sending it into 1k point batches for writing to influx. For passing through 10k point batches throughput was closer to 80k points a second.

When writing 10 point batches and outputting 10k point batches, more the design point, testing locally the plugin was easily able to handle 4k requests a second. It also pegged 2 of my CPUs.

I also tested two telegraf instances configured as follows:

$ telegraf -sample-config -input filter http_listener -output-filter influxdb > telegraf1.conf
$ telegraf -sample-config -input filter cpu:mem:disk -output-filter influxdb > telegraf2.conf

And after just changing the output address for the instance collecting cpu:mem:disk the relaying function worked just fine.

I think this looks good! LGTM 👍

kaey · 2016-07-21T07:32:46Z

LGTM

superbool · 2016-08-04T14:11:28Z

why not support json format and when it will be published ? thank you !

bagelswitch · 2016-08-04T17:46:56Z

@superbool - I didn't think it would make sense to support other formats in this particular input, as it's replicating the InfluxDB write endpoint, which only supports the line protocol.

kostasb · 2016-08-22T15:06:00Z

A note that needs to go into documentation for this plugin when/if merged:

When chaining Telegraf instances, CREATE DATABASE requests receive 200 OK with message body `{"results":[]}` but they are not relayed. The edge Telegraf instance, meaning the one that will utterly submit data to InfluxDB, determines the destination database.

The PR currently needs a re-base due to new commits in the master branch.

swang-pro · 2016-08-25T15:55:21Z

Great news ! When will this PR be integrated in an official release ?
Thank you

harlow · 2016-08-28T17:11:32Z

+1 on getting this into next release

ddes · 2016-08-29T13:05:00Z

+1, looks good after some tests. Thanks for your work, I really need it.

maccman · 2016-08-31T15:11:00Z

👍 - would love to see this merged

also remove locking around adding metrics. Instead, keep a waitgroup on the ServeHTTP function and wait for that to finish before returning from the Stop() function closes #1407

sparrc · 2016-09-06T14:30:28Z

@bagelswitch just an FYI that I've made a couple test additions, removed the locks around adding metrics (I had previously done this for the udp and tcp listeners as well), and squashed commits

this PR will be closed once I've merged that (will be done by the end of today) #1715

also remove locking around adding metrics. Instead, keep a waitgroup on the ServeHTTP function and wait for that to finish before returning from the Stop() function closes #1407

ncohensm added 10 commits June 22, 2016 14:24

initial http_listener implementation

933bdaa

fix incredibly stupid bugs

c4e0452

populate README

6b5a9d3

support query endpoint and change default listen port

6e79135

set response headers for query endpoint

a0cb608

add unit tests

121d71f

revert erroneous Godeps change

ac9e653

add plugin ref to top-level README

e9d3870

remove debug output and add empty post body test

cf08b56

fix linter errors

2f36631

sparrc reviewed Jun 23, 2016
View reviewed changes

move stoppableListener into repo

2dee84d

sparrc reviewed Jun 23, 2016
View reviewed changes

ncohensm added 4 commits June 23, 2016 08:59

use constants for http status codes

13c4f95

add CHANGELOG entry

7c17468

Merge branch 'master' of https://github.com/influxdata/telegraf

5866acd

Merge branch 'master' into master

bb09363

kaey reviewed Jul 7, 2016
View reviewed changes

ncohensm added 2 commits July 8, 2016 09:55

address code review comments re. style/structure

6da73de

Merge branch 'master' of https://github.com/bagelswitch/telegraf

7843c3a

kaey reviewed Jul 11, 2016
View reviewed changes

ncohensm added 2 commits July 12, 2016 09:35

address further code review comments

4b424a2

Merge branch 'master' of https://github.com/influxdata/telegraf

4a812b0

sparrc mentioned this pull request Jul 12, 2016

InfluxDB input #1485

Closed

jackzampolin mentioned this pull request Jul 14, 2016

output plugin to support multi-stage Telegraf architecture #1421

Closed

bagelswitch added 3 commits July 18, 2016 08:28

Merge branch 'master' into master

51215b5

Merge branch 'master' into master

0eebee9

Merge branch 'master' of https://github.com/influxdata/telegraf

2d460d7

ncohensm added 2 commits July 25, 2016 11:41

Merge branch 'master' of https://github.com/influxdata/telegraf

f715c17

correct linter error after merge

54385d0

Merge branch 'master' of https://github.com/influxdata/telegraf

739e0af

kostasb mentioned this pull request Aug 22, 2016

should be able to squelch "CREATE DATABASE" statements #1655

Closed

ncohensm added 3 commits August 23, 2016 10:00

Merge branch 'master' of https://github.com/influxdata/telegraf

bbeb5ae

add note to README re. database creation calls per PR comments

d660903

Merge branch 'master' of https://github.com/influxdata/telegraf

142818a

theist mentioned this pull request Sep 4, 2016

New Tcp Forwarder Output plugin. #1526

Closed

sparrc closed this in 301c79e Sep 6, 2016

sparrc added this to the 1.1.0 milestone Sep 7, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add HTTP listener service input plugin #1407

Add HTTP listener service input plugin #1407

bagelswitch commented Jun 23, 2016 •

edited

Loading

sparrc Jun 23, 2016

bagelswitch Jun 23, 2016 •

edited

Loading

sparrc Jun 23, 2016

bagelswitch Jun 23, 2016

kaey commented Jul 6, 2016

bagelswitch commented Jul 6, 2016

kaey Jul 7, 2016

bagelswitch commented Jul 8, 2016

kaey Jul 11, 2016

bagelswitch commented Jul 12, 2016

jackzampolin commented Jul 20, 2016 •

edited

Loading

kaey commented Jul 21, 2016

superbool commented Aug 4, 2016

bagelswitch commented Aug 4, 2016

kostasb commented Aug 22, 2016

swang-pro commented Aug 25, 2016

harlow commented Aug 28, 2016

ddes commented Aug 29, 2016

maccman commented Aug 31, 2016

sparrc commented Sep 6, 2016

Add HTTP listener service input plugin #1407

Add HTTP listener service input plugin #1407

Conversation

bagelswitch commented Jun 23, 2016 • edited Loading

Required for all PRs:

sparrc Jun 23, 2016

Choose a reason for hiding this comment

bagelswitch Jun 23, 2016 • edited Loading

Choose a reason for hiding this comment

sparrc Jun 23, 2016

Choose a reason for hiding this comment

bagelswitch Jun 23, 2016

Choose a reason for hiding this comment

kaey commented Jul 6, 2016

bagelswitch commented Jul 6, 2016

kaey Jul 7, 2016

Choose a reason for hiding this comment

bagelswitch commented Jul 8, 2016

kaey Jul 11, 2016

Choose a reason for hiding this comment

bagelswitch commented Jul 12, 2016

jackzampolin commented Jul 20, 2016 • edited Loading

kaey commented Jul 21, 2016

superbool commented Aug 4, 2016

bagelswitch commented Aug 4, 2016

kostasb commented Aug 22, 2016

swang-pro commented Aug 25, 2016

harlow commented Aug 28, 2016

ddes commented Aug 29, 2016

maccman commented Aug 31, 2016

sparrc commented Sep 6, 2016

bagelswitch commented Jun 23, 2016 •

edited

Loading

bagelswitch Jun 23, 2016 •

edited

Loading

jackzampolin commented Jul 20, 2016 •

edited

Loading