Expose metrics through http endpoint

The following PR exposes beats metrics through a configurable http endpoint. This allows when enabled to get an insight into a running beat. For security reasons the endpoint is off by default.

**Configuration**

The configuration options are in the `http` namespace. This config naming is borrowed from Logstash. By default the http endpoint is disabled. If enabled the metrics are only exposed on localhost on port 5066.

```
http.enabled: false
http.host: localhost
http.port: 5066
```

**-httpprof ?**

The http endpoint can be enabled also in production if needed. The additional endpoint httpprof endpoint which can be enabled through `-httpprof` still exists but is only recommended for debugging purpose. The httpprof endpoint exposes many more metrics and runtime data then the metrics endpoint.

**Endpoints**

The current implementation has two endpoints:

* `/`: The standard endpoint exposes info about the beat
* `/stats`: Stats exposes all metrics collected by monitoring

The output of the data is in json. The flag `?pretty` can be used to have formatted json as output to make it more human readable. Below is an example of each endpoint.

**/**

```
{
  "beat": "metricbeat",
  "hostname": "ruflin",
  "name": "ruflin",
  "uuid": "9d6e0c3a-1677-424c-aead-097e597e09f9",
  "version": "6.0.0-alpha1"
}
```

**/stat**

```
{
  "beat": {
    "memstats": {
      "gc_next": 6262080,
      "memory_alloc": 3879968,
      "memory_total": 479284520
    }
  },
  "libbeat": {
    "config": {
      "module": {
        "running": 0,
        "starts": 0,
        "stops": 0
      },
      "reloads": 0
    }
  },
  "metricbeat": {
    "system": {
      "cpu": {
        "events": 7,
        "failures": 0,
        "success": 7
      },
      "filesystem": {
        "events": 28,
        "failures": 0,
        "success": 7
      },
      "fsstat": {
        "events": 7,
        "failures": 0,
        "success": 7
      },
      "load": {
        "events": 7,
        "failures": 0,
        "success": 7
      },
      "memory": {
        "events": 7,
        "failures": 0,
        "success": 7
      },
      "network": {
        "events": 70,
        "failures": 0,
        "success": 7
      },
      "process": {
        "events": 1324,
        "failures": 0,
        "success": 7
      }
    }
  },
  "output": {
    "elasticsearch": {
      "events": {
        "acked": 1445,
        "not_acked": 0
      },
      "publishEvents": {
        "call": {
          "count": 34
        }
      },
      "read": {
        "bytes": 17213,
        "errors": 0
      },
      "write": {
        "bytes": 1185991,
        "errors": 0
      }
    },
    "events": {
      "acked": 1445
    },
    "kafka": {
      "events": {
        "acked": 0,
        "not_acked": 0
      },
      "publishEvents": {
        "call": {
          "count": 0
        }
      }
    },
    "logstash": {
      "events": {
        "acked": 0,
        "not_acked": 0
      },
      "publishEvents": {
        "call": {
          "count": 0
        }
      },
      "read": {
        "bytes": 0,
        "errors": 0
      },
      "write": {
        "bytes": 0,
        "errors": 0
      }
    },
    "messages": {
      "dropped": 0
    },
    "redis": {
      "events": {
        "acked": 0,
        "not_acked": 0
      },
      "read": {
        "bytes": 0,
        "errors": 0
      },
      "write": {
        "bytes": 0,
        "errors": 0
      }
    },
    "write": {
      "bytes": 1185991,
      "errors": 0
    }
  },
  "publisher": {
    "events": {
      "count": 1450
    },
    "queue": {
      "messages": {
        "count": 1450
      }
    }
  }
}
```

**Questions**

* Are these good endpoints paths?
* Which should be our default port?

codecov.yml

@@ -6,7 +6,7 @@ coverage:
       default:
         # basic
         target: auto
-        threshold: null
+        threshold: 0.1
         base: auto
         # advanced
         branches: null

filebeat/filebeat.full.yml

@@ -938,6 +938,19 @@ output.elasticsearch:
 # dashboards and index pattern. Example: testbeat-*
 #dashboards.index:
 
+#================================ HTTP Endpoint ======================================
+# Each beat can expose internal data points through a http endpoint. For security
+# reason the endpoint is disabled by default. This feature is currently in beta.
+
+# Defines if http endpoint is enabled
+#http.enabled: false
+
+# Host to expose the http endpoint to. It is recommended to use only localhost.
+#http.host: localhost
+
+# Port on which the http endpoint is exposed. Default is 5066
+#http.port: 5066
+
 #================================ Logging ======================================
 # There are three options for the log output: syslog, file, stderr.
 # Under Windows systems, the log files are per default sent to the file output,

heartbeat/heartbeat.full.yml

@@ -786,6 +786,19 @@ output.elasticsearch:
 # dashboards and index pattern. Example: testbeat-*
 #dashboards.index:
 
+#================================ HTTP Endpoint ======================================
+# Each beat can expose internal data points through a http endpoint. For security
+# reason the endpoint is disabled by default. This feature is currently in beta.
+
+# Defines if http endpoint is enabled
+#http.enabled: false
+
+# Host to expose the http endpoint to. It is recommended to use only localhost.
+#http.host: localhost
+
+# Port on which the http endpoint is exposed. Default is 5066
+#http.port: 5066
+
 #================================ Logging ======================================
 # There are three options for the log output: syslog, file, stderr.
 # Under Windows systems, the log files are per default sent to the file output,

libbeat/_meta/config.full.yml

@@ -588,6 +588,19 @@ output.elasticsearch:
 # dashboards and index pattern. Example: testbeat-*
 #dashboards.index:
 
+#================================ HTTP Endpoint ======================================
+# Each beat can expose internal data points through a http endpoint. For security
+# reason the endpoint is disabled by default. This feature is currently in beta.
+
+# Defines if http endpoint is enabled
+#http.enabled: false
+
+# Host to expose the http endpoint to. It is recommended to use only localhost.
+#http.host: localhost
+
+# Port on which the http endpoint is exposed. Default is 5066
+#http.port: 5066
+
 #================================ Logging ======================================
 # There are three options for the log output: syslog, file, stderr.
 # Under Windows systems, the log files are per default sent to the file output,

libbeat/api/config.go

@@ -0,0 +1,15 @@
+package api
+
+type Config struct {
+	Enabled bool
+	Host    string
+	Port    int
+}
+
+var (
+	DefaultConfig = Config{
+		Enabled: false,
+		Host:    "localhost",
+		Port:    5066,
+	}
+)

libbeat/api/server.go

@@ -0,0 +1,73 @@
+package api
+
+import (
+	"fmt"
+	"net/http"
+	"net/url"
+	"strconv"
+
+	"github.com/elastic/beats/libbeat/common"
+	"github.com/elastic/beats/libbeat/logp"
+	"github.com/elastic/beats/libbeat/monitoring"
+)
+
+// Start starts the metrics api endpoint on the configured host and port
+func Start(cfg *common.Config, info common.BeatInfo) {
+	logp.Beta("Metrics endpoint is enabled.")
+	config := DefaultConfig
+	cfg.Unpack(&config)
+
+	logp.Info("Starting stats endpoint")
+	go func() {
+		mux := http.NewServeMux()
+
+		// register handlers
+		mux.HandleFunc("/", rootHandler(info))
+		mux.HandleFunc("/stats", statsHandler)
+
+		url := config.Host + ":" + strconv.Itoa(config.Port)
+		logp.Info("URL: %s", url)
+		endpoint := http.ListenAndServe(url, mux)
+		logp.Info("finished starting stats endpoint: %v", endpoint)
+	}()
+}
+
+func rootHandler(info common.BeatInfo) func(http.ResponseWriter, *http.Request) {
+	return func(w http.ResponseWriter, r *http.Request) {
+		// Return error page
+		if r.URL.Path != "/" {
+			http.NotFound(w, r)
+			return
+		}
+
+		w.Header().Set("Content-Type", "application/json; charset=utf-8")
+
+		data := common.MapStr{
+			"version":  info.Version,
+			"beat":     info.Beat,
+			"name":     info.Name,
+			"uuid":     info.UUID,
+			"hostname": info.Hostname,
+		}
+
+		print(w, data, r.URL)
+	}
+}
+
+// statsHandler report expvar and all libbeat/monitoring metrics
+func statsHandler(w http.ResponseWriter, r *http.Request) {
+	w.Header().Set("Content-Type", "application/json; charset=utf-8")
+
+	data := monitoring.CollectStructSnapshot(nil, monitoring.Full, false)
+
+	print(w, data, r.URL)
+}
+
+func print(w http.ResponseWriter, data common.MapStr, u *url.URL) {
+	query := u.Query()
+	if _, ok := query["pretty"]; ok {
+		fmt.Fprintf(w, data.StringToPrint())
+	} else {
+		fmt.Fprintf(w, data.String())
+	}
+}

libbeat/beat/beat.go

@@ -46,6 +46,7 @@ import (
 	"github.com/satori/go.uuid"
 
 	"github.com/elastic/beats/filebeat/input/file"
+	"github.com/elastic/beats/libbeat/api"
 	"github.com/elastic/beats/libbeat/cfgfile"
 	"github.com/elastic/beats/libbeat/common"
 	"github.com/elastic/beats/libbeat/dashboards"
@@ -110,6 +111,7 @@ type BeatConfig struct {
 	Processors processors.PluginConfig   `config:"processors"`
 	Path       paths.Path                `config:"path"`
 	Dashboards *common.Config            `config:"dashboards"`
+	Http       *common.Config            `config:"http"`
 }
 
 var (
@@ -252,6 +254,10 @@ func (b *Beat) launch(bt Creator) error {
 	defer logp.Info("%s stopped.", b.Info.Beat)
 	defer logp.LogTotalExpvars(&b.Config.Logging)
 
+	if b.Config.Http.Enabled() {
+		api.Start(b.Config.Http, b.Info)
+	}
+
 	return beater.Run(b)
 }
 

metricbeat/metricbeat.full.yml

@@ -920,6 +920,19 @@ output.elasticsearch:
 # dashboards and index pattern. Example: testbeat-*
 #dashboards.index:
 
+#================================ HTTP Endpoint ======================================
+# Each beat can expose internal data points through a http endpoint. For security
+# reason the endpoint is disabled by default. This feature is currently in beta.
+
+# Defines if http endpoint is enabled
+#http.enabled: false
+
+# Host to expose the http endpoint to. It is recommended to use only localhost.
+#http.host: localhost
+
+# Port on which the http endpoint is exposed. Default is 5066
+#http.port: 5066
+
 #================================ Logging ======================================
 # There are three options for the log output: syslog, file, stderr.
 # Under Windows systems, the log files are per default sent to the file output,

packetbeat/packetbeat.full.yml

@@ -1043,6 +1043,19 @@ output.elasticsearch:
 # dashboards and index pattern. Example: testbeat-*
 #dashboards.index:
 
+#================================ HTTP Endpoint ======================================
+# Each beat can expose internal data points through a http endpoint. For security
+# reason the endpoint is disabled by default. This feature is currently in beta.
+
+# Defines if http endpoint is enabled
+#http.enabled: false
+
+# Host to expose the http endpoint to. It is recommended to use only localhost.
+#http.host: localhost
+
+# Port on which the http endpoint is exposed. Default is 5066
+#http.port: 5066
+
 #================================ Logging ======================================
 # There are three options for the log output: syslog, file, stderr.
 # Under Windows systems, the log files are per default sent to the file output,

winlogbeat/winlogbeat.full.yml

@@ -623,6 +623,19 @@ output.elasticsearch:
 # dashboards and index pattern. Example: testbeat-*
 #dashboards.index:
 
+#================================ HTTP Endpoint ======================================
+# Each beat can expose internal data points through a http endpoint. For security
+# reason the endpoint is disabled by default. This feature is currently in beta.
+
+# Defines if http endpoint is enabled
+#http.enabled: false
+
+# Host to expose the http endpoint to. It is recommended to use only localhost.
+#http.host: localhost
+
+# Port on which the http endpoint is exposed. Default is 5066
+#http.port: 5066
+
 #================================ Logging ======================================
 # There are three options for the log output: syslog, file, stderr.
 # Under Windows systems, the log files are per default sent to the file output,