This repository is made available to support users wishing to experiment with consumption of streaming telemetry using off-the-shelf stacks.
Three different stacks are prepackaged in this repository:
- The
stack_elkstack deploys a fleet of docker containers with elasticsearch, logstash and kibana. - The
stack_prometheusstack deploys a fleet of docker containers with logstash, prometheus, pushgateway and promdash. - The
stack_signalfxstack deploys alogstashcontainer configured to feed telemetry into the cloud based Signal Fx monitoring system. Note that, while a free trial is available, the SignalFx monitoring service is not free.
A very thin stack (i.e. logstash set up with telemetry input codecs, and kafka output plugin configuration) is also provided in order to publish telemetry content on to a kafka bus.
The systems are independent and can be deployed independently or together (UDP/TCP stream endpoint ports may need to be changed in environment files).
Note: The streaming telemetry project is work in progress, and both the on and off box components of streaming telemetry are likely to evolve at a fast pace.
You will need a working git and docker setup; look for "Docker tips" below if you need help with this. If host is behind an HTTP proxy, refer to the pertinent section below.
Clone the repository, pick the stack you would like to run, and follow the steps below. I use stack_elk as an example, but the same applies for stack_prometheus, stack_signalfx or stack_kafka:
git clone https://github.com/cisco/bigmuddy-network-telemetry-stacks.git
#
# Change to the directory for the stack of your choice e.g.:
#
cd stack_elk
sudo COLLECTOR=a.b.c.d ./stack_build
where a.b.c.d is the local IP address you wish to use. stack_build is executed to set up the default configurations, build the docker images etc, and is only required once.
By default, the ELK stack supports model-driven-telemetry (MDT) in XR 6.1.1. To use the stacks for telemetry in XR 6.0.0 and XR 6.0.1, you will need to modify bigmuddy-network-telemetry-stacks/stack_elk/src/ls_telemetry.conf by changing mdt => true to mdt => false
Start the fleet of containers using:
sudo ./stack_run
This is it.
At this point your stack should be running and telemetry streams can be pointed at it.
If you are using default configurations, port 2103 will be consumed by stack_elk, port 2104 will be consumed by stack_prometheus, and port 2105 by stack_signalfx. TCP supports compressed JSON, whereas UDP supports protobuf (ELK stack only). The output plugin telemetry_metrics used by stack_prometheus and stack_elk does not support content exported over protobuf.
###Using Google Protocol Buffer (GPB) encoding.
If you are using GPB encoding, place the appropriate .proto files in the location /var/local/stack_elk/logstash_data/proto between running stack_build and stack_run. If the files change at an point, restart the stack - this will cause the ruby bindings to be rebuilt. To prevent namespace overlap, only copy the .proto for the encoding and version that you need.
- For "encoding self-describing-gpb" with model-driven telemetry (MDT) in XR 6.1 onward, copy
bigmuddy-network-telemetry-stacks/protos/gpb_bis/telemetry_bis.prototo/var/local/stack_elk/logstash_data/proto - For key-value gpb encoding over TCP with policy-driven telemetry (PDT) in XR 6.0, copy
bigmuddy-network-telemetry-stacks/protos/gpb_kv/telemetry_kv.prototo/var/local/stack_elk/logstash_data/proto - For compact gpb encoding over UDP with policy-driven telemetry (PDT) in XR 6.0, copy
telemetry.proto,cisco.protoanddescriptor.protofrombigmuddy-network-telemetry-stacks/protos/gpb_compact/resources/xr6.0.0to/var/local/stack_elk/logstash_data/proto. In addition, copy any router-generated .protos to the same directory. Some example router-generated protos can be found inbigmuddy-network-telemetry-stacks/protos/gpb_compact/resources/xr6.0.0/protos/
Stopping the stack involves running stack_stop.
sudo ./stack_stop
The stack can be started and stopped over and over (no intervening stack_build required). Configuration and data is preserved across stop/run cycles in the host mounted volumes. A host mounted volume is a per-stack-component directory which is mapped into the container.
If the stack is (re)built, configuration files which are copied from the repository are regenerated and any changes to those files in the host mounted volume will be overwritten. Any other data is preserved unless the host mounted volumes are deleted explicitly. If you wish to purge all data and configuration to start from scratch, simply delete the host mounted volumes, and rerun stack_build.
If you are using the default setup, and specified a collector address of a.b.c.d, then;
- for
stack_elkthen you can access kibana by pointing your browser at;
http://a.b.c.d:5601/
- for
stack_prometheusthen you can access promdash by pointing your browser at:
http://a.b.c.d:3000/
- to access prometheus or the gateways directly you can also point at ports 9090, 9091 and 9092 by default.
Below are a couple of dashboard images; one from stack_elk and one from stack_prometheus:
In order to use stack_signalfx, registration is required at https://signalfx.com/. Once registered, an organisation API Token can be retrieved from the profile page. This token should be setup in the stack_signalfx/src/environment as shown here (note the token is not made up in the example):
export SIGNALFXTOKEN="DuMMyExaMPLeT0KEn"
Streams should be pointed at the logstash setup as for the other stacks. Go to https://app.signalfx.com/ to visualise the data. The Usage Metric dashboard should show some number of datapoints received per second. Below is an example of dashboard setup to show IP SLA and interface counter data.
The minimum required configuration for the kafka stack is, as with all stacks, listed at the top of the environment file, and needs to be set prior to stack_build. Alternatively it can be set in the running configuration; by default /var/local/stack_kafka/logstash_data/conf.d/ls_telemetry.conf. Changes here would be overwritten if the stack is rebuilt.
We have made the logstash codecs and output plugins used in the stacks in this repository available independently of the stacks. Pertinent repositories are logstash-codec-bigmuddy-network-telemetry, logstash-codec-bigmuddy-network-telemetry-gpb and logstash-output-bigmuddy-network-telemetry-metrics. These plugins are expected to be published as Ruby gems shortly.
The plugins should be useful if you are assembling your own stack and using logstash. For example, the codec plugin has been used in logstash to collect telemetry streams and publish content onto a kafka bus. The same logstash codec plugin was used to collect telemetry and feed it in to Splunk.
While NOT required, a large degree of customisation is possible.
For emphasis, the above should be all you need to do in order to feed telemetry streams to the collector stack.
There are two aspects of customisation possible:
-
the build installation is customised by modifying the files in
srcdirectory for each stack. Primarily, the set of attributes insrc/environment(e.g. portslogstashlistens on, host volumes to use etc) dictate what is set up. These configuration files can be tweaked to influence the individual components in thesrcdirectory preserving the changes acrossstack_builditerations.. -
on top of build time customisation, the configuration files for the various components are staged in host mounted volumes, and can be modified in situ in the host mounted volumes. As per above, note that those files are regenerated with every
stack_builditeration, and any changes made to the files in the host mounted volume will be lost. Default host mounted volumes for each component per stack are here:
/var/local/stack_elk/elasticsearch_data
/var/local/stack_elk/kibana_data
/var/local/stack_elk/logstash_data
/var/local/stack_prometheus/logstash_data
/var/local/stack_prometheus/prometheus_data
/var/local/stack_prometheus/promdash_data
/var/local/stack_signalfx/logstash_data
The host mounted volume location can be changed by modifying src/environment should it be necessary. Volumes are set up at build time if they do not exist. Do note, that data will be preserved across stack_run/stack_stop cycles, This means that if the host volume is archived, it should be possible to snapshot the state of the stack.
An example of an interesting configuration file (re)generated at build time is the logstash pipeline configuration for each stack:
/var/local/stack_elk/logstash_data/conf.d/ls_telemetry.conf
/var/local/stack_prometheus/logstash_data/conf.d/ls_telemetry.conf
/var/local/stack_signalfx/logstash_data/conf.d/ls_telemetry.conf
Is the host where you are installing the stack behind an HTTP proxy? There is section below with instructions if so.
Remember to check out the docker troubleshooting tips below too.
Logs are written to <stack>/log/*.log. These logs may shed some light and include the execution command to run docker containers - this can be useful if you wish to start and stop individual containers.
The section does not include per-component troubleshooting, but it is worth mentioning, that for logstash, creating the file <host volume>/logstash.overrides and including the line LS_OPTS="--debug" will generate per message logging in <host volume>/logstash.log. Remember to ./stack_stop and ./stack_run to pick up overrides.
For the stack you are using, simply edit the src/environment file and populate the proxy as per the example.
Alternatively set http_proxy and https_proxy as environment variables, but note that, by default, those environment variables will not be propagated across sudo. One way to workaround this restriction is to pass them in explicitly on the command line as follows. For example:
sudo COLLECTOR=a.b.c.d http_proxy=http://example.org:80 https_proxy=http://example.org:80 ./stack_build
The stack_prometheus subsystem is built on top of the un-versioned prometheus components in the docker registry. At some point, we will move to versioned instances.
We start by making sure that git and docker are set up correctly. From a fresh installation (say an Ubuntu distribution), we might first install docker:
sudo apt-get install git
wget -qO- https://get.docker.com/ | sh
sudo service docker start
Ensure that /etc/default/docker is updated with the following, if you are using a desktop Ubuntu distribution:
DOCKER_OPTS="--dns <DNSSERVER> --dns 8.8.8.8 --dns 8.8.4.4"
export http_proxy=http://<HTTPPROXY>:80/
The http_proxy setup is required if behind a proxy.
Do remember to restart docker (e.g. sudo service docker restart).
Handy docker commands include ps -a showing container instances, inspect <name from ps e.g. stack_elk_logstash> displaying the runtime configurtion like volume bindings, state etc for a container, and logs <name from ps e.g. stack_prometheus_prometheus> which dumps the output from executed process.
With both stack_elk and stack_prometheus installed, the following set of images and containers should show up:
collector:~$ sudo docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
logstash 1.5.2 b36a820946e4 4 minutes ago 1.029 GB
kibana 4.1.1 53561efe97d6 9 minutes ago 262.7 MB
<none> <none> 071dd8fc171a 10 minutes ago 1.029 GB
elasticsearch 1.6.0 8f2207fb95ca 28 hours ago 514.8 MB
java 8 49ebfec495e1 31 hours ago 816.4 MB
prom/pushgateway latest 47190258fc7c 2 days ago 24.61 MB
prom/promdash latest db9f914e1858 4 days ago 169.1 MB
ubuntu 14.04 d2a0ecffe6fa 5 days ago 188.4 MB
prom/prometheus latest 8c05d89135bd 6 weeks ago 37.29 MB
collector:~$ sudo docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
dc15dc08bf54 logstash:1.5.2 "/bin/sh -c '/start. 4 minutes ago Up 4 minutes stack_prometheus_logstash
f1e9c3eaed11 prom/promdash "./run ./bin/thin st 4 minutes ago Up 4 minutes 0.0.0.0:3000->3000/tcp stack_prometheus_promdash
78c9a3259a09 prom/prometheus "/bin/prometheus -co 5 minutes ago Up 5 minutes 0.0.0.0:9090->9090/tcp stack_prometheus_prometheus
14bbcc3182db prom/pushgateway "/bin/go-run" 5 minutes ago Up 5 minutes 0.0.0.0:9092->9091/tcp stack_prometheus_pushgw_lpr
4a28535c2ea8 prom/pushgateway "/bin/go-run" 5 minutes ago Up 5 minutes 0.0.0.0:9091->9091/tcp stack_prometheus_pushgw
76317c87a98c logstash:1.5.2 "/bin/sh -c '/start. 7 minutes ago Up 7 minutes stack_elk_logstash
0916ea88edd8 kibana:4.1.1 "/bin/sh -c '/start. 7 minutes ago Up 7 minutes 0.0.0.0:5601->5601/tcp stack_elk_kibana
6e183e68de67 elasticsearch:1.6.0 "/docker-entrypoint. 7 minutes ago Up 7 minutes 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp stack_elk_elasticsearch