Move scripts to dev-tools de manage Kibana dashboards (#1359)

- Add bash/powershell scripts to import and export the dashboards and its
dependencies in Kibana.
- Run `make import-dashboards` in a Beat repository to import all the dashboards and its dependencies from etc/kibana to Elasticsearch
- Run `make export-dashboards` in a Beat repository to export all the dashboards and its dependencies  from etc/kibana to Elasticsearch
- Add documentation
- Update changelog
- Recommend to use testing/environments when creating dashboards.

CHANGELOG.asciidoc

@@ -13,6 +13,7 @@ https://github.com/elastic/beats/compare/v5.0.0-alpha1...master[Check the HEAD d
 ==== Breaking changes
 
 *Affecting all Beats*
+- Add scripts for managing the dashboards of a single Beat {pull}1359[1359]
 
 *Packetbeat*
 - Configuration of redis topology support changed. {pull}1353[1353]

dev-tools/README.md

@@ -16,3 +16,14 @@ Other scripts:
 |----------------------|-------------|
 | aggregate_coverage.py | Used to create coverage reports that contain both unit and system tests data |
 | merge_pr | Used to make it easier to open a PR that merges one branch into another. |
+
+
+Import / export the dashboards of a single Beat:
+
+| File                  | Description |
+|-----------------------|-------------|
+| import_dashboards.sh  | Bash script to import the Beat dashboards from a local directory in Elasticsearch |
+| import_dashboards.ps1 | Powershell script to import the Beat dashboards from a local directory in Elasticsearch |
+| export_dashboards.py  | Python script to export the Beat dashboards from Elasticsearch to a local directory|
+
+

libbeat/scripts/kibana_export.py → dev-tools/export_dashboards.py

@@ -58,8 +58,6 @@ def ExportVisualization(es, visualization, kibana_index, output_directory):
             search,
             kibana_index,
             output_directory)
-    else:
-        print("Missing savedSearchId from {}".format(doc["_source"]))
 
 
 def ExportSearch(es, search, kibana_index, output_directory):
@@ -115,8 +113,8 @@ def main():
                         help="Elasticsearch index for the Kibana dashboards. "
                         "E.g. .kibana",
                         default=".kibana")
-    parser.add_argument("--dir", help="Output directory",
-                        default="saved")
+    parser.add_argument("--dir", help="Output directory. E.g. output",
+                        default="output")
 
     args = parser.parse_args()
 

libbeat/scripts/kibana_import.ps1 → dev-tools/import_dashboards.ps1

@@ -5,13 +5,14 @@ param(
   [String] $d, [String] $dir,
   [switch] $h = $false, [switch] $help = $false
 )
+function prompt { "$pwd\" }
 
 # The default value of the variable. Initialize your own variables here
 $ELASTICSEARCH="http://localhost:9200"
 $CURL="Invoke-RestMethod"
 $KIBANA_INDEX=".kibana"
 $SCRIPT=$MyInvocation.MyCommand.Name
-$KIBANA_DIR=
+$KIBANA_DIR=prompt
 
 # Verify that Invoke-RestMethod is present. It was added in PS 3.
 if (!(Get-Command $CURL -errorAction SilentlyContinue))
@@ -33,6 +34,7 @@ Options:
     Print the help menu.
   -d | -dir
     Local directory where the dashboards, visualizations, searches and index pattern are saved.
+    By default is $KIBANA_DIR.
   -l | -url
     Elasticseacrh URL. By default is $ELASTICSEARCH.
   -u | -user

libbeat/scripts/kibana_import.sh → dev-tools/import_dashboards.sh

@@ -10,13 +10,16 @@
 ELASTICSEARCH=http://localhost:9200
 CURL=curl
 KIBANA_INDEX=".kibana"
-DIR=
+DIR=.
 
 print_usage() {
   echo "
   
-Load the dashboards, visualizations and index patterns into the given
-Elasticsearch instance.
+Import the dashboards, visualizations and index patterns into Kibana. 
+
+The Kibana dashboards together with its dependencies are saved into a
+special index pattern in Elasticsearch (by default .kibana), so you need to 
+specify the Elasticsearch URL and optionally an username and password.
 
 Usage:
   $(basename "$0") -url ${ELASTICSEARCH} -user admin:secret -index ${KIBANA_INDEX}
@@ -26,6 +29,7 @@ Options:
     Print the help menu.
   -d | -dir
     Local directory where the dashboards, visualizations, searches and index pattern are saved.
+    By default is current directory.
   -l | -url
     Elasticseacrh URL. By default is ${ELASTICSEARCH}.
   -u | -user

libbeat/docs/index.asciidoc

@@ -28,4 +28,6 @@ include::./dashboards.asciidoc[]
 
 include::./newbeat.asciidoc[]
 
+include::./newdashboards.asciidoc[]
+
 include::./release.asciidoc[]

libbeat/docs/newdashboards.asciidoc

@@ -0,0 +1,154 @@
+[[new-dashboards]]
+== Developer Guide: Creating new Kibana dashboards
+
+This guide walks you through the steps for creating a new Kibana dashboards
+or changing the existing Kibana dashboards for a single Beat.
+
+If the Beat you are targeting has already few dashboards, the first step would be to import
+those dashboards to Kibana and then start changing or adding a dashboard from the existing ones. 
+
+Kibana saves the dashboards together with all the dependencies (visualizations, searches and
+index patterns) in a special index in Elasticsearch. By default the index is `.kibana`, but it can be changed to anything.
+
+After you have created or changed a dashboard in Kibana for a certain Beat, you can export it together with all dependencies to
+your local directory.
+
+We recommend you use the virtual environment under
+https://github.com/elastic/beats/tree/master/testing/environments[beats/testing/environments] with the latest version of
+Kibana and Elasticsearch to import, create and export the Kibana dashboards, so the latest dashboards are exported from 
+the same Kibana version.
+
+=== Import existing Beat dashboards
+
+For Unix systems, you can use the bash script `import_dashboards.sh` 
+and for Windows you can use the powershell script `import_dashboards.ps1` from
+https://github.com/elastic/beats/tree/master/dev-tools[dev-tools].
+
+The command has the following options:
+
+[source,shell]
+----------------------------------------------------------------------
+$ ./import_dashboards.sh -h
+
+
+Import the dashboards, visualizations and index patterns into Kibana.
+
+The Kibana dashboards together with its dependencies are saved into a
+special index pattern in Elasticsearch (by default .kibana), so you need to
+specify the Elasticsearch URL and optionally an username and password.
+
+Usage:
+  import_dashboards.sh -url http://localhost:9200 -user admin:secret -index .kibana
+
+Options:
+  -h | -help
+    Print the help menu.
+  -d | -dir
+    Local directory where the dashboards, visualizations, searches and index 
+    pattern are saved.
+  -l | -url
+    Elasticseacrh URL. By default is http://localhost:9200.
+  -u | -user
+    Username and password for authenticating to Elasticsearch using Basic
+    Authentication. The username and password should be separated by a
+    colon (i.e. admin:secret). By default no username and password are
+    used.
+  -i | -index
+    Kibana index pattern where to save the dashboards, visualizations,
+    index patterns. By default is .kibana.
+
+----------------------------------------------------------------------
+
+==== dir
+The input directory with the dashboards together with its dependencies. The default value is current directory.
+
+==== url
+The Elasticsearch URL. The default value is http://localhost:9200.
+
+==== user
+The username and password for authenticating the connection to Elasticsearch using Basic Authentication. The username and password should be separated by a colon. By default no username and password are used.
+
+==== kibana
+The Elasticsearch index pattern where Kibana saved its configuration. The default value is `.kibana`.
+
+
+To import all the dashboards together with  all the dependencies (visualizations, searches and index patterns), you just
+need to run the following command in the beats repository:
+
+On Unix systems:
+
+[source,shell]
+----------------------------------------------------------------------
+../dev-tools/import_dashboards.sh -dir etc/kibana
+----------------------------------------------------------------------
+
+On Windows systems:
+
+[source,shell]
+----------------------------------------------------------------------
+..\dev-tools\import_dashboards.ps1 -dir .\etc\kibana
+----------------------------------------------------------------------
+
+
+=== Export the Beat dashboards
+
+To export all the dashboards for a Beat together with all dependencies (visualizations, searches and index patterns), 
+you can use the python script `export_dashboards.py` from 
+https://github.com/elastic/beats/tree/master/dev-tools[dev-tools].  
+
+The command has the following options:
+
+[source,shell]
+----------------------------------------------------------------------
+$ python export_dashboards.py -h
+usage: export_dashboards.py [-h] [--url URL] --beat BEAT [--index INDEX]
+                            [--kibana KIBANA] [--dir DIR]
+
+Export the Kibana dashboards together with all used visualizations, searches
+and index pattern
+
+optional arguments:
+  -h, --help       Show this help message and exit
+  --url URL        Elasticsearch URL. E.g. http://localhost:9200
+  --beat BEAT      Beat name e.g. topbeat
+  --index INDEX    Elasticsearch index for the Beat data. E.g. topbeat-*
+  --kibana KIBANA  Elasticsearch index for the Kibana dashboards. E.g. .kibana
+  --dir DIR        Output directory. E.g. output
+
+----------------------------------------------------------------------
+
+==== url
+The Elasticsearch URL. The default value is http://localhost:9200.
+
+==== beat
+The name of the Beat. This argument is required.
+
+==== index
+The Elasticsearch index pattern where the Beat is storing the data. The default value is constructed from the Beat name + `-*` string.
+
+==== kibana
+The Elasticsearch index pattern where Kibana saved its configuration. The default value is `.kibana`.
+
+==== dir
+The output directory where to save the dashboards together with its dependencies. The default value is `output`.
+
+For example to export all Packetbeat dashboards, you can run the following command in the `packetbeat` repository:
+
+[source,shell]
+----------------------------------------------------------------------
+../dev-tools/export_dashboards.py --beat packetbeat
+----------------------------------------------------------------------
+
+NOTE:: We have made it even easier for you to import and export the dashboards
+by running the following commands in the Beats repository:
+
+[source,shell]
+----------------------------------------------------------------------
+make import-dashboards
+----------------------------------------------------------------------
+
+[source,shell]
+----------------------------------------------------------------------
+make export-dashboards
+----------------------------------------------------------------------
+

libbeat/scripts/Makefile

@@ -207,16 +207,15 @@ update:
 	bash ${ES_BEATS}/libbeat/scripts/update.sh ${BEATNAME} ${BEAT_DIR}/${BEATNAME} ${ES_BEATS}/libbeat
 
 ### KIBANA FILES HANDLING ###
+ES_URL?=http://localhost:9200/
 
-KIBANA_ES?=http://localhost:9200/
 .PHONY: update
-kibana-export:
-	python ${ES_BEATS}/libbeat/scripts/kibana_export.py --url ${KIBANA_ES} --dir $(shell pwd)/etc/kibana --beat ${BEATNAME}
+export-dashboards:
+	python ${ES_BEATS}/dev-tools//export_dashboards.py --url ${ES_URL} --dir $(shell pwd)/etc/kibana --beat ${BEATNAME}
 
 .PHONY: update
-kibana-import:
-	python ${ES_BEATS}/libbeat/scripts/kibana_import.py --url ${KIBANA_ES} --dir $(shell pwd)/etc/kibana
-
+import-dashboards:
+	bash ${ES_BEATS}/dev-tools/import_dashboards.sh -url ${ES_URL} -dir $(shell pwd)/etc/kibana
 
 ### CONTAINER ENVIRONMENT ####