Merge remote-tracking branch 'origin/develop' into scetron-mroutes-an…

…d-igmp

.pylintrc

@@ -763,5 +763,5 @@ min-public-methods=2
 
 # Exceptions that will emit a warning when being caught. Defaults to
 # "BaseException, Exception".
-overgeneral-exceptions=BaseException,
-                       Exception
+overgeneral-exceptions=builtins.BaseException,
+                       builtins.Exception

docs/config_file.md

@@ -43,8 +43,8 @@ coalescer:
 | ---------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |:-------------------------------- | ------------------- |
 | data-directory               | The directory where the poller stores the data collected from the network                                                                                                                                                                  | -                                | yes                 |
 | temp-directory               | Where SuzieQ stores temporary data                                                                                                                                                                                                         | /tmp/.suzieq/                    | no                  |
-| rest.rest_certfile           | certificate for the REST server                                                                                                                                                                                                            | -                                | no                  |
-| rest.rest_keyfile            | keyfile for the REST server                                                                                                                                                                                                                | -                                | no                  |
+| rest.rest-certfile           | certificate for the REST server                                                                                                                                                                                                            | -                                | no                  |
+| rest.rest-keyfile            | keyfile for the REST server                                                                                                                                                                                                                | -                                | no                  |
 | rest.API_KEY                 | API key for the REST server                                                                                                                                                                                                                | -                                | yes (if using rest) |
 | rest.address                 | IP address of the REST server.                                                                                                                                                                                                             | 127.0.0.1                        | no                  |
 | rest.port                    | port of the REST server                                                                                                                                                                                                                    | 80                               | no                  |

docs/gui.md

@@ -15,3 +15,121 @@ Some additional arguments are allowed to set up the GUI:
 
 - `-p --port`: defines in which port the GUI is going to run. By default is `8501`
 - `-c --config`: SuzieQ configuration file to use. Check the [Configuration](config_file.md) page to check the default configuration file location.
+
+## Start a GUI behind an nginx reverse proxy
+
+This section is an step by step guide on how to set up an [nginx](http://nginx.org/en/docs/) reverse proxy for the SuzieQ GUI.
+
+### Requirements
+
+- nginx: [nginx installation guide](http://nginx.org/en/docs/install.html)
+- SuzieQ GUI running
+
+### Configure and start nginx
+
+Move into the nginx config directory
+
+``` shell
+cd /etc/nginx
+```
+
+!!! info
+        Some of the commands below may need `sudo`
+
+Fill the values into the following template and put it in a file under `/etc/nginx/sites-available/suzieq.conf`
+
+```
+server {
+        listen <PROXY_SERVER_PORT> default_server;
+        listen [::]:<PROXY_SERVER_PORT> default_server;
+        server_name <SERVER_NAME>;
+        location /<PROXY_PATH> {
+                proxy_pass <SUZIEQ_GUI_URL>;
+        }
+        # streamlit redirects config
+        location /<PROXY_PATH>/streamlit-components-demo {
+                proxy_pass <SUZIEQ_GUI_URL>/;
+        }
+        location ^~ /<PROXY_PATH>/static {
+                proxy_pass <SUZIEQ_GUI_URL>/static/;
+        }
+        location ^~ /<PROXY_PATH>/healthz {
+                proxy_pass <SUZIEQ_GUI_URL>/healthz;
+        }
+        location ^~ /<PROXY_PATH>/vendor {
+                proxy_pass <SUZIEQ_GUI_URL>/vendor;
+        }
+        location ^~ /<PROXY_PATH>/component {
+                proxy_pass <SUZIEQ_GUI_URL>/component;
+        }
+
+        location /<PROXY_PATH>/stream {
+                proxy_pass <SUZIEQ_GUI_URL>/stream;
+                proxy_http_version 1.1;
+                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+                proxy_set_header Host $host;
+                proxy_set_header Upgrade $http_upgrade;
+                proxy_set_header Connection "upgrade";
+                proxy_read_timeout 86400;
+        }
+}
+```
+
+The example below shows a configuration file shows how to create a reverse proxy
+to serve a SuzieQ GUI running on `http:localhost:8501` to be mapped to `http:localhost:80/suzieq`
+
+```
+server {
+        listen 80 default_server;
+        listen [::]:80 default_server;
+        server_name localhost;
+        location /suzieq {
+                proxy_pass http://127.0.0.1:8501;
+        }
+        # streamlit redirects config
+        location /suzieq/streamlit-components-demo {
+                proxy_pass http://127.0.0.1:8501/;
+        }
+        location ^~ /suzieq/static {
+                proxy_pass http://127.0.0.1:8501/static/;
+        }
+        location ^~ /suzieq/healthz {
+                proxy_pass http://127.0.0.1:8501/healthz;
+        }
+        location ^~ /suzieq/vendor {
+                proxy_pass http://127.0.0.1:8501/vendor;
+        }
+        location ^~ /suzieq/component {
+                proxy_pass http://127.0.0.1:8501/component;
+        }
+
+        location /suzieq/stream {
+                proxy_pass http://127.0.0.1:8501/stream;
+                proxy_http_version 1.1;
+                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+                proxy_set_header Host $host;
+                proxy_set_header Upgrade $http_upgrade;
+                proxy_set_header Connection "upgrade";
+                proxy_read_timeout 86400;
+        }
+}
+```
+
+!!! warning
+        These examples, do **NOT** provide any security.
+        Please check the [nginx documentation](http://nginx.org/en/docs/) to know how add security to your nginx server
+
+
+Now copy the configuration from `/etc/nginx/sites-available` to `/etc/nginx/sites-enabled`.
+It is recommended to use a symbolic link.
+
+``` shell
+ln -s /etc/nginx/sites-available/suzieq.conf /etc/nginx/sites-enabled/suzieq.conf
+```
+
+Now restart the nginx service
+``` shell
+sudo systemctl restart nginx
+```
+
+You can open you browser and connect to your configured nginx to see the SuzieQ GUI

docs/inventory.md

@@ -14,6 +14,7 @@ The new inventory is structured in 4 major pieces, explained in its own section:
 - `namespaces`: where you put together all the above. A namespace is be defined by a `source`, an `auth` and a `device`
 
 Here is an example of an inventory file with a bunch of different options, but non-exhaustive, for each section:
+
 ```yaml
 sources:
 - name: netbox-instance-123
@@ -86,6 +87,7 @@ namespaces:
     - It is possible to [map different sources to the same namespace](#mapping-different-sources-to-the-same-namespace)
 
 ## <a name='sensitive-data'></a>Sensitive data
+
 A sensitive data is an information that the user doesn't want to store in plain-text inside the inventory.
 For this reason, SuzieQ inventory now supports three different options to store these kind of informations
 
@@ -94,6 +96,7 @@ For this reason, SuzieQ inventory now supports three different options to store
 - `ask`: the user can write the sensitive information on the stdin
 
 Currently this method is used to specify passwords, passphrases and tokens.
+
 ## <a name='inventory-sources'></a>Sources
 
 The device sources currently supported are:
@@ -121,9 +124,11 @@ Whenever a source has many fields in common with another, you don't have to rewr
   - suzieq-copy
 
 ```
+
 ### <a name='source-host-list'></a>Host list
 
 The host list contains the IP address, the access method (SSH or REST), the IP address of the node, the user name, the type of OS if using REST and the access token such as a private key file. Here is an example of a native suzieq source type. For example (all possible combinations are shown for illustration):
+
 ```yaml
 - name: dc-01-native
   type: native # optional, if type is not present this is the default value
@@ -149,14 +154,16 @@ ansible-inventory --list > ansible.json
 ```
 
 Now you can set the path of the ansible inventory in the source:
+
 ```yaml
 - name: ansible-01
   type: ansible
   path: /path/to/ansible.json
 ```
 
+Since Ansible devices cannot really be split up, the device and auth sections apply to **all** the devices in the Ansible inventory file. This is a limitaion of the Ansible source input. We always assume ssh as the transport unless otherwise specified in the device section of the SuzieQ inventory file. 
 !!! info
-    The Ansible source assumes REST transport with Arista EOS and PanOs devices by default, and SSH for the others
+    From 0.21.0, with Ansible inventories, the device type and transport are taken from the specification in the device section of the suzieq inventory file. You must specify the transport as rest if you want to use rest as the transport for EOS devices. By default, we assume ssh as the transport. For PANOS also, you must specify the device type and transport. Before version 0.21.0, Ansible inventory assumed REST as the transport for EOS, even if the user specified the transport as SSH in the device section.
 
 ### <a name='source-netbox'></a>Netbox
 
@@ -173,6 +180,7 @@ Since Netbox is a _dynamic source_, the data are periodically pulled, the period
     If the user manually sets `ssl-verify: true` with an http netbox server, an error will be notified.
 
 Here is an example of the configuration of a netbox type source:
+
 ```yaml
 - name: netbox-dc-01
   type: netbox
@@ -183,6 +191,7 @@ Here is an example of the configuration of a netbox type source:
   period: 3600        # How frequently Netbox should be polled
   ssl-verify: false   # Netbox certificate validation will be skipped
 ```
+
 #### Selecting devices from Netbox
 
 Starting from 0.19, it's possible to specify more than one tag to be matched, defining a list of one or more rules.
@@ -198,6 +207,7 @@ A device is polled by SuzieQ if it matches at least one of the defined rules.
   - alpha
   - bravo, charlie
 ```
+
 For example, the source above tells SuzieQ to select from Netbox all the devices having the `alpha` OR `bravo & charlie` tags.
 
 !!!Warning
@@ -210,6 +220,7 @@ Netbox type source is capable to assign each device to a namespace which corresp
 To obtain this behaviour, we need to declare a `namespace` object with `name: netbox-sitename`.
 
 Here is an example:
+
 ```yaml
 sources:
 - name: netbox-dc-01
@@ -270,6 +281,7 @@ In case you want to ignore the check of the device's key against the `known_host
 ```
 
 Moreover if all the devices inside a namespace run the same NOS, it is possible to specify it via the `devtype` option:
+
 ```yaml
 - name: eos-devices
   devtype: eos
@@ -294,13 +306,15 @@ This section is optional in case SuzieQ native and ansible source types. Here a
 Currently for both SSH and REST API, the only supported is username and password, therefore you will not be able to set api keys.
 
 The simplest method is defining either username and password/private key.
+
 ```yaml
 - name: suzieq-user
   username: suzieq
   password: plain:pass
 ```
 
 In case a private key is used to authenticate:
+
 ```yaml
 - name: suzieq-user
   keyfile: path/to/private/key
@@ -326,6 +340,7 @@ A `cred-file` is an external file where you store credentials for all the device
 Each device credentials can be specified via its `hostname` or its `address`
 (with Netbox, it's encouraged the usage of `hostname`).
 The credential file should look like this:
+
 ```yaml
 - namespace: testing
   devices:
@@ -348,6 +363,7 @@ The credential file should look like this:
 
 In the `namespaces` section sources, auths and devices can be put together to define namespaces.
 For example the following namespace will be defined by the source named `netbox-1`, the auths named `dc-01-credentials`, and the device named `ssh-jump-devs`:
+
 ```yaml
 namespaces:
 - name: example
@@ -430,15 +446,14 @@ Suppose we have this inventory valid for version 0.15.x:
     - url: ssh://vagrant@192.168.123.54:2023  keyfile=/home/netenglabs/cloud-native-data-center-networking/topologies/dual-attach/.vagrant/machines/server104/libvirt/private_key
     - url: https://vagrant@192.168.123.123 password=vagrant
 ```
-The new inventory format consists of four sections (sources, auths, devices, namespaces) which are described above. We need to add the devices specified in the old inventory format in a new source inside the `sources` section and link it to a namespace.
 
+The new inventory format consists of four sections (sources, auths, devices, namespaces) which are described above. We need to add the devices specified in the old inventory format in a new source inside the `sources` section and link it to a namespace.
 
 Here is how the new format will look like:
 
 !!! important
     Sections [auths](#auths) and [devices](#devices) are optional. See the full documentation to know how to use them.
 
-
 ```yaml
 sources:
 - name: eos-source # namespace is defined below, this is only a name to be used as reference

docs/poller.md

@@ -20,9 +20,13 @@ The inventory file format is covered in the [inventory page](./inventory.md).
 
 To monitor the status of the poller, you can look at the log files created (by default) in the `/tmp` directory. All the aspects related to the creation/update of the inventory are logged into `sq-poller-controller.log`, while the each worker logs into `sq-poller-<x>.log` where `x` is the worker id.
 
+!!! warning
+    If you want to start the poller process as background task, remember to redirect the stdout to `/dev/null` otherwise the poller might crash when it tries to write something on the terminal.<br>
+    `sq-poller -I inventory.yml >/dev/null &`
+
 ## Polling Modes
 
-SuzieQ poller can run in either **continuous mode** or **snapshot mode**. In continuous mode, the poller runs forever, gathering data at the specified period from the supplied inventory file. Alternately, it can run in snapshot mode where it gathers the data just once and stops once it has gathered data from all the nodes in the inventory file. If we cannot gather data from a node, we do not persist in gathering data in the snapshot mode. 
+SuzieQ poller can run in either **continuous mode** or **snapshot mode**. In continuous mode, the poller runs forever, gathering data at the specified period from the supplied inventory file. Alternately, it can run in snapshot mode where it gathers the data just once and stops once it has gathered data from all the nodes in the inventory file. If we cannot gather data from a node, we do not persist in gathering data in the snapshot mode.
 
 The default mode is the continuous mode. To use the snapshot mode, add the option `--run-once=update`.
 
@@ -92,7 +96,7 @@ have production versions of that code.
 
 ## Polling Period
 
-When polling in continuous mode, SuzieQ uses the default period specified in the suzieq-cfg.yml [configuration file](./config_file.md) (you can change the default location via the `-c` option when launching the poller). 
+When polling in continuous mode, SuzieQ uses the default period specified in the suzieq-cfg.yml [configuration file](./config_file.md) (you can change the default location via the `-c` option when launching the poller).
 
 Independent of this, you can change the polling period of any individual service by modifying (or adding) the `period:<time in secs>` key-value pair to the service configuration file (located under lib/python\<python version\>/site-packages/suzieq/config) wherever SuzieQ is installed.
 
@@ -128,7 +132,7 @@ jpietsch> sqpoller show status=fail namespace=dual-evpn
      354  dual-evpn  server104  ospfNbr       1         []        []       []      []        []                    0 2020-06-17 05:14:40.980
 ```
 
-In this case the errors are because we aren't running any of those services (mlag, evpn etc.) on those nodes (server101, edge01 etc.).  
+In this case the errors are because we aren't running any of those services (mlag, evpn etc.) on those nodes (server101, edge01 etc.).
 
 ## Database and Data Persistence
 

docs/rate-limiting-AAA.md

@@ -20,8 +20,8 @@ rest:
 # The certificates listed below are provided purely to get started, In any
 # secure deployment, these must be generated specifically for the site and
 # these lines uncommented and containing the location of the site-specific file.
-# rest_certfile: /suzieq/cert.pem
-# rest_keyfile: /suzieq/key.pem
+# rest-certfile: /suzieq/cert.pem
+# rest-keyfile: /suzieq/key.pem
 #
   API_KEY: 496157e6e869ef7f3d6ecb24a6f6d847b224ee4f
   logging-level: WARNING