Merge pull request #3271 from digitalocean/develop

Release v2.6.0

.travis.yml

@@ -1,6 +1,7 @@
 sudo: required
 services:
   - postgresql
+  - redis-server
 addons:
   postgresql: "9.4"
 language: python

CHANGELOG.md

@@ -1,3 +1,223 @@
+v2.6.0 (2019-06-20)
+
+## New Features
+
+### Power Panels and Feeds ([#54](https://github.com/digitalocean/netbox/issues/54))
+
+NetBox now supports power circuit modeling via two new models: power panels and power feeds. Power feeds are terminated
+to power panels and are optionally associated with individual racks. Each power feed defines a supply type (AC/DC),
+amperage, voltage, and phase. A power port can be connected directly to a power feed, but a power feed may have only one
+power port connected to it.
+
+Additionally, the power port model, which represents a device's power input, has been extended to include fields
+denoting maximum and allocated draw, in volt-amperes. This allows a device (e.g. a PDU) to calculate its total load
+compared to its connected power feed.
+
+### Caching ([#2647](https://github.com/digitalocean/netbox/issues/2647))
+
+To improve performance, NetBox now supports caching for most object and list views. Caching is implemented using Redis,
+which is now a required dependency. (Previously, Redis was required only if webhooks were enabled.)
+
+A new configuration parameter is available to control the cache timeout:
+
+```
+# Cache timeout (in seconds)
+CACHE_TIMEOUT = 900
+```
+
+### View Permissions ([#323](https://github.com/digitalocean/netbox/issues/323))
+
+Django 2.1 introduced the ability to enforce view-only permissions for different object types. NetBox now enforces
+these by default. You can grant view permission to a user or group by assigning the "can view" permission for the
+desired object(s).
+
+To exempt certain object types from the enforcement of view permissions, so that any user (including anonymous users)
+can view them, add them to the new `EXEMPT_VIEW_PERMISSIONS` setting in `configuration.py`:
+
+```
+EXEMPT_VIEW_PERMISSIONS = [
+    'dcim.site',
+    'ipam.prefix',
+]
+```
+
+To exclude _all_ objects, effectively disabling view permissions and restoring pre-v2.6 behavior, set:
+
+```
+EXEMPT_VIEW_PERMISSIONS = ['*']
+```
+
+### Custom Links ([#969](https://github.com/digitalocean/netbox/issues/969))
+
+Custom links are created under the admin UI and will be displayed on each object of the selected type. Link text and
+URLs can be formed from Jinja2 template code, with the viewed object passed as context data. For example, to link to an
+external NMS from the device view, you might create a custom link with the following URL:
+
+```
+https://nms.example.com/nodes/?name={{ obj.name }}
+```
+
+Custom links appear as buttons at the top of the object view. Grouped links will render as a dropdown menu beneath a
+single button.
+
+### Prometheus Metrics ([#3104](https://github.com/digitalocean/netbox/issues/3104))
+
+NetBox now supports exposing native Prometheus metrics from the application. [Prometheus](https://prometheus.io/) is a
+popular time series metric platform used for monitoring. Metric exposition can be toggled with the `METRICS_ENABLED`
+configuration setting; it is not enabled by default. NetBox exposes metrics at the `/metrics` HTTP endpoint, e.g.
+`https://netbox.local/metrics`.
+
+NetBox makes use of the [django-prometheus](https://github.com/korfuri/django-prometheus) library to export a number of
+different types of metrics, including:
+
+* Per model insert, update, and delete counters
+* Per view request counters
+* Per view request latency histograms
+* Request body size histograms
+* Response body size histograms
+* Response code counters
+* Database connection, execution, and error counters
+* Cache hit, miss, and invalidation counters
+* Django middleware latency histograms
+* Other Django related metadata metrics
+
+For the exhaustive list of exposed metrics, visit the `/metrics` endpoint on your NetBox instance. See the documentation
+for more details on using Prometheus metrics in NetBox.
+
+## Changes
+
+### New Dependency: Redis
+
+[Redis](https://redis.io/) is an in-memory data store similar to memcached. While Redis has been an optional component
+of NetBox since the introduction of webhooks in version 2.4, it is now required to support NetBox's new caching
+functionality (as well as other planned features). Redis can be installed via your platform's package manager: for
+example, `sudo apt-get install redis-server` on Ubuntu or `sudo yum install redis` on CentOS.
+
+The Redis database is configured using a configuration setting similar to `DATABASE` in `configuration.py`:
+
+```
+REDIS = {
+    'HOST': 'localhost',
+    'PORT': 6379,
+    'PASSWORD': '',
+    'DATABASE': 0,
+    'CACHE_DATABASE': 1,
+    'DEFAULT_TIMEOUT': 300,
+    'SSL': False,
+}
+```
+
+Note that if you were using these settings in a prior release with webhooks, the `DATABASE` setting remains the same but
+an additional `CACHE_DATABASE` setting has been added with a default value of 1 to support the caching backend. The
+`DATABASE` setting will be renamed in a future release of NetBox to better relay the meaning of the setting. It is
+highly recommended to keep the webhook and cache databases seperate. Using the same database number for both may result
+in webhook processing data being lost during cache flushing events.
+
+### API Support for Specifying Related Objects by Attributes([#3077](https://github.com/digitalocean/netbox/issues/3077))
+
+Previously, specifying a related object in an API request required knowing the primary key (integer ID) of that object.
+For example, when creating a new device, its rack would be specified as an integer:
+
+```
+{
+    "name": "MyNewDevice",
+    "rack": 123,
+    ...
+}
+```
+
+The NetBox API now also supports referencing related objects by a set of sufficiently unique attrbiutes. For example, a
+rack can be identified by its name and parent site:
+
+```
+{
+    "name": "MyNewDevice",
+    "rack": {
+        "site": {
+            "name": "Equinix DC6"
+        },
+        "name": "R204"
+    },
+    ...
+}
+```
+
+There is no limit to the depth of nested references. Note that if the provided parameters do not return exactly one
+object, a validation error is raised.
+
+### API Device/VM Config Context Included by Default ([#2350](https://github.com/digitalocean/netbox/issues/2350))
+
+The rendered config context for devices and VMs is now included by default in all API results (list and detail views).
+Previously, the rendered config context was available only in the detail view for individual objects. Users with large
+amounts of context data may observe a performance drop when returning multiple objects. To combat this, in cases where
+the rendered config context is not needed, the query parameter `?exclude=config_context` may be appended to the request
+URL to exclude the config context data from the API response.
+
+### Changes to Tag Permissions
+
+NetBox now makes use of its own `Tag` model instead of the stock model which ships with django-taggit. This new model
+lives in the `extras` app and thus any permissions that you may have configured using "Taggit | Tag" should be changed
+to now use "Extras | Tag." Also note that the admin interface for tags has been removed as it was redundant to the
+functionality provided by the front end UI.
+
+### CORS_ORIGIN_WHITELIST Requires URI Scheme
+
+If you have the `CORS_ORIGIN_WHITELIST` configuration parameter defined, note that each origin must now incldue a URI
+scheme. This change was introuced in django-cors-headers 3.0.
+
+## Enhancements
+
+* [#166](https://github.com/digitalocean/netbox/issues/166) - Add `dns_name` field to IPAddress
+* [#524](https://github.com/digitalocean/netbox/issues/524) - Added power utilization graphs to power feeds, devices, and racks
+* [#1792](https://github.com/digitalocean/netbox/issues/1792) - Add CustomFieldChoices API endpoint at `/api/extras/_custom_field_choices/`
+* [#1863](https://github.com/digitalocean/netbox/issues/1863) - Add child object counts to API representation of organizational objects
+* [#2324](https://github.com/digitalocean/netbox/issues/2324) - Add `color` field for tags
+* [#2643](https://github.com/digitalocean/netbox/issues/2643) - Add `description` field to console/power components and device bays
+* [#2791](https://github.com/digitalocean/netbox/issues/2791) - Add `comments` field for tags
+* [#2920](https://github.com/digitalocean/netbox/issues/2920) - Rename Interface `form_factor` to `type` (backward-compatible until v2.7)
+* [#2926](https://github.com/digitalocean/netbox/issues/2926) - Add change logging to the Tag model
+* [#3038](https://github.com/digitalocean/netbox/issues/3038) - OR logic now used when multiple values of a query filter are passed
+* [#3264](https://github.com/digitalocean/netbox/issues/3264) - Annotate changelog retention time on UI
+
+## Bug Fixes
+
+* [#2968](https://github.com/digitalocean/netbox/issues/2968) - Correct API documentation for SerializerMethodFields
+* [#3176](https://github.com/digitalocean/netbox/issues/3176) - Add cable trace button for console server ports and power outlets
+* [#3231](https://github.com/digitalocean/netbox/issues/3231) - Fixed cosmetic error indicating a missing schema migration
+* [#3239](https://github.com/digitalocean/netbox/issues/3239) - Corrected count of tags reported via API
+
+## Bug Fixes From v2.6-beta1
+
+* [#3123](https://github.com/digitalocean/netbox/issues/3123) - Exempt `/metrics` view from authentication
+* [#3125](https://github.com/digitalocean/netbox/issues/3125) - Fix exception when viewing PDUs
+* [#3126](https://github.com/digitalocean/netbox/issues/3126) - Incorrect calculation of PowerFeed available power
+* [#3130](https://github.com/digitalocean/netbox/issues/3130) - Fix exception when creating a new power outlet
+* [#3136](https://github.com/digitalocean/netbox/issues/3136) - Add power draw fields to power port creation form
+* [#3137](https://github.com/digitalocean/netbox/issues/3137) - Add `power_port` and `feed_leg` fields to power outlet creation form
+* [#3140](https://github.com/digitalocean/netbox/issues/3140) - Add bulk edit capability for power outlets and console server ports
+* [#3204](https://github.com/digitalocean/netbox/issues/3204) - Fix interface filtering when connecting cables
+* [#3207](https://github.com/digitalocean/netbox/issues/3207) - Fix link for connecting interface to rear port
+* [#3258](https://github.com/digitalocean/netbox/issues/3258) - Exception raised when creating/viewing a circuit with a non-connected termination
+
+## API Changes
+
+* New API endpoints for power modeling: `/api/dcim/power-panels/` and `/api/dcim/power-feeds/`
+* New API endpoint for custom field choices: `/api/extras/_custom_field_choices/`
+* ForeignKey fields now accept either the related object PK or a dictionary of attributes describing the related object.
+* Organizational objects now include child object counts. For example, the Role serializer includes `prefix_count` and `vlan_count`.
+* The `id__in` filter is now deprecated and will be removed in v2.7. (Begin using the `?id=1&id=2` format instead.)
+* Added a `description` field for all device components.
+* dcim.Device: The devices list endpoint now includes rendered context data.
+* dcim.DeviceType: `instance_count` has been renamed to `device_count`.
+* dcim.Interface: `form_factor` has been renamed to `type`. Backward compatibility for `form_factor` will be maintained until NetBox v2.7.
+* dcim.Interface: The `type` filter has been renamed to `kind`.
+* dcim.Site: The `count_*` read-only fields have been renamed to `*_count` for consistency with other objects.
+* dcim.Site: Added the `virtualmachine_count` read-only field.
+* extras.Tag: Added `color` and `comments` fields to the Tag serializer.
+* virtualization.VirtualMachine: The virtual machines list endpoint now includes rendered context data.
+
+---
+
 2.5.13 (2019-05-31)
 
 ## Enhancements

base_requirements.txt

@@ -2,6 +2,10 @@
 # https://github.com/django/django
 Django
 
+# Django caching using Redis
+# https://github.com/Suor/django-cacheops
+django-cacheops
+
 # Django middleware which permits cross-domain API requests
 # https://github.com/OttoYiu/django-cors-headers
 django-cors-headers
@@ -18,6 +22,14 @@ django-filter
 # https://github.com/django-mptt/django-mptt
 django-mptt
 
+# Django integration for RQ (Reqis queuing)
+# https://github.com/rq/django-rq
+django-rq
+
+# Prometheus metrics library for Django
+# https://github.com/korfuri/django-prometheus
+django-prometheus
+
 # Abstraction models for rendering and paginating HTML tables
 # https://github.com/jieter/django-tables2
 django-tables2

docs/additional-features/caching.md

@@ -0,0 +1,21 @@
+# Caching
+
+To improve performance, NetBox supports caching for most object and list views. Caching is implemented using Redis,
+and [django-cacheops](https://github.com/Suor/django-cacheops)
+
+Several management commands are avaliable for administrators to manaully invalidate cache entries in extenuating circumstances.
+
+To invalidate a specifc model instance (for example a Device with ID 34):
+```
+python netbox/manage.py invalidate dcim.Device.34
+```
+
+To invalidate all instance of a model:
+```
+python netbox/manage.py invalidate dcim.Device
+```
+
+To flush the entire cache database:
+```
+python netbox/manage.py invalidate all
+```

docs/additional-features/prometheus-metrics.md

@@ -0,0 +1,34 @@
+# Prometheus Metrics
+
+NetBox supports optionally exposing native Prometheus metrics from the application. [Prometheus](https://prometheus.io/) is a popular time series metric platform used for monitoring.
+
+NetBox exposes metrics at the `/metrics` HTTP endpoint, e.g. `https://netbox.local/metrics`. Metric exposition can be toggled with the `METRICS_ENABLED` configuration setting. Metrics are not exposed by default.
+
+## Metric Types
+
+NetBox makes use of the [django-prometheus](https://github.com/korfuri/django-prometheus) library to export a number of different types of metrics, including:
+
+- Per model insert, update, and delete counters
+- Per view request counters
+- Per view request latency histograms
+- Request body size histograms
+- Response body size histograms
+- Response code counters
+- Database connection, execution, and error counters
+- Cache hit, miss, and invalidation counters
+- Django middleware latency histograms
+- Other Django related metadata metrics
+
+For the exhaustive list of exposed metrics, visit the `/metrics` endpoint on your NetBox instance.
+
+## Multi Processing Notes
+
+When deploying NetBox in a multiprocess mannor--such as using Gunicorn as recomented in the installation docs--the Prometheus client library requires the use of a shared directory
+to collect metrics from all the worker processes. This can be any arbitrary directory to which the processes have read/write access. This directory is then made available by use of the
+`prometheus_multiproc_dir` environment variable.
+
+This can be setup by first creating a shared directory and then adding this line (with the appropriate directory) to the `[program:netbox]` section of the supervisor config file.
+
+```
+environment=prometheus_multiproc_dir=/tmp/prometheus_metrics
+```

docs/additional-features/webhooks.md

@@ -4,14 +4,6 @@ A webhook defines an HTTP request that is sent to an external application when c
 
 An optional secret key can be configured for each webhook. This will append a `X-Hook-Signature` header to the request, consisting of a HMAC (SHA-512) hex digest of the request body using the secret as the key. This digest can be used by the receiver to authenticate the request's content.
 
-## Installation
-
-If you are upgrading from a previous version of Netbox and want to enable the webhook feature, please follow the directions listed in the sections below.
-
-* [Install Redis server and djano-rq package](../installation/2-netbox/#install-python-packages)
-* [Modify configuration to enable webhooks](../installation/2-netbox/#webhooks-configuration)
-* [Create supervisord program to run the rqworker process](../installation/3-http-daemon/#supervisord-installation)
-
 ## Requests
 
 The webhook POST request is structured as so (assuming `application/json` as the Content-Type):