Object change logging

[jeremystretch]

Issue type

[x] Feature request
[ ] Bug report
[ ] Documentation

Environment

• Python version: 3.5.2
• NetBox version: 2.2.9

Description

This issue seeks to unify and supersede the following open issues addressing different shortcomings of the current user activity logging function:

• Device history (user activity) / searchable history #683 - Device history
• Convert UserAction logging to use signals #1127 - Logging of actions should be triggered via signals rather than explicitly within each view
• application logs missing failed actions #1834 - User activity is not exported via standard logging facilities
• Footprint retention post a user is gone #1896 - Object history is lost when the associated user account is deleted

The new system should address all of the above points, as well as incorporate a more robust representation of object history, perhaps using JSON fields to record before and after versions of the object being modified.

[troxil]

Hey @jeremystretch will this enable API calls to be logged? CRUD via the API.

[jeremystretch]

@troxil Yes, the goal is to log every change to a model regardless of how it was effected (GUI, API, or shell).

[jeremystretch]

I've started work on this under the 1898-activity-logging branch.

[jeremystretch]

Implemented in #2198. Change logging will be included in the v2.4 beta.

[pm17788]

Stood up an instance (with blank data) using the devel-2.4 branch.
Saw the changelog attached to the device, as well as /extras/changelog. Two questions came up:

1. How do I get my hands on that data?
2. How do I push that data to an external source? (For auditing purposes, change management data really should be logged outside of the thing being audited)

[pm17788]

@jeremystretch:
It does not seem that the ChangeLog record captures everything.
To wit:
Device creation JSON:

{
  "custom_fields": {
    "LastUpdate": "2018-06-27",
    "OS_Version": "Ubuntu 14.04.5 LTS",
    "bios_date": "2011-01-01",
    "bios_version": "Bochs",
    "cpu_dies": 0,
    "cpus": 0,
    "hdd_count": 0,
    "kernel_version": "4.4.0-128-generic",
    "operational_status": 3,
    "ram_capacity": 64429
  },
  "device_role": 52,
  "device_type": 71,
  "name": "testsystem",
  "serial": "",
  "site": 1,
  "status": 1
}

Whereas the ChangeLog record shows:

{
    "asset_tag": null,
    "cluster": null,
    "comments": "",
    "created": "2018-06-27",
    "device_role": 52,
    "device_type": 71,
    "face": null,
    "last_updated": "2018-06-27T17:46:25.599Z",
    "name": "testsystem",
    "platform": null,
    "position": null,
    "primary_ip4": null,
    "primary_ip6": null,
    "rack": null,
    "serial": "",
    "site": 1,
    "status": 1,
    "tenant": null,
    "vc_position": null,
    "vc_priority": null,
    "virtual_chassis": null
}

[jeremystretch]

Custom fields aren't being captured because it's just using Django's built-in serializer for now. I've been toying with reusing the API serializers for this, but the representation of nested objects gets a little tricky. Although the feature branch has been merged into develop-2.4, I'm still working on this.

[lampwins]

I will chime in and say I have been investigating #2137 with relation to Date type custom fields and serialization for webhooks. These are related so we might want to look into a particular serialization method to cover both use cases.

[pm17788]

@jeremystretch: Would you prefer a new issue filed to keep track, or keep it here? What works for your workflow?

[jeremystretch]

Please do not open issues against anything other than a release. The v2.4 beta has not been released yet.