Allow reading config context data from a file on disk

[jeremystretch]

NetBox version

v3.2.0

Feature type

New functionality

Proposed functionality

Currently, when creating a config context, the user must JSON populate data on the object, which gets stored in the database. This issues proposes providing the option of instead specifying the path to a file on disk (within one of the configured directories) from which data can be read. This will take the place of locally-stored data; a config context instance can define local data or a source file, but not both.

Data from a file will be read at initialization of the config context instance and retained until its deletion. Some degree of caching may also be supported, but further research is needed to determine its impact and feasibility.

Implementation will likely involve the addition of a new configuration parameter to define the permitted file paths.

Use case

This approach allows users to reference external data where necessary e.g. to better accommodate a change control process, while still empowering users to store data locally where sufficient.

Database changes

An optional data_file FilePath field will be added to the ConfigContext model, and the existing data field will become optional. Model validation will ensure that one of these two fields has been populated on save. Additionally, model validation will handle validating the source data upon save.

External dependencies

No response

[jeremystretch]

Related: #8505

[jasonyates]

For a true CI/CD environment, what about having the ability to specify a Github repository & resulting file?

One of my hesitations to using config contexts in automation against my devices is there's no peer review process. An integration with Github would allow us to store the config contexts there and require a PR etc. If Netbox then had a supportable API, you could configure a webhook on the GH repository to trigger Netbox to pull the latest versions on a successful merge.

[jeremystretch]

For a true CI/CD environment, what about having the ability to specify a Github repository & resulting file?

You can certainly do that, although the revision tracking function (e.g. git) can operate outside of NetBox. (Given the support for remote storage, it can even happen on some other system entirely.) You would just define in NetBox the path to a file that happens to live in a git repo.

[jcralbino]

I would like to see this type of git remote file. The issue I see here is related to how the change logging inside netbox would appear when changes in the config context data are made outside netbox

I would like to have a better view of what changed in the configuration context information also within the change log of netbox

[jeremystretch]

I would like to have a better view of what changed in the configuration context information also within the change log of netbox

This would be tracked by e.g. git, not NetBox. NetBox would track only changes to the ConfigContext object itself (such as if the file path is modified). This ensures a clear delineation of revision control.

[ryanmerolle]

Yea I think this makes sense. Have got track the file and have netbox be able to reference git.

[jcralbino]

For me that makes sense to have this separation for revision control well defined and I not against it

I would like to have only a easy way to show the information that git has for that object in the user interface.
Maybe we could store that information cached locally and display it in the user interface when we are using git remote files ?

[jeremystretch]

Currently we are aggregating config context data as part of the database query, which is not possible when dealing with data from files. We have two options for implementing this:

1. Gather the JSON data from applicable config contexts as part of the query, but render it (along with the data from files) in-memory within the view.
2. Dictate that all database contexts take precedence over all file-based contexts, or vice versa.
3. Synchronize the file data to the database via some yet to be determined means.

Personally I strongly prefer the first option, however I'm concerned about performance. The second option, a hybrid approach, would work but assumes that an organization uses predominantly one type or the other. Attempting to keep the database data in sync with the source files is likely to be extremely unreliable (as well as inefficient).

[ryanmerolle]

After talking about this a little, it feels like the feature needs to be fleshed out a little.

A couple of items I am thinking about off the bat:

1. How do users get the file to NetBox? Are we expecting them to have access to the server or container that netbox is running on to then copy the file or git clone a remote repo? That seems shortsighted because most users, besides the deploying admin, will not have access to said file system. This is especially true for the SAAS version of NetBox. - I think a feature may have to come before this to allow git repos or http hosted files via the ui. This could be then used for config_context, export_templates, and other extensibility features
2. When/how will files be checked for updates and loaded as config_context?

As more items come to mind about gaps OR how to approach this, I will comment. I just rather get the discussion going before going to far down a thought on my own.

Since nautobot constantly lifts ideas from NetBox, I am listing some references form their implementation:

• https://github.com/nautobot/nautobot/tree/develop/nautobot/extras/datasources

[jeremystretch]

I think I like the idea of allowing a user to specify a generic "source" URL for a config context. This would point to either a local file (e.g. file://foo/bar/baz.json) or a remote resource. When and how the content of that file gets updated is beyond the scope of NetBox's control. I imagine the most common scenario would see the source file managed by some revision control system (e.g. git).

Regarding the aggregation of data (see my comment above), the most efficient approach is likely to be copying the source data into the database, where it can be aggregated just like "native" config contexts. There are various mechanisms by which we can effect this replication: a UI button, a REST API endpoint, a scheduled task, etc. Ultimately it will be up to the user to determine when this data is replicated, but we should ensures options are available for both automatic and manual synchronizations.

[jeremystretch]

Given that there hasn't been a ton of interest in this proposal, I'm going to shelve it for now. We may want to spend more time considering how we might use this same pattern elsewhere in NetBox and plan accordingly.

[ryanmerolle]

This would be useful for more than just config context.

I can envision some scripts, reports, plugins that I would build to either read from the latest git commit or push commits to git.

One example would be for configuration compliance. I would connect to the following git repos:

• Daily config backups
• YAML based config_context
• Roles based jinja templates defining intended config

[jeremystretch]

Marking this as blocked by #11558, which seeks to implement support for remote data replication in a more abstract sense.

[jeremystretch]

Happy to report that #11558 has been completed and work on this feature can now move forward.