Network ssot integration

docs/dev/extending.md

@@ -7,3 +7,16 @@ Extending the application is welcome, however it is best to open an issue first,
 This plugin provides methods to customize onboarding behavior. By creating onboarding extensions, it is possible to onboard switch stacks, HA pairs and perform other customizations.
 
 Please see the dedicated FAQ for [device onboarding extensions](onboarding_extensions.md).
+
+!!! warn
+    This is the legacy onboarding extensions.  The next section covers how to extend the new framework.
+
+## Extending SSoT jobs (Sync Devices, and Sync Network Data)
+
+Extending the platform support for the SSoT specific jobs should be accomplished with adding a yaml file that defines commands, jdiff jmespaths, and post_processors. A PR into this library is welcomed, but this app exposes the Nautobot core datasource capabilities to be able to load in overrides from a Git repository.
+
+### Adding Platform/OS Support
+
+New platform support should be simplified in this framework, by providing a YAML file.
+
+The format of these YAML files are and how to extend this application is covered in [App YAML Overrides](../user/app_yaml_overrides.md).

docs/dev/onboarding_extensions.md

@@ -1,5 +1,8 @@
 # Onboarding Extensions
 
+!!! warn
+    This is the legacy onboarding extensions.
+
 ## What are onboarding extensions?
 
 Onboarding Extensions are Python modules that are dynamically loaded and executed as a part of device onboarding.

docs/user/app_detailed_design.md

@@ -0,0 +1,54 @@
+# Network SSoT Detailed Design
+
+This page will describe the newer SSoT jobs that this App exposes and how they work.
+
+## Frameworks in Use
+
+- [Nautobot SSoT](https://docs.nautobot.com/projects/ssot/en/latest/) - Utilzing the existing Nautobot SSoT framework allows a common pattern to be re-used and offers a path forward to add additional support and features.
+- [Nautobot App Nornir](https://docs.nautobot.com/projects/plugin-nornir/en/latest/) - Utilized for Nornir Inventory plugins for Nautobot (specifically for Sync Network Data Job).
+- [Nornir Netmiko](https://github.com/ktbyers/nornir_netmiko) - Used to execute commands and return results.
+- [Jdiff](https://jdiff.readthedocs.io/en/latest/usage/#extract_data_from_json) - Used to simplify parsing required data fields out of command outputs returned from command parser libraries like textFSM. Specifically `extract_data_from_json` method.
+- Parsers - Initially NTC Templates via textFSM, but future support for PyATS, TTP, etc. is expected in the future.
+
+## How the SSoT **Sync Devices** Job Works
+
+1. The job is executed with inputs selected.
+    - List of comma seperated IP/DNS names is provided.
+    - Other required fields are selected in the job inputs form.
+
+2. The SSoT framework loads the Nautobot adapter information.
+3. The SSoT frameworks network adapater `load()` method calls nornir functionality.
+    - The job inputs data is passed to the InitNornir initializer, because we only have basic information a custom `EmptyInventory` Nornir inventory plugin is packaged with the App. This get initialized in the `InitNornir` function, but actually initialzes a true inventory that is empty.
+    - Since `Platform` information may need to be auto-detected before adding a Nornir `Host` object to the inventory, a `create_inventory` function is executed that uses the SSH-Autodetect via Netmiko to try to determine the platform so it can be injected into the `Host` object.
+    - Finally, all the platform specific commands to run, along with all the jpath, `post_processor` information loaded from the platform specific YAML files must be injected into the Nornir data object to be accessible later in the extract, transform functions.
+4. Within the context of a Nornir `with_processor` context manager call the `netmiko_send_commands` Nornir task.
+    - Access the loaded platform specific YAML data and deduplicate commands to avoid running the same command multiple times. E.g. Multiple required data attributes come from the same show command.
+5. Utilize native Nornir Processor to overload functionality on `subtask_instance_completed()` to run command outputs through extract and transformation functions.
+    - This essentially is our "ET" portion of a "ETL" process.
+    - Next, the JSON result from the show command after the parser executes (E.g. textfsm), gets run through the jdiff function `extract_data_from_json()` with the data and the `jpath` from the YAML file definition.
+    - Finally, an optional `post_processor` jinja2 capable execution can further transform the data for that command before passing it to finish the SSoT syncronizaton.
+
+## How the SSoT **Sync Network Data** Job Works
+
+1. The job is executed with inputs selected.
+    - One or multiple device selection.
+    - Other required fields are selected in the job inputs form.
+    - Toggle certain metadata booleans to True if you want more data synced.
+
+2. The SSoT framework loads the Nautobot adapter information.
+3. The SSoT frameworks network adapater `load()` method calls Nornir functionality.
+    - The job inputs data is passed to the InitNornir initializer, because devices now exist in Nautobot we use `NautobotORMInventory` Nornir inventory plugin comes from `nautobot-plugin-nornir`.
+    - Finally, all the platform specific `commands` to run, along with all the `jpath`, `post_processor` information loaded from the platform specific YAML files must be injected into the Nornir data object to be accessible later in the extract, transform functions.
+4. Within the context of a Nornir `with_processor` context manager call the `netmiko_send_commands` Nornir task.
+    - Access the loaded platform specific YAML data and deduplicate commands to avoid running the same command multiple times. E.g. Multiple required data attributes come from the same show command.
+5. Utilize native Nornir Processor to overload functionality on `subtask_instance_completed()` to run command outputs through extract and transformation functions.
+    - This essentially is our "ET" portion of a "ETL" process.
+    - Next, the JSON result from the show command after the parser executes (E.g. textfsm), gets run through the jdiff function `extract_data_from_json()` with the data and the `jpath` from the YAML file definition.
+    - Finally, an optional `post_processor` jinja2 capable execution can further transform the data for that command before passing it to finish the SSoT syncronizaton.
+
+## Detailed Design Diagram
+
+Here are two diagrams detailing the SSoT based jobs in deeper detail.
+
+![Sync Devices]().
+![Sync Network Data]().

docs/user/app_getting_started.md

@@ -8,6 +8,8 @@ To install the App, please follow the instructions detailed in the [Installation
 
 ## First steps with the App
 
+This App exposes a legacy device onboarding job, as well as two new SSoT based jobs that are considered the future of the App.
+
 ### Prerequisites
 
 You will need:
@@ -21,9 +23,16 @@ The device must be reachable from the Nautobot and Nautobot worker instances (us
 
 Locations are the only other Nautobot prerequisite for the plugin to onboard a device.
 
+!!! info
+    There are a few other requirements for the new SSoT based jobs, but can also support some defaults.
+
 ### Onboarding a Device
 
-Navigate to the Device Onboarding Job: Jobs > Perform Device Onboarding.
+Navigate to the Device Onboarding Job: Jobs > Perform Device Onboarding (Legacy).
+
+or
+
+Navigate to the SSoT dashboard and run `Sync Devices` to get basic device and information onboarding, followed by `Sync Network Data` to add additonal details from the network to these devices. E.g. Interfaces, IPs, VRFs, VLANs.
 
 ## What are the next steps?
 

docs/user/app_overview.md

@@ -9,6 +9,11 @@ This [Nautobot](https://github.com/nautobot/nautobot) App allows to easily onboa
 
 ## Description/Overview
 
+### Legacy Implementation
+
+!!! info
+    The legacy job and extensions pattern will remain a part of this App for the near future, this will allow custom extensions to continue working without causes issues to users that have taken the time and understand the legacy framework.  The newer SSoT implementation will be discussed in the next section.
+
 The `nautobot-device-onboarding` app uses the [netmiko](https://github.com/ktbyers/netmiko) and [NAPALM](https://napalm.readthedocs.io/en/latest/) libraries to simplify the onboarding process of a new device into Nautobot down to, in many cases, an *IP Address* and a *Location*. In some cases, the user may also have to specify a specific *Device Platform* and *Device Port*.
 
 Regardless, the Onboarding App greatly simplifies the onboarding process by allowing the user to specify a small amount of info and having the app populate a much larger amount of device data in Nautobot.
@@ -38,6 +43,53 @@ For example, getting the Management IP and Platform data into Nautobot allows a
 
 One example of a solution that can retrieve that additional device data and import it into Nautobot is the [Network Importer](https://github.com/networktocode/network-importer). Other options would include an Ansible playbook or a Python script.
 
+### New SSoT Implementation
+
+The new implementation of device onboarding in this app is utilizing the SSoT framework; the main reasons for providing the new mechanisms were to solve the following challenges:
+
+- Make it easier to extending and add additonal vendor/OS support.
+- Collapse this app and the external [Network Importer](https://github.com/networktocode/network-importer) into the same Nautobot app for simplified device onboarding with more object support.'
+    - Remove the Batfish dependency.
+- Re-use backend plugins and libraries such as `nautobot-app-nornir` to provide the a similar feeling to other plugins like `nautobot-app-golden-config`.
+- Utilize SSoT framework and the new `contrib` functionality to speed up development of new features.
+- By collapsing 
+
+Expose two new SSoT based Nautobot jobs to perform the syncing of data.
+
+1. `Sync Device SSoT Job` - Takes mininum inputs nearly identical to the legacy job (IP, Locaiton, SecretGroup), and create a device with bare minium information to be able to manage a device. This job syncs data from the network itself and creates a device with the follow attributes.
+    - Hostname
+    - Serial Number
+    - Device Type
+    - Platform
+    - Management Interface
+    - Management IP address (creates a prefix if one doesn't exist for the IP discovered.)
+
+2. `Sync Network Data SSoT Job` - From a provided list of existing Nautobot device objects, sync in additional metadata from a network device to enhance the available data from the network in Nautobot.
+    - All interfaces on the device with plus the attributes below:
+        - Interface Name
+        - MTU
+        - Description
+        - Interface type (limited support. Default: 'Other')
+        - Mac Address
+        - Link Status
+        - Interface Mode
+    - VLANs
+        - Vlans
+        - Untagged and Tagged
+    - VRFs
+        - VRF Names
+        - Route Distinguishers (RD)
+    - Cabling (Coming soon...)
+
+!!! info
+    For more information look at the provided jsonschema definitions for each of the jobs.
+
+Additional References:
+
+- For more information see [App Use Cases](./app_use_cases.md).
+- To understand the lower level details of how the Network-SSoT framework is designed see [Network-SSoT Design](./app_detailed_design.md)
+- To learn how to add additonal platform/OS support visit [Extending](./external_interactions.md).
+
 ## Audience (User Personas) - Who should use this App?
 
 The Onboarding App is meant for new Nautobot users who want to start importing their devices directly rather than from another, existing, source. Even with other sources for device information, they may not include everything that is necessary.
@@ -48,7 +100,7 @@ Existing Nautobot users may want to incorporate the Onboarding App as part of on
 
 ### Authors
 
-@mzb and @dgarros
+@mzb and @dgarros and many other great contributors!
 
 ### Maintainers
 
@@ -61,3 +113,4 @@ Existing Nautobot users may want to incorporate the Onboarding App as part of on
 
 - Secrets & SecretsGroup
 - Jobs
+- Datasources