Prep 4.0.0b0 for a tagged release to pypi (#182)

* add a more advanced jdiff jpath query to ios for ni interfaces * updates for parsing * bump version * update network importer * add platform none check * fix formatting * add conditional for no platform * add conditional for no platform * add conditional for no platform * add unsupported check * add readtimeout * add failed task UI logging * add junos support to do * add logging for failed reason * add junos support to do * update junos, add json load to linterfaces * update junos, add json load to linterfaces * update ios to use tojson filter * update ios to use tojson filter * formatting, update function name and doc strings * use j2 sandbox and nb core render function * poetry lock and black * black * fix juniper_junos.yml * juniper_junos.yml formatting * update ssot network adapter, models and mock data * ios NI initial working * fix do for junos * add csv support wip * update csv import feature * update csv support * update csv support * fix dup ip address extraction * fix mgmt_interface from list to string * update csv support * add orm caching method to onboarding adapter * update csv support * remove print statement * send interfaces in ios as json * update csv support * bump version * update NI * remove tag filter option * remove tag filter * decouple nornir play from job to remove enqueue job option * clean up via black * update csv import logging * fix bug in _process_csv_data * remove command getter job * update NI * updates for description and enabled * update mock data * formatting * black * pylint * update logging message * flake8 * update CSV support * update lock file * update lock and toml files * update logging around existing devies * change command mapper formatting and DO for junos * fix junos * remove prefix_length from OnboardingDeviceModel * add data check for fields returned by devices * enhance junos mgmt int and mask captures * enhance junos mgmt int and mask captures * fixes for most jnos * rework yaml format and processor formatter * fixed nxos serial * improve logging and calls to secrets providers * update onboarding adapter * set defaut platform value to None when loading devices in onboarding adapter * yamllint, ruff * add more performant lookup to mgmt interface cisco ios * bump version, fix bug with DO adapter * update DO adapters, models and job * bump version * formatting changes * changed ip address to list * fix yamllint, pylint and juniper yaml jpaths * updates to NI formatting * fix junos mgmt intrface post processor * fix junos * Update juniper_junos.yml * Update juniper_junos.yml * update ni add nxos * update vlans * update NI adapters and models, add untagged vlan model * bump * linting * update lock file * updates for multiple devices * update mac address conversion * add cache for primary ips to NI adapter * update sync complete * update sync complete * update sync complete * add sync complete doc string * updated interface mapping * bump * black * pylint * add error handling * bump version * black * yamlint * updates for vrf * revamps and cleanups * update list conversion * more refactors and cleanups * fix custom filters development config * remove jinja2 fitler registration in favor of native support with decorator * linters, formatters, stuff * fix platform parsing info * fix return indent level * fix return indent level * add default dict * more cleanups remove transform function * more refactors and few cleanups * add vrf to ssot sync, fix vlan bug, update logging * remove unnecessary logging * updates for VRF support * removed exception for vrf without rd * first DO for WLC * added 802.1q support for ios * fix / add needed info for wlc * updates for vlans ios * adjust vrf adapter to load an RD of "None" if not present * update vrf model * update vrf model for NI * add schema and some more new filters and cleanups * update for handling * update * vlans add * first pass at adding docs for new ssot features * more doc adds * more doc adds * vlans for nxos working * linting * fix example dict keys * remove RD from vrf sync * update mock data * removed rd from vrf * add wlc mapper and support for do * update jinja_filters.py, lock and bump version * fix link in docs * update mkdocs * removed extra rd config * cleanup * Jkala prep beta (#174) * rename jobs throughout codebase * rename jobs throughout docs * few more updates to original definition * more rename changes with black * bump to beta version * fix lock file inconsistances * initial round of fixes for mapping etc. * change use_textfsm key to parser and update interworks phase1 * change use_textfsm key to parser and update interworks phase2 * interface mapping comments * fix dedup function to work with dict or list of commands * fix support of dict or list def of commands * use mapping netmiko specifically * mock data * add platform to NI device filters * updates to vlan and vrf * update dependencies, pre-merge linting, black * update lock * update nautobot version in github ci * update role filter params * fix error messages, add defaults * New framework and cleanups. (#178) * prototype working first commit * fix a few device onboarding issues * more updates to framework * getting nxos at least working with sync network data * finish renames and cleanup of old functions * logic based on vlans and vrfs toggle * sync net data for arista * testing * another round of fixes * handle main task failures in processor * handle main task failures in processor * handle main task failures in processor * update for wlc wlc_ssh oddities * wlc updates and upgrade ntc-templates to master branch * temp juniper * temp juniper * temp juniper * add lag and vrf to ios and xe more junos fixes * add lag and vrf to ios and xe more junos fixes * add new filter * fix lags for ios and xe * fix lags for ios and xe * fix lags for ios and xe * fix lags for ios and xe * fix logic for dot1q * sync ios and xe mappers * sync ios and xe mappers * more updates * more updates * working on interface syncs * ios and xe fix vrf, lags for xes with and without switchport command * junos work * junos work * junos work * junos work * junos work * junos work * junos work * junos work * junos work * more junos fixes * junos work * junos work * fix junos unlimited mtu issue * junos fix vrf jpath * junos fix vrf jpath and command * junos fix vrf jpath and command * fix lag parsing to only parent physical interface * junos fix vrf jpath and command * fix arista lags and vrfs * remove prints and add a specific logger for ETL * update to beta2 * linting and formatting cleanups * linting and formatting cleanups round 2 * fix lock file * revert change to dockerfile --------- Co-authored-by: Jeff Kala <jeff.kala@chevron.com> * few fixes for parsing issues when result is empty list * update lag constants for nxos * hide vlans sync option * fix vrf for ios and xe * add default of empty list * wlc fixes * wlc fixes * more juniper work * more juniper work * changed job description * fix junos ip address * fix bug in network data models * yamlint, black * update version and lock * add schema validation * clean up logging * add date tracking and additional error handling (#180) * add date tracking to sync * update logging and error handling * remove unnecessary line * black . * update lock * udpate comment --------- Co-authored-by: David Cates <david.cates@networktocode.com> * fix commit * add unittest v1 * remove merge conflict additions in processor, bandit cleanups * pylint disables and formatting * update import exception * add debug for ntc-template parsed result * clean up processor logging * add test for SyncDevicesNetworkAdapter * remove mock data file * add more test * first sync network data test and run formatting * update custom field * prep 4.0 beta0 release and get all test passing * fix sorting on test * fix sorting on test * add troubleshooting processor and hidden job * add hidden job with json loads for none parsers * finalize new unittest for sync devices --------- Co-authored-by: Susan Hooks <susan.hooks@chevron.com> Co-authored-by: David Cates <david.cates@networktocode.com> Co-authored-by: David Cates <57967713+Dav-C@users.noreply.github.com> Co-authored-by: Jeff Kala <jeff.kala@chevron.com>
nautobot · May 29, 2024 · 5122fb1 · 5122fb1
1 parent ae6896b
commit 5122fb1
Show file tree

Hide file tree

Showing 45 changed files with 29,581 additions and 2,405 deletions.
diff --git a/.gitignore b/.gitignore
@@ -297,7 +297,7 @@ fabric.properties
 
 # Rando
 creds.env
-development/*.txt
+development/
 
 # Invoke overrides
 invoke.yml

diff --git a/development/docker-compose.base.yml b/development/docker-compose.base.yml
@@ -13,7 +13,6 @@ x-nautobot-base: &nautobot-base
     - "creds.env"
   tty: true
 
-version: "3.8"
 services:
   nautobot:
     depends_on:

diff --git a/development/docker-compose.dev.yml b/development/docker-compose.dev.yml
@@ -3,7 +3,6 @@
 # any override will need to include these volumes to use them.
 # see:  https://github.com/docker/compose/issues/3729
 ---
-version: "3.8"
 services:
   nautobot:
     command: "nautobot-server runserver 0.0.0.0:8080"

diff --git a/development/docker-compose.mysql.yml b/development/docker-compose.mysql.yml
@@ -1,6 +1,4 @@
 ---
-version: "3.8"
-
 services:
   nautobot:
     environment:

diff --git a/development/docker-compose.postgres.yml b/development/docker-compose.postgres.yml
@@ -1,6 +1,4 @@
 ---
-version: "3.8"
-
 services:
   nautobot:
     environment:

diff --git a/development/docker-compose.redis.yml b/development/docker-compose.redis.yml
@@ -1,5 +1,4 @@
 ---
-version: "3.8"
 services:
   redis:
     image: "redis:6-alpine"

diff --git a/development/nautobot_config.py b/development/nautobot_config.py
@@ -127,11 +127,7 @@
 #
 
 # Enable installed Apps. Add the name of each App to the list.
-PLUGINS = [
-    "nautobot_device_onboarding",
-    "nautobot_ssot",
-    "nautobot_plugin_nornir",
-]
+PLUGINS = ["nautobot_device_onboarding", "nautobot_ssot", "nautobot_plugin_nornir"]
 
 # Apps configuration settings. These settings are used by various Apps that the user may have installed.
 # Each key in the dictionary is the name of an installed App and its value is a dictionary of settings.

diff --git a/nautobot_device_onboarding/choices.py b/nautobot_device_onboarding/choices.py
@@ -0,0 +1,7 @@
+"""Choices used througout the app."""
+
+SSOT_JOB_TO_COMMAND_CHOICE = (
+    ("sync_devices", "Sync Devices"),
+    ("sync_network_data", "Sync Network Data"),
+    ("both", "Both"),
+)
diff --git a/nautobot_device_onboarding/diffsync/adapters/sync_network_data_adapters.py b/nautobot_device_onboarding/diffsync/adapters/sync_network_data_adapters.py
diff --git a/nautobot_device_onboarding/diffsync/models/sync_network_data_models.py b/nautobot_device_onboarding/diffsync/models/sync_network_data_models.py
@@ -2,14 +2,19 @@
 
 from typing import List, Optional
 
+try:
+    from typing import Annotated  # Python>=3.9
+except ImportError:
+    from typing_extensions import Annotated  # Python<3.9
+
 from diffsync import DiffSync, DiffSyncModel
 from diffsync import exceptions as diffsync_exceptions
 from django.core.exceptions import MultipleObjectsReturned, ObjectDoesNotExist, ValidationError
 from nautobot.dcim.choices import InterfaceTypeChoices
 from nautobot.dcim.models import Device, Interface, Location
 from nautobot.extras.models import Status
 from nautobot.ipam.models import VLAN, VRF, IPAddress, IPAddressToInterface
-from nautobot_ssot.contrib import NautobotModel
+from nautobot_ssot.contrib import CustomFieldAnnotation, NautobotModel
 
 from nautobot_device_onboarding.utils import diffsync_utils
 
@@ -48,11 +53,16 @@ class SyncNetworkDataDevice(FilteredNautobotModel):
         "name",
         "serial",
     )
+    _attributes = ("last_network_data_sync",)
     _children = {"interface": "interfaces"}
 
     name: str
     serial: str
 
+    last_network_data_sync: Annotated[
+        Optional[str], CustomFieldAnnotation(key="last_network_data_sync", name="last_network_data_sync")
+    ]
+
     interfaces: List["SyncNetworkDataInterface"] = []
 
     @classmethod

diff --git a/nautobot_device_onboarding/jobs.py b/nautobot_device_onboarding/jobs.py
@@ -7,15 +7,19 @@
 
 from diffsync.enum import DiffSyncFlags
 from django.conf import settings
+from django.contrib.contenttypes.models import ContentType
 from django.core.exceptions import ObjectDoesNotExist, ValidationError
 from django.forms import HiddenInput
-from nautobot.apps.jobs import BooleanVar, FileVar, IntegerVar, Job, MultiObjectVar, ObjectVar, StringVar
+from nornir import InitNornir
+from nornir.core.plugins.inventory import InventoryPluginRegister
+from nautobot.apps.jobs import BooleanVar, FileVar, IntegerVar, Job, MultiObjectVar, ObjectVar, StringVar, ChoiceVar
 from nautobot.core.celery import register_jobs
 from nautobot.dcim.models import Device, DeviceType, Location, Platform
-from nautobot.extras.choices import SecretsGroupAccessTypeChoices, SecretsGroupSecretTypeChoices
-from nautobot.extras.models import Role, SecretsGroup, SecretsGroupAssociation, Status
+from nautobot.extras.choices import CustomFieldTypeChoices, SecretsGroupAccessTypeChoices, SecretsGroupSecretTypeChoices
+from nautobot.extras.models import CustomField, Role, SecretsGroup, SecretsGroupAssociation, Status
 from nautobot.ipam.models import Namespace
 from nautobot_ssot.jobs.base import DataSource
+from nautobot_plugin_nornir.constants import NORNIR_SETTINGS
 
 from nautobot_device_onboarding.diffsync.adapters.sync_devices_adapters import (
     SyncDevicesNautobotAdapter,
@@ -28,6 +32,13 @@
 from nautobot_device_onboarding.exceptions import OnboardException
 from nautobot_device_onboarding.netdev_keeper import NetdevKeeper
 from nautobot_device_onboarding.utils.helper import onboarding_task_fqdn_to_ip
+from nautobot_device_onboarding.nornir_plays.empty_inventory import EmptyInventory
+from nautobot_device_onboarding.nornir_plays.inventory_creator import _set_inventory
+from nautobot_device_onboarding.nornir_plays.command_getter import netmiko_send_commands, _parse_credentials
+from nautobot_device_onboarding.choices import SSOT_JOB_TO_COMMAND_CHOICE
+from nautobot_device_onboarding.nornir_plays.processor import TroubleshootingProcessor
+
+InventoryPluginRegister.register("empty-inventory", EmptyInventory)
 
 PLUGIN_SETTINGS = settings.PLUGINS_CONFIG["nautobot_device_onboarding"]
 
@@ -614,6 +625,28 @@ def run(
         self.sync_vlans = sync_vlans
         self.sync_vrfs = sync_vrfs
 
+        # Check for last_network_data_sync CustomField
+        if self.debug:
+            self.logger.debug("Checking for last_network_data_sync custom field")
+        try:
+
+            cf = CustomField.objects.get(key="last_network_data_sync")
+        except ObjectDoesNotExist:
+            cf, _ = CustomField.objects.get_or_create(
+                label="Last Network Data Sync",
+                key="last_network_data_sync",
+                type=CustomFieldTypeChoices.TYPE_DATE,
+                required=False,
+            )
+
+            cf.content_types.add(ContentType.objects.get_for_model(Device))
+
+            if self.debug:
+                self.logger.debug("Custom field found or created")
+        except Exception as err:  # pylint: disable=broad-exception-caught
+            self.logger.error(f"Failed to get or create last_network_data_sync custom field, {err}")
+            return
+
         # Filter devices based on form input
         device_filter = {}
         if self.devices:
@@ -634,7 +667,7 @@ def run(
             self.logger.info("No devices returned based on filter selections.")
             return
 
-        # Log selected devices
+        # Log the devices that will be synced
         filtered_devices_names = list(self.filtered_devices.values_list("name", flat=True))
         self.logger.info(f"{len(filtered_devices_names)} devices will be synced")
         if len(filtered_devices_names) <= 300:
@@ -656,5 +689,72 @@ def run(
         super().run(dryrun, memory_profiling, *args, **kwargs)
 
 
-jobs = [OnboardingTask, SSOTSyncDevices, SSOTSyncNetworkData]
+class DeviceOnboardingTroubleshootingJob(Job):
+    """Simple Job to Execute Show Command."""
+
+    debug = BooleanVar()
+    ip_addresses = StringVar()
+    port = IntegerVar(default=22)
+    timeout = IntegerVar(default=30)
+    secrets_group = ObjectVar(model=SecretsGroup)
+    platform = ObjectVar(model=Platform, required=True)
+    ssot_job_type = ChoiceVar(choices=SSOT_JOB_TO_COMMAND_CHOICE)
+
+    class Meta:
+        """Meta object boilerplate for onboarding."""
+
+        name = "Runs Commands on a Device to simulate SSoT Command Getter."
+        description = "Login to a device(s) and run commands."
+        has_sensitive_variables = False
+        hidden = True
+
+    def run(self, *args, **kwargs):
+        """Process onboarding task from ssot-ni job."""
+        ip_addresses = kwargs["ip_addresses"].replace(" ", "").split(",")
+        port = kwargs["port"]
+        platform = kwargs["platform"]
+        username, password, secret = _parse_credentials(kwargs["secrets_group"])  # pylint:disable=unused-variable
+
+        # Initiate Nornir instance with empty inventory
+        compiled_results = {}
+        try:
+            with InitNornir(
+                runner=NORNIR_SETTINGS.get("runner"),
+                logging={"enabled": False},
+                inventory={
+                    "plugin": "empty-inventory",
+                },
+            ) as nornir_obj:
+                for entered_ip in ip_addresses:
+                    single_host_inventory_constructed, _ = _set_inventory(
+                        entered_ip, platform, port, username, password
+                    )
+                    nornir_obj.inventory.hosts.update(single_host_inventory_constructed)
+                nr_with_processors = nornir_obj.with_processors([TroubleshootingProcessor(compiled_results)])
+                if kwargs["ssot_job_type"] == "both":
+                    nr_with_processors.run(
+                        task=netmiko_send_commands,
+                        command_getter_yaml_data=nornir_obj.inventory.defaults.data["platform_parsing_info"],
+                        command_getter_job="sync_devices",
+                        **kwargs,
+                    )
+                    nr_with_processors.run(
+                        task=netmiko_send_commands,
+                        command_getter_yaml_data=nornir_obj.inventory.defaults.data["platform_parsing_info"],
+                        command_getter_job="sync_network_data",
+                        **kwargs,
+                    )
+                else:
+                    nr_with_processors.run(
+                        task=netmiko_send_commands,
+                        command_getter_yaml_data=nornir_obj.inventory.defaults.data["platform_parsing_info"],
+                        command_getter_job=kwargs["ssot_job_type"],
+                        **kwargs,
+                    )
+        except Exception as err:  # pylint: disable=broad-exception-caught
+            self.logger.info("Error During Sync Devices Command Getter: %s", err)
+        return compiled_results
+
+
+jobs = [OnboardingTask, SSOTSyncDevices, SSOTSyncNetworkData, DeviceOnboardingTroubleshootingJob]
 register_jobs(*jobs)
diff --git a/nautobot_device_onboarding/nornir_plays/command_getter.py b/nautobot_device_onboarding/nornir_plays/command_getter.py
@@ -1,5 +1,6 @@
 """CommandGetter."""
 
+import json
 from typing import Dict
 
 from django.conf import settings
@@ -51,7 +52,17 @@ def _get_commands_to_run(yaml_parsed_info, sync_vlans, sync_vrfs):
     """Using merged command mapper info and look up all commands that need to be run."""
     all_commands = []
     for key, value in yaml_parsed_info.items():
-        if not key.startswith("_metadata"):
+        if key == "pre_processor":
+            for _, v in value.items():
+                current_root_key = v.get("commands")
+                if isinstance(current_root_key, list):
+                    # Means their is any "nested" structures. e.g multiple commands
+                    for command in v["commands"]:
+                        all_commands.append(command)
+                else:
+                    if isinstance(current_root_key, dict):
+                        all_commands.append(current_root_key)
+        else:
             # Deduplicate commands + parser key
             current_root_key = value.get("commands")
             if isinstance(current_root_key, list):
@@ -124,6 +135,14 @@ def netmiko_send_commands(task: Task, command_getter_yaml_data: Dict, command_ge
                             except Exception:  # https://github.com/networktocode/ntc-templates/issues/369
                                 task.results[result_idx].result = []
                                 task.results[result_idx].failed = False
+            else:
+                if command["parser"] == "none":
+                    try:
+                        jsonified = json.loads(current_result.result)
+                        task.results[result_idx].result = jsonified
+                        task.results[result_idx].failed = False
+                    except Exception:
+                        task.result.failed = False
         except NornirSubTaskError:
             # We don't want to fail the entire subtask if SubTaskError is hit, set result to empty list and failt to False
             # Handle this type or result latter in the ETL process.