Cache the openapi schema

[candlerb]

ISSUE TYPE

• Feature Idea

SOFTWARE VERSIONS

Ansible:

2.10.8

Netbox:

v2.11.3

Collection:

v3.0.1

SUMMARY

Every time the netbox inventory starts, the first thing it does is read the entire openapi schema. This is slow, as it is generated dynamically - it adds about an 8 second overhead, and transfers about 800KB of data over the network.

STEPS TO REPRODUCE

# time ansible -m ping wrn-vm1
wrn-vm1 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

real	0m11.870s
user	0m1.450s
sys	0m0.197s

But if I frig it to read a local copy of the openapi schema (see below):

# time ansible -m ping wrn-vm1
wrn-vm1 | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

real	0m3.964s
user	0m1.306s
sys	0m0.166s

This is about a 3-times speedup overall.

EXPECTED RESULTS

I would like to have the openapi schema cached.

Note: this is different to inventory caching, which can be configured, but caches the entire lookup results. I still want the most up-to-date data returned from Netbox, but without the overhead of fetching the schema.

ACTUAL RESULTS

The openapi schema is fetched in its entirety every time the inventory is accessed.

PROPOSAL

Create a cache file automatically somewhere (e.g. in the same directory that ansible uses to keep state), and ignore it if it's more than (say) 24 hours old.

As a proof-of-concept, I frigged the inventory to read a static file /etc/ansible/netbox-openapi.json if it exists.

--- ansible_collections/netbox/netbox/plugins/inventory/nb_inventory.py.orig	2021-05-15 14:37:31.057693343 +0100
+++ ansible_collections/netbox/netbox/plugins/inventory/nb_inventory.py	2021-05-15 15:19:44.055282352 +0100
@@ -1121,9 +1121,14 @@
             thread_exceptions = None

     def fetch_api_docs(self):
-        openapi = self._fetch_information(
-            self.api_endpoint + "/api/docs/?format=openapi"
-        )
+        openapi = None
+        try:
+            with open("/etc/ansible/netbox-openapi.json") as f:
+                openapi = json.load(f)
+        except FileNotFoundError:
+            openapi = self._fetch_information(
+                self.api_endpoint + "/api/docs/?format=openapi"
+            )

         self.api_version = version.parse(openapi["info"]["version"])
         self.allowed_device_query_parameters = [
--- ./ansible_collections/netbox/netbox/plugins/module_utils/netbox_utils.py.orig	2021-05-15 14:37:31.041692774 +0100
+++ ./ansible_collections/netbox/netbox/plugins/module_utils/netbox_utils.py	2021-05-15 15:16:20.455694158 +0100
@@ -579,7 +579,10 @@
         # Fetch the OpenAPI spec to perform validation against
         base_url = self.nb.base_url
         junk, endpoint_url = nb_endpoint.url.split(base_url)
-        response = open_url(base_url + "/docs/?format=openapi")
+        try:
+            response = open("/etc/ansible/netbox-openapi.json")
+        except FileNotFoundError:
+            response = open_url(base_url + "/docs/?format=openapi")
         try:
             raw_data = to_text(response.read(), errors="surrogate_or_strict")
         except UnicodeError:

[devon-mar]

Could a parameter such as openapi_spec be added which would point to a local JSON file containing a saved OpenAPI spec?

[rodvand]

I'm closing this as it seems to have been addressed in NetBox itself netbox-community/netbox#6423

[candlerb]

It works as well as it goes. If you've not used it for 24 hours you still get the slow startup though. I think the cache timeout could safely be made very long (e.g. 6 months) and be wiped by ./upgrade.sh

In the mean time, a nightly cronjob will probably do the trick:

curl -H 'Accept: application/json' http://localhost:8001/api/docs/