Error "ScriptModule matching query does not exist." after upgrading netbox to 3.5.1

[kemeris2000]

I have updated netbox to version 3.5.1, removed netbox-dns and installed netbox-plugin-dns packages.
Now on dns zone update event triggered custom script fails with error "ScriptModule matching query does not exist". I am novice in python, can anyone help me with this problem?

Error: requests.exceptions.RequestException: Status 500 returned with content 'b'{"error": "ScriptModule matching query does not exist.", "exception": "DoesNotExist", "netbox_version": "3.5.1", "python_version": "3.9.2"}'', webhook FAILED to process.

/opt/netbox/netbox/scripts/bindrndc.py:

from extras.scripts import *
from utilities.exceptions import AbortScript
from collections import OrderedDict
from netbox_dns.models import NameServer, Record
import sys
import os
import subprocess
sys.path.append("/opt/netbox/netbox/scripts/isc")
import rndc
import dns.query
import dns.update
import dns.name
import hashlib
import paramiko
from dotmap import DotMap
import json
import yaml

class bindrndc(Script):
    class Meta:
       name = "bind"
       description = "Manage DNS zones on name servers"


    def zone_file_generate(self, modeldata, records):
        zone_file_content = f"; {modeldata.name}\n"
        zone_file_content += f"{modeldata.name}. {modeldata.soa_ttl} IN SOA {modeldata.soa_mname.name}. {modeldata.soa_rname}. {modeldata.soa_serial} {modeldata.soa_refresh} {modeldata.soa_retry} {modeldata.soa_expire} {modeldata.soa_minimum}\n"
        reqlen_offset = 2
        reqlen_name = len(max([d['name'] for d in records], key=len)) + reqlen_offset
        reqlen_in = len('IN') + reqlen_offset
        reqlen_type = len(max([d['type'] for d in records], key=len)) + reqlen_offset
        for i in records:
            zone_file_content += f"{i['name'].ljust(reqlen_name)}{'IN'.ljust(reqlen_in)}{i['type'].ljust(reqlen_type)} {i['value']}\n"
        return zone_file_content


    def zone_check_if_exist(self, modeldata, nameserver):
        try:
            r = rndc.rndc((nameserver['custom_field_data']['ns_ip'], nameserver.custom_field_data.rndc_port), nameserver.custom_field_data.rndc_key_algorithm, nameserver.custom_field_data.rndc_key_value)
        except Exception as e:
             return({"status": "error", "stage": "zone_exists", "server": nameserver.name +" ("+ nameserver.custom_field_data.ns_ip +")", "value":  "zone "+ modeldata.name +" script
error: "+ format(e).replace("'", "")})
        try:
            response = r.call('zonestatus '+ modeldata.name)
            if response['result'].decode('utf-8') == "0":
                return({"status": "exists"})
            else:
                return({"status": "notexists" })
        except Exception as e:
             return({"status": "error", "stage": "zone_exists", "server": nameserver.name +" ("+ nameserver.custom_field_data.ns_ip +")", "value": "zone "+ modeldata.name +" script error: "+ format(e).replace("'", "")})


    def zone_validate(self, modeldata, zone_file_content):
        try:
            result = subprocess.run(f"/usr/sbin/named-checkzone {modeldata.name} <(echo -e \"{zone_file_content}\")", stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, executable='/bin/bash', shell=True)
            if result.stdout.find('OK') != -1:
                return({"status": "success", "stage": "zone_validate", "server": "localhost", "value": "OK"})
            else:
                if result.stderr != "":
                    return({"status": "error", "stage": "zone_validate", "server": "localhost", "value": result.stderr})
                else:
                    return({"status": "error", "stage": "zone_validate", "server": "localhost", "value": result.stdout})
        except Exception as e:
             return({"status": "error", "stage": "zone_validate", "server": "localhost", "value": "zone "+ modeldata.name +" script error: "+ format(e).replace("'", "")})


    def zone_file_upload(self, modeldata, nameserver, zone_file_content):
        ssh = paramiko.SSHClient()
        ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
        try:
            ssh.connect(nameserver.custom_field_data.ns_ip, username='netbox', timeout=10)
            cmd = "sudo /bin/bash -c 'echo \""+ zone_file_content +"\" > /var/cache/bind/primary/"+ modeldata.name +".db'"
            stdin, stdout, stderr = ssh.exec_command(cmd, get_pty=True)
            stdout = stdout.read().decode('utf-8').replace("\r\n", "")
            stderr = stderr.read().decode('utf-8').replace("\r\n", "")
            if stdout == "":
                return({"status": "success", "stage": "zone_file_upload", "server": nameserver.name + " ("+ nameserver.custom_field_data.ns_ip +")" ,"value": "zone "+ modeldata.name
+" file upload successfull"})
            else:
                if stderr != "":
                    return({"status": "error", "stage": "zone_file_upload", "server": nameserver.name + " ("+ nameserver.custom_field_data.ns_ip +")" ,"value": "zone "+ modeldata.name +" file upload error: "+ stderr})
                else:
                    return({"status": "error", "stage": "zone_file_upload", "server": nameserver.name + " ("+ nameserver.custom_field_data.ns_ip +")" ,"value": "zone "+ modeldata.name +" file upload error: "+ stdout})
        except Exception as e:
            return({"status": "error", "stage": "zone_file_upload", "server": nameserver.name + " ("+ nameserver.custom_field_data.ns_ip +")", "value": "zone "+ modeldata.name +" script error: "+ format(e).replace("'", "")})


    def zone_add(self, modeldata, nameserver):
        r = rndc.rndc((nameserver['custom_field_data']['ns_ip'], nameserver.custom_field_data.rndc_port), nameserver.custom_field_data.rndc_key_algorithm, nameserver.custom_field_data.rndc_key_value)
        response = r.call('addzone '+ modeldata.name +' {type primary; masterfile-format text; file "primary/'+ modeldata.name +'.db";};')
        if "err" in response:
            return({"status": "error", "value": nameserver.name + " (" + nameserver.custom_field_data.ns_ip + ") zone "+ modeldata.name +" add error: "+ response['err'].decode('utf-8')})
        else:
            if response['result'].decode('utf-8') == "0":
                return({"status": "success", "value": nameserver.name + " (" + nameserver.custom_field_data.ns_ip + ") zone "+ modeldata.name +" added successfully"})
            else:
                return({"status": "error", "value": nameserver.name + " (" + nameserver.custom_field_data.ns_ip + ") zone "+ modeldata.name +" add error: "+ response['result'].decode('utf-8')})


    def zone_mod(self, modeldata, nameserver):
        r = rndc.rndc((nameserver['custom_field_data']['ns_ip'], nameserver.custom_field_data.rndc_port), nameserver.custom_field_data.rndc_key_algorithm, nameserver.custom_field_data.rndc_key_value)
        response = r.call('modzone '+ modeldata.name +' {type primary; masterfile-format text; file "primary/'+ modeldata.name +'.db";};')
        if "err" in response:
            return({"status": "error", "stage": "zone_mod", "server": nameserver.name + " ("+ nameserver.custom_field_data.ns_ip +")", "value": "zone "+ modeldata.name +" update error: "+ response['err'].decode('utf-8')})
        else:
            if response['result'].decode('utf-8') == "0":
                return({"status": "success", "stage": "zone_mod", "server": nameserver.name + " ("+ nameserver.custom_field_data.ns_ip +")", "value": "zone "+ modeldata.name +" modify successfully"})
            else:
                return({"status": "error", "stage": "zone_mod", "server": nameserver.name + " ("+ nameserver.custom_field_data.ns_ip +")", "value": "zone "+ modeldata.name +" modify
error: "+ response['result'].decode('utf-8')})


    def zone_delete(self, modeldata, nameserver):
        try:
            r = rndc.rndc((nameserver['custom_field_data']['ns_ip'], nameserver.custom_field_data.rndc_port), nameserver.custom_field_data.rndc_key_algorithm, nameserver.custom_field_data.rndc_key_value)
        except Exception as e:
            return({"status": "error", "stage": "zone_delete", "server": nameserver.name +" ("+ nameserver.custom_field_data.ns_ip +")", "value": "zone "+ modeldata.name +" script error1: "+ format(e).replace("'", "")})
        try:
            response = r.call('delzone -clean '+ modeldata.name)
            if response['result'].decode('utf-8') == "0":
                return({"status": "success", "stage": "zone_delete", "server": nameserver.name + " ("+ nameserver.custom_field_data.ns_ip +")", "value": response['text'].decode('utf-8')})
            else:
                return({"status": "error", "stage": "zone_delete", "server": nameserver.name + " ("+ nameserver.custom_field_data.ns_ip +")", "value": response['text'].decode('utf-8')})
        except Exception as e:
            return({"status": "error", "stage": "zone_delete", "server": nameserver.name + " ("+ nameserver.custom_field_data.ns_ip +")", "value": "zone "+ modeldata.name +" script error2: "+ format(e).replace("'", "")})


    def run(self, data, commit):
        data = DotMap(data)
        netbox_event = data.netbox_event
        #return data.model_data
        modeldata = eval(data.model_data)
        if isinstance(modeldata['view'], dict):
            modeldata['view'] = dict(modeldata['view'])
        else:
            modeldata['view'] = ""
#        modeldata['nameservers'] = [ dict(i) for i in modeldata['nameservers']]
        modeldata['nameservers'] = list(NameServer.objects.filter(name__in = [d['name'] for d in modeldata['nameservers']]).values())
        modeldata = DotMap(modeldata)
        records = list(Record.objects.filter(zone_id = modeldata.id, active = True).exclude(type = 'SOA').values())

        ###
        ### Zone created event
        ###
        if netbox_event == 'created':
            zone_add_result = []
            for i in modeldata.nameservers:
                ##
                ## Check if exists
                ##
                zone_check_if_exist = self.zone_check_if_exist(modeldata, i)
                if zone_check_if_exist['status'] == 'exists':
                    #
                    # Zone exists, error
                    #
                    zone_add_result.append([{"status": "error", "stage": "zone_exists", "server": i.name +" ("+ i.custom_field_data.ns_ip +")", "value": "zone "+ modeldata.name +" already exists"}])
#                    return zone_add_result
                    continue
                elif zone_check_if_exist['status'] == 'notexists':
                    #
                    # Zone does not exists - create
                    #
                    # generate zone file
                    zone_file_content = self.zone_file_generate(modeldata, records)
                    # validate zone
                    zone_validate = self.zone_validate(modeldata, zone_file_content)
                    if zone_validate['status'] != 'success':
                        raise AbortScript(zone_validate['value'])
                    # upload zone file
                    zone_file_upload_result = []
                    for i in modeldata.nameservers:
                        zone_file_upload = self.zone_file_upload(modeldata, i, zone_file_content)
                        zone_file_upload_result.append(zone_file_upload)
                    if any(d['status'] == 'error' for d in zone_file_upload_result):
                        raise AbortScript('\n\n'.join([d['value'] for d in zone_file_upload_result]))
                    # add zone
                    zone_add_result = []
                    for i in modeldata.nameservers:
                        zone_add = self.zone_add(modeldata, i)
                        zone_add_result.append(zone_add)
                    if any(d['status'] == 'error' for d in zone_add_result):
                        raise AbortScript('\n\n'.join([d['value'] for d in zone_add_result]))
                    return zone_add
#                else: unknown zone status
                return zone_add_result

        ###
        ### Zone updated event
        ###
        if netbox_event == 'updated':
            return "updated"         
        ###
        ### Zone deleted event
        ###
        if netbox_event == 'deleted':
            zone_check_if_exist_result = []
            for i in modeldata.nameservers:
                ##
                ## Check if exists
                ##
                zone_check_if_exist = self.zone_check_if_exist(modeldata, i)
                if zone_check_if_exist['status'] == 'exists':
                    #
                    # Zone exists, delete
                    #
                    # generate zone file
                    zone_delete_result = self.zone_delete(modeldata, i)
                    result = zone_delete_result
                    #raise AbortScript('\n\nError: zone already exists. '.join([d['value'] for d in zone_check_if_exist_result]))
                    #result = ({"status": "error", "value": nameserver.name + " (" + nameserver.custom_field_data.ns_ip + ") zone "+ modeldata.name +" script error: "+ format(e).replace("'", "")})
                elif zone_check_if_exist['status'] == 'notexists':
                    #
                    # Zone not exists, skip
                    #
                    result = ({"status": "success", "stage": "zone_delete", "server": i.name +" ("+ i.custom_field_data.ns_ip +")", "value": "zone "+ modeldata.name +" does not exists - skipping"})
                else:
                    result = zone_check_if_exist
                zone_check_if_exist_result.append(result)
            if any(d['status'] == 'error' for d in zone_check_if_exist_result):
                raise AbortScript('\n\n'+ yaml.dump(zone_check_if_exist_result))
            else:
                return yaml.dump(zone_check_if_exist_result)

[peteeckel]

Hi @kemeris2000, did this script work before you upgraded to NetBox 3.5.1/NetBox DNS 0.18.0?

If it did, could you please provide everything that is required to make it work? There is at least one reference to the path /opt/netbox/netbox/scripts/isc in the code, so there is content you did not provide that will (probably) be required to run the script successfully.

It would also be helpful to have a minimal version of the script that demonstrates the issue. Minimal in this context means that in the best case there no code that is not required to cause the problem, but the problem is still present. This helps a lot in finding and fixing the problem. In your example you wrote that the issue is occurring when zones are updated, so at the very least all the other event handlers probably can be omitted.

[kemeris2000]

Thank you for reply peteeckel. This script is not finished but was fully working on Netbox 3.4.7. After upgrade rest of the scripts works, except this one. Have reduced script to minimum and get same problem, no matter which content type triggers it. Tried to change class name, still does not work. I suspect this is netbox problem and not netbox-plugin-dns.

from extras.scripts import *

class bindrndc(Script):
    class Meta:
       name = "bind"
       description = "Manage DNS zones on name servers"


    def run(self, data, commit):
        return "aaaa"

[peteeckel]

Hi @kemeris2000, I converted this issue to a discussion as it isn't an issue with NetBox DNS (or NetBox).

With your minimal script, we can run the tests quite easily. Let's do a walkthrough so you get an idea how that can be done.

Test the script itself

I just recreated your script on my NetBox 3.5.1 test instance (as file test.py):

#!/usr/bin/env python3

from extras.scripts import *

class bindrndc(Script):
    class Meta:
       name = "bind"
       description = "Manage DNS zones on name servers"


    def run(self, data, commit):
        return "aaaa"

In the GUI, look under 'Customization/Scripts' if it shows up. If it doesn't, one possible reason is permission problems - I stumbled across that myself in the first attempt. Another issue might be SELinux, look at the audit log for hints. But the script itself is OK. If it shows up like this, you should be fine:

Run the script from the GUI

You can also run the script from the GUI, and as expected it works like a charm:

Test the API invocation of the script

The next step is to test whether the script can be invoked from the API, which is what your webhook uses. You probably have an API token for this, so let's see what happens. And voilá:

pete@macallan ~ % curl -k -X "POST" "https://192.168.106.105/api/extras/scripts/test.bindrndc/" \
     -H 'Authorization: Token c6e0d365eeb5eb225bd5ae68916b1d3b8b8c5b24' \
     -H 'Content-Type: application/json' \
     -H 'Accept: application/json; indent=4' \
     -d $'{
  "commit": "false",
  "data": {}
}'

{
    "url": "http://192.168.106.105/api/extras/scripts/test.bindrndc/",
    "id": "test.bindrndc",
    "module": "test",
    "name": "bind",
    "description": "Manage DNS zones on name servers",
    "vars": {},
    "result": {
        "id": 4,
        "url": "http://192.168.106.105/api/core/jobs/4/",
        "display": "ecd2d3d3-3f10-45a0-9372-491ade8b63ce",
        "object_type": "extras.scriptmodule",
        "object_id": 1,
        "name": "bindrndc",
        "status": {
            "value": "pending",
            "label": "Pending"
        },
        "created": "2023-05-14T11:42:27.021189Z",
        "scheduled": null,
        "interval": null,
        "started": null,
        "completed": null,
        "user": {
            "id": 1,
            "url": "http://192.168.106.105/api/users/users/1/",
            "display": "admin",
            "username": "admin"
        },
        "data": null,
        "job_id": "ecd2d3d3-3f10-45a0-9372-491ade8b63ce"
    }
}%                                                                                                                                  

Reproduce the error

Now how can we recreate your error message ScriptModule matching query does not exist?

Easy: Get the URL wrong. More precisely: Get the name of the script file wrong, which is the first part of the last path element in the URL:

pete@macallan ~ % curl -k -X "POST" "https://192.168.106.105/api/extras/scripts/testxxx.bindrndc/" \
     -H 'Authorization: Token c6e0d365eeb5eb225bd5ae68916b1d3b8b8c5b24' \
     -H 'Content-Type: application/json' \
     -H 'Accept: application/json; indent=4' \
     -d $'{
  "commit": "false",
  "data": {}
}'

{"error": "ScriptModule matching query does not exist.", "exception": "DoesNotExist", "netbox_version": "3.5.1", "python_version": "3.8.13"}%                                                                                                                           

When you mess up the second part (the name of your Script subclass) instead, you get a different error message:

pete@macallan ~ % curl -k -X "POST" "https://192.168.106.105/api/extras/scripts/test.bindrndcxxx/" \
     -H 'Authorization: Token c6e0d365eeb5eb225bd5ae68916b1d3b8b8c5b24' \
     -H 'Content-Type: application/json' \
     -H 'Accept: application/json; indent=4' \
     -d $'{
  "commit": "false",
  "data": {}
}'

{
    "detail": "Not found."
}%                                                                                                                                  

Conclusion

If you can rule out the first possible issue by checking the script in the GUI, you should try with the curl command shown above (adjusted to your file name and class name, and obviously your server URL). Very likely the error is the incorrectly specified file name, which should be easy to fix. Once that works, your full script and webhook should work as well.

If your script file is named bindrndc.py (I renamed it for the test to make the error more visible), your URL should end on api/extras/scripts/bindrndc.bindrndc/.

[kemeris2000]

Very informative post, you have made my life easier.
I forgot to fill "paramiko" package to requirements.txt before upgrade which led to netbox not loading custom script.
And I have managed to screw up script naming while trying to fix that.

Thank you

[peteeckel]

I forgot to fill "paramiko" package to requirements.txt ...

You should really use local_requirements.txt instead.

requirements.txt will be overwritten with NetBox upgrades, so your changes will be lost and things like this happen :-)