After upgrade from 3.5 to 3.7, instantiating a DeviceType which contains nested InventoryItems throws "InventoryItem matching query does not exist"

[padthaitofuhot]

Deployment Type

Self-hosted

NetBox Version

v3.7.4

Python Version

3.11

Steps to Reproduce

1. Upgrade from netbox-docker v3.5-2.6.1 to v3.7-2.8.0 via git release branch
2. Upgrade from Postgres 15 to Postgres 16 via sql dump/restore
3. Observe migrations complete successfully
4. Create a new DeviceType
5. Add InventoryItem A to new DeviceType
6. Add InventoryItem B to new DeviceType with InventoryItem A as Parent
7. Create a new Device from this DeviceType

Expected Behavior

A new Device is created with nested InventoryItems from DeviceType.

Observed Behavior

Instantiating a DeviceType which contains nested InventoryItems throws:

DoesNotExist at /dcim/devices/add/

InventoryItem matching query does not exist.

Request Method: 	POST
Request URL: 	http://netbox.mydomain.tld/dcim/devices/add/
Django Version: 	4.2.11
Exception Type: 	DoesNotExist
Exception Value: 	

InventoryItem matching query does not exist.

Exception Location: 	/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/query.py, line 637, in get
Raised during: 	dcim.views.DeviceEditView
Python Executable: 	/opt/netbox/venv/bin/python
Python Version: 	3.11.4
Python Path: 	

['/opt/netbox/netbox/',
 '/usr/lib/python311.zip',
 '/usr/lib/python3.11',
 '/usr/lib/python3.11/lib-dynload',
 '/opt/netbox/venv/lib/python3.11/site-packages']

Server time: 	Tue, 02 Apr 2024 00:05:31 +0000

Environment:


Request Method: POST
Request URL: http://netbox.mydomain.tld/dcim/devices/add/

Django Version: 4.2.11
Python Version: 3.11.4
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'django.forms',
 'corsheaders',
 'debug_toolbar',
 'graphiql_debug_toolbar',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'graphene_django',
 'mptt',
 'rest_framework',
 'social_django',
 'taggit',
 'timezone_field',
 'core',
 'account',
 'circuits',
 'dcim',
 'ipam',
 'extras',
 'tenancy',
 'users',
 'utilities',
 'virtualization',
 'vpn',
 'wireless',
 'django_rq',
 'drf_spectacular',
 'drf_spectacular_sidecar']
Installed Middleware:
['graphiql_debug_toolbar.middleware.DebugToolbarMiddleware',
 'django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'netbox.middleware.RemoteUserMiddleware',
 'netbox.middleware.CoreMiddleware',
 'netbox.middleware.MaintenanceModeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']



Traceback (most recent call last):
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
    response = get_response(request)
               ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
    return self.dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/netbox/views/generic/object_views.py", line 175, in dispatch
    return super().dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/netbox/views/generic/base.py", line 26, in dispatch
    return super().dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/utilities/views.py", line 106, in dispatch
    return super().dispatch(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
    return handler(request, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/netbox/views/generic/object_views.py", line 269, in post
    obj = form.save()
          ^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/forms/models.py", line 542, in save
    self.instance.save()
    ^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/dcim/models/devices.py", line 1054, in save
    self._instantiate_components(self.device_type.inventoryitemtemplates.all(), bulk_create=False)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/dcim/models/devices.py", line 999, in _instantiate_components
    components = [obj.instantiate(device=self) for obj in queryset]
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/dcim/models/devices.py", line 999, in <listcomp>
    components = [obj.instantiate(device=self) for obj in queryset]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/netbox/dcim/models/device_component_templates.py", line 765, in instantiate
    parent = InventoryItem.objects.get(name=self.parent.name, **kwargs) if self.parent else None
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/netbox/venv/lib/python3.11/site-packages/django/db/models/query.py", line 637, in get
    raise self.model.DoesNotExist(
    ^

Exception Type: DoesNotExist at /dcim/devices/add/
Exception Value: InventoryItem matching query does not exist.

[padthaitofuhot]

This issue looks a lot like #12118 which was fixed in PR #12170.

[padthaitofuhot]

BTW - I opened this issue here (rather than in docker-netbox) because it does not seem like an issue with the Docker image or related tooling and rather more like an issue with schema or a script.

[jeremystretch]

Thank you for opening a bug report. I was unable to reproduce the reported behavior on NetBox v3.7.4. Please re-confirm the reported behavior on the current stable release and adjust your post above as necessary. Remember to provide detailed steps that someone else can follow using a clean installation of NetBox to reproduce the issue. Remember to include the steps taken to create any initial objects or other data.

[kkthxbye-code]

Replicates fine here following his steps. The offending commit is this:

32264ac

The cause is pretty obvious, inventory items are intentionally created with bulk_create=False to avoid trying to reference a non-existant object.

The above commit moves the instantiation of the inventory items outside the bulk_create check. Reverting the commit allows creating the device again.

[jeremystretch]

Thanks @kkthxbye-code, would you like to submit a PR for this?

[padthaitofuhot]

@jeremystretch @kkthxbye-code I got it. I'll have a PR shortly.

[padthaitofuhot]

[padthaitofuhot]

Commit: padthaitofuhot@415cb30

@jeremystretch apparently I need you to assign this issue to me so my PR doesn't get automatically deleted. Please assign this issue to me?

IF YOUR PULL REQUEST DOES NOT REFERENCE AN ISSUE WHICH HAS BEEN ASSIGNED
TO YOU, IT WILL BE CLOSED AUTOMATICALLY.

[jeremystretch]

Thanks @padthaitofuhot!

[padthaitofuhot]

@jeremystretch PR submitted. Do I need to ping somewhere to ask for a review or is there some other process? Every project is a little different.

[jeremystretch]

@padthaitofuhot nope, we review PRs daily as they come in. I've just reviewed & merged yours. Thank you for contributing to NetBox!