MPTT issues with using nested modules on pre v4.1 data

[sleepinggenius2]

Deployment Type

Self-hosted

NetBox Version

v4.1.0

Python Version

3.12

Steps to Reproduce

1. Navigate to a device with module bays that existed before v4.1
2. Set a parent on a module bay
   a. Edit an existing module bay and set the parent to an existing module; OR
   b. Add a new module bay with the parent set to an existing module

This has been verified on local systems that have been upgraded to v4.1, but you can also see an example of this happening on the demo site if you add a new module bay "Subslot 0/0" to module "Slot 0" on device "ncsu-coreswitch1". I understand that the demo site cannot be relied on and the example is purely for reference using data included in the standard database refresh.

Expected Behavior

The module bay appears in the list with the correct depth under the module bay for the parent module.

Observed Behavior

When the parent is changed on an existing module bay, the error "A node may not be made a child of any of its descendants." is presented. When a new module bay is added, the new module bay has the correct depth, but does not appear in the correct location in the module bay list.

A new module bay being added to a parent module in a new module bay appears to exhibit the expected behavior.

This looks like classic symptoms of MPTT being out of sync. When RackGroups were made nestable back in v2.8, the following migration was included:
https://github.com/netbox-community/netbox/blob/v2.8.0/netbox/dcim/migrations/0102_nested_rackgroups_rebuild.py. I do not see a similar migration accompanying https://github.com/netbox-community/netbox/blob/v4.1.0/netbox/dcim/migrations/0190_nested_modules.py.

[arthanson]

@sleepinggenius2 can you please re-check the repro steps and add further - I used 4.0 with a device that had existing module-bays, then upgraded it to 4.1 and tried step 2.b and it seemed to work fine.

[sleepinggenius2]

This was done locally on a system that was upgraded from 3.7.8 -> 4.0.11 -> 4.1.0.

This is the parent database row from 3.7.8 and 4.0.11:

-[ RECORD 1 ]-----+------------------------------
created           | 2023-07-27 22:53:09.771248+00
last_updated      | 2024-01-29 20:44:12.433466+00
custom_field_data | {}
id                | 10475
name              | Power Tray Slot 0
_name             | Power Tray Slot 00000000
label             | PS0
position          | 0/PT0
description       | 
device_id         | 36

This is the parent database row after the 4.1.0 upgrade:

-[ RECORD 1 ]-----+------------------------------
created           | 2023-07-27 22:53:09.771248+00
last_updated      | 2024-01-29 20:44:12.433466+00
custom_field_data | {}
id                | 10475
name              | Power Tray Slot 0
_name             | Power Tray Slot 00000000
label             | PS0
position          | 0/PT0
description       | 
device_id         | 36
level             | 0
lft               | 0
module_id         | 
parent_id         | 
rght              | 0
tree_id           | 0

In fact, every row in the dcim_modulebay table has the following additional fields after then migration:

level             | 0
lft               | 0
module_id         | 
parent_id         | 
rght              | 0
tree_id           | 0

After adding the child module bay in the UI, I have this new row in the database:

-[ RECORD 1 ]-----+------------------------------
created           | 2024-09-09 21:55:35.449377+00
last_updated      | 2024-09-09 21:55:35.449389+00
custom_field_data | {}
id                | 60803
name              | PEM Bay 0
_name             | PEM Bay 00000000
label             | M0
position          | 0/PT0-PM0
description       | 
device_id         | 36
level             | 1
lft               | 0
module_id         | 2331
parent_id         | 10475
rght              | 1
tree_id           | 0

And this is the record for the parent:

-[ RECORD 1 ]-----+------------------------------
created           | 2023-07-27 22:53:09.771248+00
last_updated      | 2024-01-29 20:44:12.433466+00
custom_field_data | {}
id                | 10475
name              | Power Tray Slot 0
_name             | Power Tray Slot 00000000
label             | PS0
position          | 0/PT0
description       | 
device_id         | 36
level             | 0
lft               | 2
module_id         | 
parent_id         | 
rght              | 2
tree_id           | 0

Coincidentally, every row (except the new one) in the dcim_modulebay table now has the following values, presumably because they have the same tree_id value:

level             | 0
lft               | 2
module_id         | 
parent_id         | 
rght              | 2
tree_id           | 0

Here are the exact steps that I just followed to re-verify:

1. Spin up clean postgres:13-alpine container with Docker Compose
2. Load database dump from 3.7.8 server with gunzip -c db_dump.sql.gz | docker compose exec -T postgres sh -c 'psql -U $POSTGRES_USER $POSTGRES_DB'
3. Check parent database record with echo 'SELECT * FROM dcim_modulebay WHERE id = 10475\x\g\x' | docker compose exec -T postgres sh -c 'psql -U $POSTGRES_USER $POSTGRES_DB'
4. git clone https://github.com/netbox-community/netbox.git
5. cd netbox
6. git fetch
7. git checkout v4.0.11
8. ./upgrade.sh
9. cd -
10. Check parent database record with echo 'SELECT * FROM dcim_modulebay WHERE id = 10475\x\g\x' | docker compose exec -T postgres sh -c 'psql -U $POSTGRES_USER $POSTGRES_DB'
11. cd -
12. git checkout v4.1.0
13. ./upgrade.sh
14. cd -
15. Check parent database record with echo 'SELECT * FROM dcim_modulebay WHERE id = 10475\x\g\x' | docker compose exec -T postgres sh -c 'psql -U $POSTGRES_USER $POSTGRES_DB'
16. cd -
17. Start NetBox development server with venv/bin/python3 netbox/manage.py runserver
18. Add new module bay in browser
19. Stop NetBox development server
20. cd -
21. Check parent database record with echo 'SELECT * FROM dcim_modulebay WHERE id = 10475\x\g\x' | docker compose exec -T postgres sh -c 'psql -U $POSTGRES_USER $POSTGRES_DB'
22. Check child database record with echo 'SELECT * FROM dcim_modulebay WHERE id = 60803\x\g\x' | docker compose exec -T postgres sh -c 'psql -U $POSTGRES_USER $POSTGRES_DB'

[DanSheps]

I also cannot recreate this issue using the original steps (the steps above are just building a docker instance then installing one module bay, which is not helpful)

[arthanson]

@sleepinggenius2 we are still not able to reproduce this and the docker steps are not helpful, just include the steps in NetBox that are needed. Remember to provide detailed steps that someone else can follow using a clean installation of NetBox to reproduce the issue. Remember to include the steps taken to create any initial objects or other data.

[sleepinggenius2]

Ok, let's try this:

1. Start with NetBox 4.0.11 and a clean database
2. Load https://raw.githubusercontent.com/netbox-community/netbox-demo-data/master/sql/netbox-demo-v4.0.sql into the database
3. Start NetBox
4. Navigate to /dcim/devices/96/module-bays/
5. Verify module bays Slot 0 to Slot 13 are present and modules are populated in Slot 0 and Slot 1
6. Stop NetBox
7. Upgrade NetBox to 4.1.0
8. Check Slot 0 in the database (SELECT * FROM dcim_modulebay WHERE id = 1) and observe tree_id=0, lft=0, right=0, should be tree_id=<unique value>, lft=1, right=2
9. Check Slot 1 in the database (SELECT * FROM dcim_modulebay WHERE id = 2) and observe tree_id=0, lft=0, right=0, should be tree_id=<unique value>, lft=1, right=2
10. Start NetBox
11. Navigate to /dcim/devices/96/module-bays/
12. Verify module bays Slot 0 through Slot 13 are present and modules are populated in Slot 0 and Slot 1
13. Click on + Add Module Bays or + Add Components > Module Bays
14. In the Module field, select Slot 0: EX9200-32XS (1)
15. In the Name field, enter Subslot 0/0
16. Click Create
17. Observe the newly created module bay Subslot 0/0 has the correct depth, but is at the top of the list and not nested under Slot 0, as seen in the screenshot below
18. Check Slot 0 in the database (SELECT * FROM dcim_modulebay WHERE id = 1) and observe tree_id=0, lft=2, right=2, should be tree_id=<unique value>, lft=1, right=4
19. Check Slot 1 in the database (SELECT * FROM dcim_modulebay WHERE id = 2) and observe tree_id=0, lft=2, right=2, should be unchanged
20. Check Subslot 0/0 in the database (SELECT * FROM dcim_modulebay WHERE id = 29) and observe tree_id=0, lft=0, right=1, should be tree_id=<unique value, same as Slot 0>, lft=2, right=3

It looks like there are actually two issues here. First, is that the default value for lft and rght have been set to 0 instead of 1 and 2 respectively, like was done in https://github.com/netbox-community/netbox/blob/v2.8.0/netbox/dcim/migrations/0101_nested_rackgroups.py or https://github.com/netbox-community/netbox/blob/v2.8.0/netbox/tenancy/migrations/0007_nested_tenantgroups.py. Second, is the missing rebuild migration, like was done in https://github.com/netbox-community/netbox/blob/v2.8.0/netbox/dcim/migrations/0102_nested_rackgroups_rebuild.py or https://github.com/netbox-community/netbox/blob/v2.8.0/netbox/tenancy/migrations/0008_nested_tenantgroups_rebuild.py. I can confirm that changing those values in netbox/dcim/migrations/0190_nested_modules.py (lft: default 0 -> 1, rght: default 0 -> 2) and adding the code below to netbox/dcim/migrations/0191_nested_modules_rebuild.py before running the 4.1.0 migration fixes this issue.

from django.db import migrations


def rebuild_mptt(apps, schema_editor):
    ModuleBay = apps.get_model('dcim', 'ModuleBay')
    for i, modulebay in enumerate(ModuleBay.objects.all(), start=1):
        ModuleBay.objects.filter(pk=modulebay.pk).update(tree_id=i)


class Migration(migrations.Migration):

    dependencies = [
        ('dcim', '0190_nested_modules'),
    ]

    operations = [
        migrations.RunPython(
            code=rebuild_mptt,
            reverse_code=migrations.RunPython.noop
        ),
    ]

[DanSheps]

Unfortunately we still cannot reproduce any error with your steps. You are describing a cause but not giving us the actual problem. Could you please provide a way for us to replicate this issue on a fresh database?

[sleepinggenius2]

Edit: I saw that NetBox 4.1.1 was released while I was writing this and have confirmed that the issue is still present in that release as well.

The problem is that the MPTT structure in the database is not correct, which also causes incorrect rendering in the UI. Both RackGroups and Tenants were migrated to MPTT in NetBox 2.8.0 and used different migration steps than was done for ModuleBays in NetBox 4.1.0. I thought using the demo dataset would be sufficient for the initial database population, but I have included steps below for creating all the necessary objects through the UI from a clean database. I used a brand new database and a brand new clone of the git repo for the steps below, so I don't know what else to do, if you are still not able to see the same thing I am.

1. Start with completely clean NetBox 4.0.11 environment (git checkout v4.0.11; ./upgrade.sh)
2. Navigate to Organization > Sites > Add
3. Enter Name = Test, Slug = test
4. Click Create
5. Navigate to Devices > Device Roles > Add
6. Enter Name = Test, Slug = test
7. Click Create
8. Navigate to Devices > Manufacturers > Add
9. Enter Name = Test, Slug = test
10. Click Create
11. Navigate to Devices > Device Types > Add
12. Enter Manufacturer = Test, Model = Test, Slug = test
13. Click Create
14. Navigate to Devices > Module Types > Add
15. Enter Manufacturer = Test, Model = Test
16. Click Create
17. Navigate to Devices > Devices > Add
18. Enter Name = Test, Device role = Test, Device type = Test, Site = Test
19. Click Create
20. Click + Add Components > Module Bays
21. Enter Name = Slot [0-5]
22. Click Create
23. On Slot 0, click Install module
24. Enter Module type = Test
25. Click Create
26. On Slot 1, click Install module
27. Enter Module type = Test
28. Click Create
29. Database should look like this:

psql (13.16)
Type "help" for help.

netbox=# SELECT * FROM dcim_modulebay;
 id |            created            |         last_updated          | custom_field_data |  name  |     _name     | label | position | description | device_id 
----+-------------------------------+-------------------------------+-------------------+--------+---------------+-------+----------+-------------+-----------
  1 | 2024-09-12 18:29:58.161728+00 | 2024-09-12 18:29:58.161738+00 | {}                | Slot 0 | Slot 00000000 |       |          |             |         1
  2 | 2024-09-12 18:29:58.180585+00 | 2024-09-12 18:29:58.180593+00 | {}                | Slot 1 | Slot 00000001 |       |          |             |         1
  3 | 2024-09-12 18:30:31.815205+00 | 2024-09-12 18:30:31.815219+00 | {}                | Slot 2 | Slot 00000002 |       |          |             |         1
  4 | 2024-09-12 18:30:31.835268+00 | 2024-09-12 18:30:31.835276+00 | {}                | Slot 3 | Slot 00000003 |       |          |             |         1
  5 | 2024-09-12 18:30:31.848436+00 | 2024-09-12 18:30:31.848443+00 | {}                | Slot 4 | Slot 00000004 |       |          |             |         1
  6 | 2024-09-12 18:30:31.861349+00 | 2024-09-12 18:30:31.861356+00 | {}                | Slot 5 | Slot 00000005 |       |          |             |         1
(6 rows)

netbox=# SELECT * FROM dcim_module;
 id |            created            |         last_updated          | custom_field_data | local_context_data | serial | asset_tag | comments | device_id | module_bay_id | module_type_id | description | status 
----+-------------------------------+-------------------------------+-------------------+--------------------+--------+-----------+----------+-----------+---------------+----------------+-------------+--------
  1 | 2024-09-12 18:31:18.922154+00 | 2024-09-12 18:31:18.922169+00 | {}                |                    |        |           |          |         1 |             1 |              1 |             | active
  2 | 2024-09-12 18:31:25.944087+00 | 2024-09-12 18:31:25.944102+00 | {}                |                    |        |           |          |         1 |             2 |              1 |             | active
(2 rows)

30. Upgrade to NetBox 4.1.0 (git checkout v4.1.0; ./upgrade.sh)
31. Database should look like this:

psql (13.16)
Type "help" for help.

netbox=# SELECT * FROM dcim_modulebay;
 id |            created            |         last_updated          | custom_field_data |  name  |     _name     | label | position | description | device_id | level | lft | module_id | parent_id | rght | tree_id 
----+-------------------------------+-------------------------------+-------------------+--------+---------------+-------+----------+-------------+-----------+-------+-----+-----------+-----------+------+---------
  1 | 2024-09-12 18:29:58.161728+00 | 2024-09-12 18:29:58.161738+00 | {}                | Slot 0 | Slot 00000000 |       |          |             |         1 |     0 |   0 |           |           |    0 |       0
  2 | 2024-09-12 18:29:58.180585+00 | 2024-09-12 18:29:58.180593+00 | {}                | Slot 1 | Slot 00000001 |       |          |             |         1 |     0 |   0 |           |           |    0 |       0
  3 | 2024-09-12 18:30:31.815205+00 | 2024-09-12 18:30:31.815219+00 | {}                | Slot 2 | Slot 00000002 |       |          |             |         1 |     0 |   0 |           |           |    0 |       0
  4 | 2024-09-12 18:30:31.835268+00 | 2024-09-12 18:30:31.835276+00 | {}                | Slot 3 | Slot 00000003 |       |          |             |         1 |     0 |   0 |           |           |    0 |       0
  5 | 2024-09-12 18:30:31.848436+00 | 2024-09-12 18:30:31.848443+00 | {}                | Slot 4 | Slot 00000004 |       |          |             |         1 |     0 |   0 |           |           |    0 |       0
  6 | 2024-09-12 18:30:31.861349+00 | 2024-09-12 18:30:31.861356+00 | {}                | Slot 5 | Slot 00000005 |       |          |             |         1 |     0 |   0 |           |           |    0 |       0
(6 rows)

netbox=# SELECT * FROM dcim_module;
 id |            created            |         last_updated          | custom_field_data | local_context_data | serial | asset_tag | comments | device_id | module_bay_id | module_type_id | description | status 
----+-------------------------------+-------------------------------+-------------------+--------------------+--------+-----------+----------+-----------+---------------+----------------+-------------+--------
  1 | 2024-09-12 18:31:18.922154+00 | 2024-09-12 18:31:18.922169+00 | {}                |                    |        |           |          |         1 |             1 |              1 |             | active
  2 | 2024-09-12 18:31:25.944087+00 | 2024-09-12 18:31:25.944102+00 | {}                |                    |        |           |          |         1 |             2 |              1 |             | active
(2 rows)

32. Navigate to Devices, then select the Test device
33. Click on the Module Bays tab to verify everything still looks correct
34. Click + Add Components > Module Bays or + Add Module Bays
35. Enter Module = Slot 0: Test (1), Name = Subslot 0/0
36. Click Create
37. The tree structure should now look like this in the UI:
    
38. And the database should now look like this:

psql (13.16)
Type "help" for help.

netbox=# SELECT * FROM dcim_modulebay;
 id |            created            |         last_updated          | custom_field_data |    name     |           _name           | label | position | description | device_id | level | lft | module_id | parent_id | rght | tree_id 
----+-------------------------------+-------------------------------+-------------------+-------------+---------------------------+-------+----------+-------------+-----------+-------+-----+-----------+-----------+------+---------
  1 | 2024-09-12 18:29:58.161728+00 | 2024-09-12 18:29:58.161738+00 | {}                | Slot 0      | Slot 00000000             |       |          |             |         1 |     0 |   2 |           |           |    2 |       0
  2 | 2024-09-12 18:29:58.180585+00 | 2024-09-12 18:29:58.180593+00 | {}                | Slot 1      | Slot 00000001             |       |          |             |         1 |     0 |   2 |           |           |    2 |       0
  3 | 2024-09-12 18:30:31.815205+00 | 2024-09-12 18:30:31.815219+00 | {}                | Slot 2      | Slot 00000002             |       |          |             |         1 |     0 |   2 |           |           |    2 |       0
  4 | 2024-09-12 18:30:31.835268+00 | 2024-09-12 18:30:31.835276+00 | {}                | Slot 3      | Slot 00000003             |       |          |             |         1 |     0 |   2 |           |           |    2 |       0
  5 | 2024-09-12 18:30:31.848436+00 | 2024-09-12 18:30:31.848443+00 | {}                | Slot 4      | Slot 00000004             |       |          |             |         1 |     0 |   2 |           |           |    2 |       0
  6 | 2024-09-12 18:30:31.861349+00 | 2024-09-12 18:30:31.861356+00 | {}                | Slot 5      | Slot 00000005             |       |          |             |         1 |     0 |   2 |           |           |    2 |       0
  7 | 2024-09-12 18:35:52.940041+00 | 2024-09-12 18:35:52.940067+00 | {}                | Subslot 0/0 | Subslot 00000000/00000000 |       |          |             |         1 |     1 |   0 |         1 |         1 |    1 |       0
(7 rows)

netbox=# SELECT * FROM dcim_module;
 id |            created            |         last_updated          | custom_field_data | local_context_data | serial | asset_tag | comments | device_id | module_bay_id | module_type_id | description | status 
----+-------------------------------+-------------------------------+-------------------+--------------------+--------+-----------+----------+-----------+---------------+----------------+-------------+--------
  1 | 2024-09-12 18:31:18.922154+00 | 2024-09-12 18:31:18.922169+00 | {}                |                    |        |           |          |         1 |             1 |              1 |             | active
  2 | 2024-09-12 18:31:25.944087+00 | 2024-09-12 18:31:25.944102+00 | {}                |                    |        |           |          |         1 |             2 |              1 |             | active
(2 rows)

39. You can see that there is both a visual issue, as well as an issue with the underlying data in the database, as the module bay Slot 0 should have lft = 1, rght = 4, Subslot 0/0 should have lft = 2, rght = 3, and those values in the other module bays should have been unchanged.

[arthanson]

@sleepinggenius2 thank you for the great reproduction steps, I can reproduce it now with the steps.

[arthanson]

@sleepinggenius2 can you please try PR #17553 to see If that fixes your issue?

[sleepinggenius2]

@sleepinggenius2 can you please try PR #17553 to see If that fixes your issue?

I tested against the 4.1.1 database after the steps above and adding the new migration when doing the upgrade from 4.0.11 to 4.1.1 and both resulted in a correct database and UI representation. This seems like an optimal solution for both users that are newly upgrading and fixing the state for anyone that has already upgraded, so a win-win in my book. Thanks!