Specify related objects by attribute during CSV import. Case #3147

[kevintedder]

Issue #3147 answers my needs as I have duplicate named devices. However when I follow the details in this case it does not work.

The CSV file is follows:
address,status,virtual_machine.site,virtual_machine.name,interface
192.168.1.1/30,active,SITE_A,Server_1,Net1

Error message:
Row 1: Expected 4 columns but found 5

If I add another field
address,status,virtual_machine.site,virtual_machine.name,interface,dns_name
192.168.1.1/30,active,SITE_A,Server_1,Net1

Error message:
Row 1 virtual_machine: Object not found
Row 1 interface : Object not found

Using v3.3.2

What am I doing wrong?

[mtinberg]

virtual_machine.site and virtual_machine.name are not column names, I believe this is being interpreted as virtual_machine column twice, which is why it is seen a 4 columns and not 5. You probably want "site" and "name", but the import page lists the expected field names. — Mark Tinberg ***@***.***> Division of Information Technology-Network Services University of Wisconsin-Madison

…

________________________________ From: kevintedder ***@***.***> Sent: Wednesday, March 15, 2023 11:52 AM To: netbox-community/netbox ***@***.***> Cc: Subscribed ***@***.***> Subject: [netbox-community/netbox] Specify related objects by attribute during CSV import. Case #3147 (Discussion #11988) Issue #3147<#3147> answers my needs as I have duplicate named devices. However when I follow the details in this case it does not work. The CSV file is follows: address,status,virtual_machine.site,virtual_machine.name,interface 192.168.1.1/30,active,SITE_A,Server_1,Net1 Error message: Row 1: Expected 4 columns but found 5 If I add another field address,status,virtual_machine.site,virtual_machine.name,interface,dns_name 192.168.1.1/30,active,SITE_A,Server_1,Net1 Error message: Row 1 virtual_machine: Object not found Row 1 interface : Object not found Using v3.3.2 What am I doing wrong? — Reply to this email directly, view it on GitHub<#11988>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAS7UM3DQ6JR35KI6KJ4SI3W4HXTFANCNFSM6AAAAAAV4DNEYQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[candlerb]

Yes it's a weird error message, when clearly there are 5 CSV entries in the header and the row (confirmed here).

#3147 is very old, and I've never imported a CSV file in the way you show; I also don't see any test cases for the format you're using in netbox/ipam/tests/test_views.py - or in netbox/dcim/tests/test_views.py except for one instance of vrf.pk

[kevintedder]

#3147 was included and merged in Github. That would suggest that this feature is now available, unless it was only for DCIM/Devices, not for Virtual Machines (which would be disappointing). I'll just have to enter the interfaces on 30+ VM's manually - :(

[candlerb]

What I was trying to say was:

• the functionality in #3147 was not really defined in the ticket, nor in the documentation.
• there are no test cases for using the way you think it should be used, which I think comes from @jeremystretch's proposal/idea in this comment.
• there were other ideas for how it might work, such as in the initial post on that ticket.

So either it never worked in the way you think; or it did initially, but it broke because there are no tests.

The change has a long PR which completely reworks CSV import, and I can't see how (or if) it is intended to identify arbitrary related objects via multiple attributes of that object.

In this commit I can see tests with single columns vrf.name, vrf.pk, vrf.rd, i.e. using one arbitrary attribute to identify a related object.

Furthermore, the data structure returned implies that only a single identifying attribute can be used:

output = (
            {'address': None, 'status': None, 'vrf': 'name'},
                                              ^^^^^^^^^^^^^

i.e. it's not 'vrf': ['name'] or 'vrf': set('name').

EDIT: I suspect that's the source of the problem you see. I theorise that {'virtual_machine': 'name'} has overwritten {'virtual_machine': 'site'} so it thinks there's only one column heading instead of two. If so, I think it should at least report this condition as an error, rather than silently dropping a column. Making it work the way you want would be a significant rewrite.

I do see tests cases with columns "rackgroup" and "rack", or "site" and "device", where you're implicitly constraining the set of racks or devices to those within a given rackgroup or site. But they don't say "device.site" and "device.name", for example.

[candlerb]

Raised as #11990.