Inconsistent /api/virtualization/interfaces/ results

[dsanader]

Issue type

[ ] Feature request
[ x ] Bug report
[ ] Documentation

Environment

• Python version: 3.6.5
• NetBox version: 2.3.4

Description

Querying all virtualized interfaces returns inconsistents results : the count is OK, but some interfaces are missing and some are duplicated.

The underlying query is ordering by an empty column ("dcim_device"."name") which seems to fall under the non predictable results case : https://www.postgresql.org/docs/current/static/queries-limit.html

... WHERE "dcim_interface"."virtual_machine_id" IS NOT NULL 
ORDER BY "dcim_device"."name" ASC, "dcim_interface"."name" ASC LIMIT 1000 OFFSET 50

[jeremystretch]

Thank you for taking the time to report a bug. We'll need more information to investigate this issue. Per the contributing guidelines please ensure that you have included all of the following:

• The environment in which NetBox is running
• The exact steps that can be taken to reproduce the issue
• Any error messages generated
• Screenshots (if applicable)

Please update your issue to include all of the information listed above. If no response is received within a week, this issue will be closed. Thanks!

[dsanader]

import pynetbox

for vif in nb.virtualization.interfaces.all():
     print(vif.id)

Duplicate ids are printed.

I guess it requires you to have enough interfaces to get this bug triggered consistently.

I've "fixed" it by changing the ordering in the Interface model (ordering = ['id', 'name'] instead of ordering = ['device', 'name']).

[jeremystretch]

Still can't reproduce this. The queryset which retrieves the interfaces is:

Interface.objects.filter(virtual_machine__isnull=False).select_related('virtual_machine')

Not sure how that could result in duplicate objects being returned. I'll need sample data which triggers the bug to dig any further into this.

[dsanader]

Can I send you a dump of my database by email ?

[jeremystretch]

Sorry, you'll need to derive a sample set from your data which can be used to replicate the problem and post it here.

[jeremystretch]

Closing due to lack of activity.

[dsanader]

Hello Jeremy,

Finally got some time to work on this.

The key is to a have a mixed set of device and vm interfaces and to have a count above the paging size (>1000).

On a fresh 2.5.6 install with the initial data and after loading some dummy devices and vms with the attached script, you should witness the inconsistency :

vifs = [iface.id for iface in nb.virtualization.interfaces.all()]

len(vifs)
600

len(set(vifs))
567

reproduce_2207.py.txt

[jeremystretch]

@dsanader Unless you can recreate this using bare HTTP calls NetBox REST API, you'll need to open this as a bug against the pynetbox library.

[dsanader]

You're hard on me...

Anyway, same outcome with curl :

curl -H "authorization: Token d34db33f" "https://netbox-test.example.com/api/virtualization/interfaces/" > first.json
curl -H "authorization: Token d34db33f" "https://netbox-test.example.com/api/virtualization/interfaces/?limit=600&offset=50"> second.json
jq '.results[] | .id' first.json second.json | sort | uniq -c | wc -l
567

The issues goes away (600 unique ids) as soon as I :

index 3b0c02b2..f9502071 100644
--- a/netbox/virtualization/api/views.py
+++ b/netbox/virtualization/api/views.py
@@ -65,7 +65,7 @@ class VirtualMachineViewSet(CustomFieldModelViewSet):
 class InterfaceViewSet(ModelViewSet):
     queryset = Interface.objects.filter(
         virtual_machine__isnull=False
-    ).select_related('virtual_machine').prefetch_related('tags')
+    ).select_related('virtual_machine').prefetch_related('tags').order_by('id')
     serializer_class = serializers.InterfaceSerializer
     filterset_class = filters.InterfaceFilter

[jeremystretch]

This is almost identical to #2938, which was fixed in v2.5.8, except that the Interface model uses its own manager.

[pm17788]

Tested PR 2996 on my NetBox v 2.5.8 instance:

69         sql_col = '{}.name'.format(self.model._meta.db_table)
70         ordering = [
71             '_slot', '_subslot', '_position', '_subposition', '_type', '_id', '_channel', '_vc', 'name', 'pk'
72         ]
73 

Still the same issue with interfaces not being returned as a unique list.

[pm17788]

Tested on a full 2.5.9, and that works. Probably something I messed up with my test or some such. Huzzah!