fix orphan tagging if 'ignore_unknown_source_object_puning' is set to true

[bb-Ricardo]

Hi Ricardo, I remember that you said that to me, but as soon I saw this issue, I reworked my entire docker deployment 😅
The main problem arise with this condition https://github.com/bb-Ricardo/netbox-sync/blob/main/module/netbox/inventory.py#L314. A simple continue is not enough, the entire process of orphan tagging is skipped.

Maybe a condition like:

if this object has a tag that matches the current loaded source: orphan tagging is allowed.
Otherwise: object is not part of the loaded source so skip.

If I find some time, in the next few days, I will have a look at it and discuss with you a better solution.
Thank you

Originally posted by @rizlas in #176 (comment)

[bb-Ricardo]

Hey @rizlas,

I just pushed a fix to the development branch. Can you try it out and see if it fixes your issue?

thank you

[rizlas]

Hi,
I made a few runs but it needs more tuning. Orphan tagging seems ok but now it's removes "NetBox-synced" tag from item of other sources.

e.g.

Difference

[Previous]
{
    "tags": [
        "NetBox-synced",
        "Source: vSphereBA",
        "Source: vSphereBO",
        "Source: vSphereRM"
    ]
}

[Post]
{
    "tags": [
        "Source: vSphereBA",
        "Source: vSphereBO",
        "Source: vSphereRM"
    ]
}

This is about a device-type that exists in vSphereBA and vSphereRM but not in vSphereBO
For a case like this it should:

• Not add any orphans
• Remove itself if the item does not exist in its source

Expecting this result:

[Expected]
{
    "tags": [
        "NetBox-synced",
        "Source: vSphereBA",
        "Source: vSphereRM"
    ]
}

[bb-Ricardo]

Hi,

thank you. And yes, this was not handled correctly. I pushed another change and would quite happy if you could test it.

Thank you.

[rizlas]

I think you missed some cleanup/renaming in

https://github.com/bb-Ricardo/netbox-sync/blob/development/module/netbox/connection.py#L792

EDIT: with this self.inventory.source_list no exception is raised.

Traceback (most recent call last):
  File "/app/netbox-sync/./netbox-sync.py", line 153, in <module>
    main()
  File "/app/netbox-sync/./netbox-sync.py", line 138, in main
    nb_handler.prune_data()
  File "/app/netbox-sync/module/netbox/connection.py", line 792, in prune_data
    set(this_object.get_tags()).intersection(self.inventory.source_tags_of_disabled_sources)
AttributeError: 'NetBoxInventory' object has no attribute 'source_tags_of_disabled_sources'

[rizlas]

Ok we are close..
Same example above, now I have this.

Difference

[Previous]
{
    "tags": [
        "NetBox-synced",
        "Source: vSphereBA",
        "Source: vSphereBO",
        "Source: vSphereRM"
    ]
}

[Post]
{
    "tags": [
        "NetBox-synced",
        "Source: vSphereBA",
        "Source: vSphereBO",
        "Source: vSphereRM"
    ]
}

"Source: vSphereBO" should be removed because that particular device-type doesn't exist in vSphereBO

[bb-Ricardo]

sorry for all the hassle. Just pushed another commit which should fix the issue with the renamed var.

can you pull and run it again?

[rizlas]

No hassle at all, dont worry. Thank you!

Traceback (most recent call last):
  File "/app/netbox-sync/./netbox-sync.py", line 153, in <module>
    main()
  File "/app/netbox-sync/./netbox-sync.py", line 138, in main
    nb_handler.prune_data()
  File "/app/netbox-sync/module/netbox/connection.py", line 771, in prune_data
    disabled_sources_tags = [x.source_tag for x in self.source_list if getattr(x, "enabled") is False]
AttributeError: 'NetBoxHandler' object has no attribute 'source_list'

[bb-Ricardo]

Thank you for testing. Just pushed another commit. How about that?

[rizlas]

I made a few runs and now it's working fine except for:

• the removal described before but for me it is not a big deal. Usually device-type are manually managed.
• interface vlan assignment
• there are still some actions that triggers a change log without any changes (IP objects), but for this I can open a new issue if you want.

For interface vlan assignment I made three identical run and the behavior was this:

First run: untagged_vlan_id removed
Second run: untagged_vlan_id added
Third run: untagged_vlan_id removed

Note: I have currently disabled vlan sync (disable_vlan_sync = True)

Here is the log: https://privatebin.net/?e22c577bbe499011#CsziMjzPfj4nmUnSvBEWDjEbxyF6YWKKLoh4yQee5hne

EDIT: interface vlan assignment bug is alread discussed in #245 (sorry)

[bb-Ricardo]

So you get the vlan sync issue as well? Would be happy if you could provide me some detailed logs. Would need -l DEBUG3 option set and the example for one device which includes the vmware API output as well. Like three runs. For this yoh need to redirect the output of the script to a log file as the wmware output isn't passed throu the logger.

This would be a great help to find and fix the issue.

Thank you.

Cheers
Ricardo

[rizlas]

I will, with almost a clean vcenter is 60MB output. As soon as I anonymize it I'll share it with you.

[bb-Ricardo]

I made a few runs and now it's working fine except for:

Hi,

just was wondering what was still unresolved.

• the removal described before but for me it is not a big deal. Usually device-type are manually managed.

I somehow didn't get what you described here with device type.

• this device type does not exists in "vSphereBO" and therefore the tag for the source should be removed?

• interface vlan assignment

should be handled in #245

• there are still some actions that triggers a change log without any changes (IP objects), but for this I can open a new issue if you want.

Yes, that would be highly appreciated

[rizlas]

I somehow didn't get what you described here with device type.

* this device type does not exists in "vSphereBO" and therefore the tag for the source should be removed?

Yes correct

[bb-Ricardo]

@rizlas,

I pushed another commit which should remove the tag if it does not belong to the source but has the source tag.

can you test it again? maybe then this whole thread is solved and I could close this issue.

thank you very much for your patients.

[bb-Ricardo]

ping

[rizlas]

Hi, sorry for the late reply. I'm currently pretty busy and can't test it right now. I'll try to come back to you on Monday.

[bb-Ricardo]

Also, sorry. Didn't mean to rush. just let me know about your findings once you found some time to test.
Thank you.

[rizlas]

Hi, sorry for the very late answer. I just tested it. Here is the log.

2023-03-10 15:11:27,988 - INFO: Pruning orphaned data in NetBox
Traceback (most recent call last):
  File "/app/netbox-sync/./netbox-sync.py", line 143, in <module>
    main()
  File "/app/netbox-sync/./netbox-sync.py", line 128, in main
    nb_handler.prune_data()
  File "/app/netbox-sync/module/netbox/connection.py", line 711, in prune_data
    disabled_sources_tags = [x.source_tag for x in self.inventory.source_list if getattr(x, "enabled") is False]
  File "/app/netbox-sync/module/netbox/connection.py", line 711, in <listcomp>
    disabled_sources_tags = [x.source_tag for x in self.inventory.source_list if getattr(x, "enabled") is False]
AttributeError: 'VMWareHandler' object has no attribute 'enabled'

[bb-Ricardo]

Hi,

Sorry for that. This slipped through. I thought I took care of it. Obviously not.

Will fix it soon.

[bb-Ricardo]

Hi,

just pushed another commit with a fix. would you be able to test it again. thank you for your patience.

[rizlas]

Hi,
it's working fine now 👍🏻 😃 Thank you!
Only a little thing, I tried the behavior injecting manually the tag [NetBox-synced] and started sync like this:

    "tags": [
        "NetBox-synced",
        "Source: vSphereBO",
    ]

After the run i got this situation:

    "tags": [
        "NetBox-synced"
    ]

It is caused because I manually injected the tag?

[bb-Ricardo]

1. it only "looks" at items witch have the NetBox-synced tag
2. if the source is defined in your config and this item was not found for this source, then the tag gets removed

[bb-Ricardo]

Thank you for all the testing and patients. Will close this issue.