OMADA SDN plugin

[jokob-sk]

Good afternoon,
Any "easy" way to integrate with omada SDN? this is an example of data that we can pull from omada:

MAC                                   IP                   hostname            switch(port)/SSID(AccessPoint)
50-02-91-01-02-03     192.168.0.153   mickey                     swtichRoom2  (20)
60-DD-8E-01-02-04    192.168.0.226   pluto                        floor2wifi (floorSouth2AP)
FC-AA-14-01-02-05     192.168.0.10    banana                     switchRoom5 (12)

omada also has the notion of "sites" which basically divides networks into multiple locations like and the scan is on a per-site basis.

• First and foremost, thank you for NetAlertX, it is a great piece of software and it should be integrated to everything IMHO.
• I have been using tplink omada SDN to manage most of my devices and while fixing a limitation of the tplink code on my own, I cam across this great python omada library which provides very interesting insight from omada sdn in particular the ability to extract clients by MAC addresses mapped to specific omada Switches ports and Access Points SSID... Is there any way to integrate this data (ideally as a plugin) into NetAlertX?

1. install python: sudo apt install python3-pip
2. install the omada python api: pip install tplink-omada-client
3. update the parameters CHANGE_ME values in the scipt below
   omada_username="CHANGEME_mrpotatoe"
   omada_password="CHANGEME_lfdaruiRWGFD335qw324z"
   omada_site="CHANGEME_homesweethome"
   omada_url="https://CHANGEME_omada.mylocaldomain"
   omada -t myomada target --url $omada_url --user $omada_username --password $omada_password --site $omada_site --set-default
   omada clients

Originally posted by @FlyingToto in #707

[jokob-sk]

Hey @FlyingToto,

Let's continue the discussion here.

I've prepared a skeleton for you, here is how you load the plugin:

1. Fork the latest code and run sudo docker-compose --env-file ../.env in the NetAlertX location with your provided .env file (I keep it in the parent directory).
2. Load the plugin (refresh the cache with 🔃):
   
3. You then should have the settings exposed like this:
   
4. In the file front/plugins/omada_sdn_imp/omada_sdn.py you will find a rough boiler plate.

• have a look at other plugins for inspiration if needed, e.g. front/plugins/nmap_dev_scan/nmap_dev.py

I already added tplink-omada-client into the image, so it should be available via an import statement

Hope this gives you a good starting point. Ask away if you need help. Once the data is collected/processed correctly, it should show up here:

I've added more system columns, but they are not yet visible (SItename, SSID), and not sure if I didn't break something, but you should be OK working on the files in the front/plugins/omada_sdn_imp/ independently.

To answer your previous questions:

1. can I use the device PORT field to store the SSID string name ?
   

   -> omada nicely provides the port for each switch that a device is connected to and I netalertX does allow me to track the port number per device. But for Access points, omada provides the SSID of the AP that the device is connected to...

2 since I can populate the switch/AP name per device and the port/SSID, will that build the network graph automatically for me?
(that would be so cool! I better buy the older tplink switches omada compatible on ebay before people learn about that!)

3. I cant remember if someone already asked, but any places to track VLANs in netalertX? or maybe even to allow custom fields? (for instance omada's api provide more details like PoE and/or network speed etc...) here is an example:
   

1. I've created a separate cur_SSID and dev_SSID field -> Check the provided boilerplate for details.
2. If the cur_NetworkNodeMAC is populated with the device a node/device is connected to then yes, the Network map might be autopopulated - I didn't test this yet though.
3. Custom fields are not supported, I could add it though in the future - let's focus on the core functionality first.

[FlyingToto]

ok I will work on it on monday.
1 ) why don't you want to store the SSID name in the existing port field? (just curious)
2) " with your provided .env file" so basically this will allow me to run a second docker instance of netlartx from the new folder I pulled correct?
I can just do a git pull in a new directoy, change the port and the origin locations of the /app/* mounts in the docker-compose file and run this "omada" version in parallel to my existing one? correct?

[jokob-sk]

Thanks, looking forward!

1. There is already functionality attached to that field (e.g. network view) which might break. Also, there is this paradigm in development - you either have complex objects or complex code. I rather have complex objects, than hard-to-maintain code where possible, and introducing conditional display and handling of the port data sounds like a headache :)

• ✍EDIT But I'm not a network expert (I know, ironic) so if it makes sense to re-use the PORT field, I'm happy to adjust.

3. Yes, correct, theoretically the only files you should be changing are in that one folder. I'm running I think 3 instances right now with different ports (one which is always freshly pulled from the -dev image to check for any bugs that have to do with a clean setup (using the ALWAYS_FRESH_INSTALL env variable + watchtower). These instances have different docker-compose.yml and .env files, but your setup might differ.

• My dev instance uses:
  • the supplied docker-compose.yml file
  • a custom .env file that I supply to the docker-compose command ( sudo docker-compose --env-file path_to_custom/.env) that overrides the variables ${DEV_LOCATION}, ${APP_DATA_LOCATION}, ${TZ}, ${PORT}

Hope this helps.

[jokob-sk]

The more I think about it the more it makes sense to store SSID in the port field...

[FlyingToto]

The more I think about it the more it makes sense to store SSID in the port field...

one thing a bit different is that on a switch port you only have 1 device, but on an Access Point SSID you can have multiple....

[jokob-sk]

Hummm, just one FYI, the port is stored on the device that connects to that port, so if you store the SSID it's stored on the Device that connects to the SSID, not the router/AP. Maybe storing the current SSID of the device is sufficient. ANyway, those are implementation details and we can fine tune it once the main import is working :)

[FlyingToto]

so... I tried to bring up my second instance but docker-compose is not happy because I already have another instance under the same name (I did change the port numbers and folder structures). is there a way to specify a different name?

this is the actual alert:

[FlyingToto]

so... I tried to bring up my second instance but docker-compose is not happy because I already have another instance under the same name (I did change the port numbers and folder structures). is there a way to specify a different name?

this is the actual alert:

ok I figured it out, I updated the docker-compose.yml file to container_name: netalertxdev

now I got my second instance running.... will try to add my code to it...
I

[jokob-sk]

@FlyingToto great job!

I have the following setup:

Download the code:

cd /development && git clone https://github.com/jokob-sk/NetAlertX.git

touch /development/.env_dev && sudo nano /development/.env_dev

#--------------------------------
#NETALERTX
#--------------------------------
TZ=Europe/Berlin
PORT=22222
DEV_LOCATION=/development/NetAlertX
APP_DATA_LOCATION=/volume/docker_appdata
APP_CONFIG_LOCATION=/volume/

I create a folder netalertx in the APP_DATA_LOCATION with 2 subfolders db and config.

Lastly I run the container:

cd /development/NetAlertX && sudo docker-compose --env-file ../.env_dev

You can then modify the python script without restarting/rebuilding the container every time. Additionally, you can run the script via the UI:

[FlyingToto]

yep all good now... just trying to figure out the python calls...

[jokob-sk]

Just FYI, maybe you'll find this useful:

Removing the container and image

A command to stop, remove the container and the image (replace netalertx and netalertx-netalertx with the appropriate values)

• sudo docker container stop netalertx ; sudo docker container rm netalertx ; sudo docker image rm netalertx-netalertx

Restart hanging python script

SSH into the container and kill & restart the main script loop

• sudo docker exec -it netalertx /bin/bash
• pkill -f "python /app/server" && python /app/server &

[FlyingToto]

Would it be possible to setup a zoom/discord call? (I am in US timezone, usually I can find time during my afternoons

I am currently able to:

1. Run my dev version of netalerx container in parallel to my normal one, with the ability to modify and run the OMDSDN plugin.
2. Run the omada scripts from within the plugin function and extract the output into a list of devices split into a list of tokens.

I need a bit of help:

3. to understand the bit about the json table format that you use.

4. From a logical point of view, I think what I need to do is:
   4.a. Parse each odevice (Omada Device) one by one.
   4.b. Check if the MAC is missing from NetAlertX in which case I should create the Ndevice (NetAlertX Device)
   4.c. Update the Ndevice matching the odevice MAC with the fields from omada

Example of my current tokens:

10:24:04 [<unique_prefix>] token: "['C2-52-E7-F3-A1-1E', '192.168.0.196', 'Pixel-8', 'pantry12', '(48)']"
10:24:04 [<unique_prefix>] token: "['00-E2-59-00-A0-8E', '192.168.0.1', 'bastion', 'office24', '(23)']"
10:24:04 [<unique_prefix>] token: "['E6-C6-42-C2-C7-07', '192.168.0.189', 'E6-C6-42-C2-C7-07', 'froggies3', '(ompapaoffice)']"

4.c.i. Field1 MAC -> no updates since it is used for the lookup.

4.c.ii. Field2 IP -> overwrite (I assume omada would always know better, however what if someone runs multiple IPs on the same MAC? )

4.c.iii. Field3 Hostname -> overwrite unless O_Hostname matches MAC and existing Ndevice_Hostname is not blank

4.c.iv. Field5
4.c.iv.1. If Field5 matches (number) -> overwrite Ndevice_Port_number & Field4_is_SwitchName
4.c.iv.2. Else -> overwrite Ndevice_Parent_Network_Node_MAC & Field4_is_SSID

4.c.v. Field4
4.c.v.1. For Field4_is_SwitchName -> overwrite Ndevice_Parent_Network_Node_MAC
4.c.v.2. For Field4_is_SSID -> overwrite Ndevice_SSID

[jokob-sk]

Sounds like good progress. Happy to chat, can you send an email to jokob.sk@gmail.com and we can set something up 👍

In short however, you don't have to worry about when to overwrite device info. If you map the fields in this step, the application automatically does the rest.

NetAlertX/front/plugins/omada_sdn_imp/omada_sdn.py

Line 54 in 5514cf0

for device in device_data:

Currently most of the fields are overwritten with new information, as far as I remember. Exceptions include Device name and I think vendor. Everything else is overwritten if a value is passed from a plugin.

Here is the code responsible for updating devices (I don't think you need to modify this part, just as a FYI)

NetAlertX/server/device.py

Line 243 in 5514cf0

def update_devices_data_from_scan (db):

[FlyingToto]

ok so it is moving a bit....

• realized that you need the MAC of the parent switch not the name for a given device, so I added a separate call to pull the and index the switches/AP by MAC and replace their names in the list.
• handled reordering switch/port vs ssid/acccesspoint.

 tokens:           "['CC-50-E3-00-01-02', '192.168.0.203', 'alicePC', 'mySSID', '(myaccesspoint)']"
 tokens_reordered: "['CC-50-E3-00-01-02', '192.168.0.203', 'alicePC', '40-AE-30-06-07-08', 'mySSID']"

questions:

1. should we store the SSID in watched4? or just reuse watched3?

                    watched2    = device['some_id'],    #  PARENT NETWORK NODE MAC
                    watched3    = device['some_id'],    #  PORT
                    watched4    = device['some_id'],    #  SSID

3. when an omada-controlled swtich is connected to another non-omada switch itself connecte to multiple devices, they all show up under the same port. I assume it won't break anything even though you would have multiple devices all connected to a single port?
example below, 3 clients all appeared to be connected to a single port=48 on switch=pantry12 because one of them is a switch not controlled by omada:

CC-50-E3-00-01-02 192.168.0.203   alicePC       mySSID (myaccesspoint)
3E-3E-42-00-01-03 192.168.0.186   3E-3E-42-00-01-03     pantry12 (48)
C4-6E-7B-00-01-04 192.168.0.160   C4-6E-7B-00-01-04     pantry12 (48)
E0-3F-49-00-01-05 192.168.0.32    BobPC             pantry12 (48)

[jokob-sk]

Great to hear that! :)

1. Let's try to use watched3 and see if something breaks.
2. That shouldn't be an issue. The port number doesn't have to be unique

[FlyingToto]

hum... so it is not quite working yet...

1. not sure what extra and foreingKey were exactly, I dont see them in the config.json, so I just added the MAC address in foreignKey and the omada_site parameter in extra.
2. I am able to parse my devices properly (ie 1 device = "['18-C0-4D-06-F2-3C', '192.168.0.172', 'giga', 'D8-07-B6-71-FF-7F', '10']") however, once it calls plugin_objects.add_object(), it looks like each field is added separately. so I end up with a huge number of "devices" which only have 1 field populated... should add.object() take 1 list instead of 8 fields?

note: will email you the logs , script and last_result directly...

here is the main loop code:

    if len(device_data) > 0:

        # insert devices into the lats_result.log 
        # make sure the below mapping is mapped in config.json, for example: 
        #"database_column_definitions": [
        # {
        #   "column": "Object_PrimaryID",                 <--------- the value I save into primaryId
        #   "mapped_to_column": "cur_MAC",                <--------- gets unserted into the CurrentScan DB table column cur_MAC
        #  watched1    = 'null' ,
        #  figure a way to run my udpate script delayed

        for device in device_data:
                mylog('verbose', [f'[{pluginName}] main parsing device: "{device}"'])
                if device[PORT_SSID].isdigit():
                    myport = device[PORT_SSID]
                    myssid = 'null'
                else:
                    myssid = device[PORT_SSID]
                    myport = 'null'
                plugin_objects.add_object(
                    primaryId   = device[MAC],    #  MAC
                    secondaryId = device[IP],    #  IP
                    watched1    = device[NAME],    #  NAME/HOSTNAME
                    watched2    = device[SWITCH_AP],    #  PARENT NETWORK NODE MAC
                    watched3    = myport,    #  PORT
                    watched4    = myssid,    #  SSID
                    extra       = omada_site,    #  SITENAME (cur_NetworkSite) or VENDOR (cur_Vendor) (PICK one and adjust config.json -> "column": "Extra")
                    foreignKey  = device[MAC])    #  usually MAC
        mylog('verbose', [f'[{pluginName}] New entries: "{len(device_data)}"'])

    # log result
    plugin_objects.write_result_file()

[jokob-sk]

Releasing -> closing