Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mikrotik RouterOS Parser #244

Merged
merged 7 commits into from
May 11, 2023
Merged

Mikrotik RouterOS Parser #244

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e6db422
Starting adding routeros
h4ndzdatm0ld Mar 11, 2023
ef6e109
Basic RouterOS Parser capabilities
h4ndzdatm0ld Mar 21, 2023
319ff2b
Mappers
h4ndzdatm0ld Mar 21, 2023
14ccd4d
feedback: Parsing tests for system note
h4ndzdatm0ld Apr 1, 2023
2db3e58
Merge branch 'networktocode:develop' into feature/parsers/mikrotik-ros
h4ndzdatm0ld Apr 12, 2023
f28525a
if it looks like a duck, banners like a banner, then..
h4ndzdatm0ld Apr 14, 2023
4f0ac6f
banner_end satisfied, but rely on is_banner_end
h4ndzdatm0ld Apr 14, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/dev/include_parser_list.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,5 +13,6 @@
| fortinet_fortios | netutils.config.parser.FortinetConfigParser |
| juniper_junos | netutils.config.parser.JunosConfigParser |
| linux | netutils.config.parser.LINUXConfigParser |
| mikrotik_routeros | netutils.config.parser.RouterOSConfigParser |
| mrv_optiswitch | netutils.config.parser.OptiswitchConfigParser |
| nokia_sros | netutils.config.parser.NokiaConfigParser |
1 change: 1 addition & 0 deletions docs/user/lib_mapper/napalm.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,6 @@
| nxos | → | cisco_nxos |
| nxos_ssh | → | cisco_nxos |
| panos | → | paloalto_panos |
| ros | → | mikrotik_routeros |
| sros | → | nokia_sros |
| vyos | → | brocade_vyos |
1 change: 1 addition & 0 deletions docs/user/lib_mapper/napalm_reverse.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,5 +11,6 @@
| fortinet | → | fortios |
| huawei_vrp | → | huawei |
| juniper_junos | → | junos |
| mikrotik_routeros | → | ros |
| nokia_sros | → | sros |
| paloalto_panos | → | panos |
1 change: 1 addition & 0 deletions netutils/config/compliance.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
"aruba_aoscx": parser.ArubaConfigParser,
"mrv_optiswitch": parser.OptiswitchConfigParser,
"extreme_netiron": parser.NetironConfigParser,
"mikrotik_routeros": parser.RouterOSConfigParser,
}

# TODO: Once support for 3.7 is dropped, there should be a typing.TypedDict for this which should then also be used
Expand Down
44 changes: 44 additions & 0 deletions netutils/config/parser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1375,3 +1375,47 @@ class NetironConfigParser(BaseSpaceConfigParser):
def banner_end(self) -> str:
"""Demarcate End of Banner char(s)."""
raise NotImplementedError("Extreme Netiron platform doesn't have a banner.")


class RouterOSConfigParser(BaseSpaceConfigParser):
"""Mikrotik RouterOS config parser."""

comment_chars: t.List[str] = ["#"]
banner_start: t.List[str] = ["/system note set note=", "set note="]

@property
def banner_end(self) -> str:
"""Demarcate End of Banner char(s)."""
raise NotImplementedError("Mikrotik platform uses system note as a banner.")

def is_banner_end(self, line: str) -> bool:
"""Determine if end of banner."""
if line.endswith('"') or line.startswith("/"):
return True
return False

def _build_banner(self, config_line: str) -> t.Optional[str]:
"""Handle banner config lines.

Args:
config_line: The start of the banner (system note) config.

Returns:
The next configuration line in the configuration text or None when banner end is the end of the config text.

Raises:
ValueError: When the parser is unable to identify the End of the Banner.
"""
banner_config = [config_line]
for line in self.generator_config:
if not self.is_banner_end(line):
banner_config.append(line)
else:
banner_config.append(line)
line = "\n".join(banner_config)
self._update_config_lines(line)
try:
return next(self.generator_config)
except StopIteration:
return None
raise ValueError("Unable to parse banner (system note) end.")
2 changes: 2 additions & 0 deletions netutils/lib_mapper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,7 @@
"sros": "nokia_sros",
"vyos": "brocade_vyos",
"aoscx": "aruba_aoscx",
"ros": "mikrotik_routeros",
}

PYNTC_LIB_MAPPER = {
Expand Down Expand Up @@ -224,6 +225,7 @@
"paloalto_panos": "panos",
"nokia_sros": "sros",
"aruba_aoscx": "aoscx",
"mikrotik_routeros": "ros",
}

PYNTC_LIB_MAPPER_REVERSE = {
Expand Down
59 changes: 59 additions & 0 deletions tests/unit/mock/config/compliance/compliance/mikrotik_routeros/routeros_backup.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
# by RouterOS 6.49.6
# software id = WZ00-84LP
#
# model = CCR1036-8G-2S+
# serial number = C6CD0BF7A020
/interface bridge add name=google-vpc-peering-iface
/interface bridge add name=loopback
/interface bonding add mode=802.3ad name=Po1 slaves=sfp-sfpplus1,sfp-sfpplus2
/interface vlan add interface=Po1 name=vlan11-mgmt vlan-id=11
/interface vlan add interface=vlan2933-Transit-to-CC-NNI name=vlan3049-Transit-to-XXX vlan-id=3049
/interface vlan add disabled=yes interface=Po1 name=vlan3051-Transit-to-CSW-through-QoE-Disabled vlan-id=3051
/interface vlan add interface=Po1 mtu=1300 name=vlan3225-Transit-CORE-Agg1-to-COREXXX vlan-id=3225
/interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec peer add address=50.157.100.38/32 exchange-mode=ike2 local-address=55.106.77.11 name=google-vpc-peer
/ip pool add name=BNEdgeLiteTest ranges=192.168.69.254
/ip pool add name=BNEdgeLiteTest2 ranges=192.168.70.254
/ip dhcp-server add address-pool=BNEdgeLiteTest disabled=no interface=ether1 name=dhcp1
/ip dhcp-server add address-pool=BNEdgeLiteTest2 disabled=no interface=ether2 name=dhcp2
/routing bgp instance set default as=1234 router-id=10.127.1.3
/routing ospf instance set [ find default=yes ] router-id=10.127.1.3
/snmp community add addresses=::/0 name=somestringa
/system logging action set 3 bsd-syslog=yes remote=172.16.11.1 remote-port=5140
/user group set full policy=local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,password,web,sniff,sensitive,api,romon,dude,tikapp
/user group add name=prom policy=ssh,read,winbox,api,!local,!telnet,!ftp,!reboot,!write,!policy,!test,!password,!web,!sniff,!sensitive,!romon,!dude,!tikapp
/ip neighbor discovery-settings set discover-interface-list=!dynamic
/ip address add address=10.0.11.13/24 interface=vlan11-mgmt network=10.0.11.0
/ip address add address=192.168.69.1/24 interface=ether1 network=192.168.69.0
/ip dhcp-server network add address=192.168.69.0/24 gateway=192.168.69.1
/ip dhcp-server network add address=192.168.70.0/24 gateway=192.168.70.1
/ip dns set servers=8.8.8.8
/ip firewall address-list add address=34.157.17.38 list=whitelist
/ip firewall address-list add address=72.202.79.109 list=google-vpc-acl
/ip firewall address-list add address=34.157.17.38 list=google-vpc-acl
/ip firewall filter add action=drop chain=input dst-address=50.106.77.11 src-address-list=!google-vpc-acl
/ip firewall nat add action=dst-nat chain=dstnat dst-address=50.106.77.11 dst-port=443 protocol=tcp to-addresses=10.1.15.10
/ip firewall nat add action=masquerade chain=srcnat out-interface=vlan12-Servers
/ip firewall nat add action=src-nat chain=srcnat src-address=50.64.0.75 to-addresses=55.106.77.11
/ip ipsec identity add peer=google-vpc-peer secret=*****
/ip ipsec policy set 0 disabled=yes
/ip ipsec policy add disabled=yes dst-address=169.254.1.1/32 peer=google-vpc-peer src-address=169.254.1.2/32 tunnel=yes
/ip ipsec policy add disabled=yes dst-address=10.150.0.0/20 peer=google-vpc-peer src-address=0.0.0.0/0 tunnel=yes
/ip service set telnet disabled=yes
/ip service set ftp disabled=yes
/ip service set www disabled=yes
/ip service set api address=10.1.15.5/32,10.120.16.0/20,172.16.11.0/24
/ip service set api-ssl disabled=yes
/routing bgp network add network=55.106.77.11/32 synchronize=no
/routing bgp network add network=55.106.77.12/32 synchronize=no
/routing bgp peer add in-filter=ibgp-allow-default-in name=CSW out-filter=ibgp-no-default-out remote-address=10.127.1.10 remote-as=12345 update-source=loopback
/routing filter add action=discard chain=ibgp-no-default-out prefix=10.127.1.0/24
/routing filter add action=accept chain=ibgp-no-default-out prefix=10.64.0.0/10 prefix-length=22-32
/routing ospf interface add dead-interval=8s hello-interval=2s interface=vlan3049-Transit-to-85Presidential network-type=point-to-point
/routing ospf interface add cost=11 dead-interval=8s hello-interval=2s interface=vlan3166-Transit-HalseyCore-Agg1-to-Indigo-5 network-type=point-to-point
/routing ospf network add area=backbone network=10.126.0.16/29
/snmp set enabled=yes trap-community=somestring
/system clock set time-zone-name=America/New_York
/system identity set name=ag1.123site.nwk.nj
/system logging add action=remote topics=error
/system logging add action=remote topics=info
5 changes: 5 additions & 0 deletions tests/unit/mock/config/compliance/compliance/mikrotik_routeros/routeros_feature.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
features = [
{"name": "bgp", "ordered": True, "section": ["/routing bgp instance"]},
{"name": "snmp", "ordered": True, "section": ["/snmp"]},
{"name": "ospf-networks", "ordered": True, "section": ["/routing ospf network"]},
]
60 changes: 60 additions & 0 deletions tests/unit/mock/config/compliance/compliance/mikrotik_routeros/routeros_intended.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
# by RouterOS 6.49.6
# software id = WZ00-84LP
#
# model = CCR1036-8G-2S+
# serial number = C6CD0BF7A020
/interface bridge add name=google-vpc-peering-iface
/interface bridge add name=loopback
/interface bonding add mode=802.3ad name=Po1 slaves=sfp-sfpplus1,sfp-sfpplus2
/interface vlan add interface=Po1 name=vlan11-mgmt vlan-id=11
/interface vlan add interface=vlan2933-Transit-to-CC-NNI name=vlan3049-Transit-to-XXX vlan-id=3049
/interface vlan add disabled=yes interface=Po1 name=vlan3051-Transit-to-CSW-through-QoE-Disabled vlan-id=3051
/interface vlan add interface=Po1 mtu=1300 name=vlan3225-Transit-CORE-Agg1-to-COREXXX vlan-id=3225
/interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik
/ip ipsec peer add address=50.157.100.38/32 exchange-mode=ike2 local-address=55.106.77.11 name=google-vpc-peer
/ip pool add name=BNEdgeLiteTest ranges=192.168.69.254
/ip pool add name=BNEdgeLiteTest2 ranges=192.168.70.254
/ip dhcp-server add address-pool=BNEdgeLiteTest disabled=no interface=ether1 name=dhcp1
/ip dhcp-server add address-pool=BNEdgeLiteTest2 disabled=no interface=ether2 name=dhcp2
/routing bgp instance set default as=1234 router-id=10.127.1.3
/routing ospf instance set [ find default=yes ] router-id=10.127.1.3
/snmp community add addresses=::/0 name=somestringa
/system logging action set 3 bsd-syslog=yes remote=172.16.11.1 remote-port=5140
/user group set full policy=local,telnet,ssh,ftp,reboot,read,write,policy,test,winbox,password,web,sniff,sensitive,api,romon,dude,tikapp
/user group add name=prom policy=ssh,read,winbox,api,!local,!telnet,!ftp,!reboot,!write,!policy,!test,!password,!web,!sniff,!sensitive,!romon,!dude,!tikapp
/ip neighbor discovery-settings set discover-interface-list=!dynamic
/ip address add address=10.0.11.13/24 interface=vlan11-mgmt network=10.0.11.0
/ip address add address=192.168.69.1/24 interface=ether1 network=192.168.69.0
/ip dhcp-server network add address=192.168.69.0/24 gateway=192.168.69.1
/ip dhcp-server network add address=192.168.70.0/24 gateway=192.168.70.1
/ip dns set servers=8.8.8.8
/ip firewall address-list add address=34.157.17.38 list=whitelist
/ip firewall address-list add address=72.202.79.109 list=google-vpc-acl
/ip firewall address-list add address=34.157.17.38 list=google-vpc-acl
/ip firewall filter add action=drop chain=input dst-address=50.106.77.11 src-address-list=!google-vpc-acl
/ip firewall nat add action=dst-nat chain=dstnat dst-address=50.106.77.11 dst-port=443 protocol=tcp to-addresses=10.1.15.10
/ip firewall nat add action=masquerade chain=srcnat out-interface=vlan12-Servers
/ip firewall nat add action=src-nat chain=srcnat src-address=50.64.0.75 to-addresses=55.106.77.11
/ip ipsec identity add peer=google-vpc-peer secret=*****
/ip ipsec policy set 0 disabled=yes
/ip ipsec policy add disabled=yes dst-address=169.254.1.1/32 peer=google-vpc-peer src-address=169.254.1.2/32 tunnel=yes
/ip ipsec policy add disabled=yes dst-address=10.150.0.0/20 peer=google-vpc-peer src-address=0.0.0.0/0 tunnel=yes
/ip service set telnet disabled=yes
/ip service set ftp disabled=yes
/ip service set www disabled=yes
/ip service set api address=10.1.15.5/32,10.120.16.0/20,172.16.11.0/24
/ip service set api-ssl disabled=yes
/routing bgp network add network=55.106.77.11/32 synchronize=no
/routing bgp network add network=55.106.77.12/32 synchronize=no
/routing bgp peer add in-filter=ibgp-allow-default-in name=CSW out-filter=ibgp-no-default-out remote-address=10.127.1.10 remote-as=12345 update-source=loopback
/routing filter add action=discard chain=ibgp-no-default-out prefix=10.127.1.0/24
/routing filter add action=accept chain=ibgp-no-default-out prefix=10.64.0.0/10 prefix-length=22-32
/routing ospf interface add dead-interval=8s hello-interval=2s interface=vlan3049-Transit-to-85Presidential network-type=point-to-point
/routing ospf interface add cost=11 dead-interval=8s hello-interval=2s interface=vlan3166-Transit-Core-Agg1-to-SiteXYZ network-type=point-to-point
/routing ospf network add area=backbone network=10.126.0.16/29
/routing ospf network add area=backbone network=10.127.1.3/32
/snmp set enabled=yes trap-community=somestring
/system clock set time-zone-name=America/New_York
/system identity set name=ag1.123site.nwk.nj
/system logging add action=remote topics=error
/system logging add action=remote topics=info
32 changes: 32 additions & 0 deletions tests/unit/mock/config/compliance/compliance/mikrotik_routeros/routeros_received.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{
"bgp": {
"actual": "/routing bgp instance set default as=1234 router-id=10.127.1.3",
"cannot_parse": true,
"compliant": true,
"extra": "",
"intended": "/routing bgp instance set default as=1234 router-id=10.127.1.3",
"missing": "",
"ordered_compliant": true,
"unordered_compliant": true
},
"snmp": {
"actual": "/snmp community add addresses=::/0 name=somestringa\n/snmp set enabled=yes trap-community=somestring",
"cannot_parse": true,
"compliant": true,
"extra": "",
"intended": "/snmp community add addresses=::/0 name=somestringa\n/snmp set enabled=yes trap-community=somestring",
"missing": "",
"ordered_compliant": true,
"unordered_compliant": true
},
"ospf-networks": {
"actual": "/routing ospf network add area=backbone network=10.126.0.16/29",
"cannot_parse": true,
"compliant": false,
"extra": "",
"intended": "/routing ospf network add area=backbone network=10.126.0.16/29\n/routing ospf network add area=backbone network=10.127.1.3/32",
"missing": "/routing ospf network add area=backbone network=10.127.1.3/32",
"ordered_compliant": false,
"unordered_compliant": false
}
}
6 changes: 6 additions & 0 deletions ...it/mock/config/compliance/config_section_not_parsed/mikrotik_routeros/routeros_feature.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
features = [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Always like to have an example of a banner being parsed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey Ken! The comments at the top of the config files, is that considered a banner here, not just comments? There is a system note command, but it just outputs like a normal str config line.

# mar/22/2023 17:44:57 by RouterOS 6.48.5
# software id =
#
#
#
/interface wireless security-profiles set [ find default=yes ] supplicant-identity=MikroTik
/ip address add address=172.31.255.30/30 interface=ether1 network=172.31.255.28
/ip dhcp-client add disabled=no interface=ether1
/system identity set name=R2
/system note set note="SOME CRAZY NOTE!"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, perhaps simpleton of me to presume there is a banner :)

I do think it would be great to have a few options for "note", like how does it handle multi-line (is that supported), special characters, can you have a double quote in there, etc.. Think of ways to break it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@itdependsnetworks I added to the parsing tests to capture the lines that are children of the /system note set note=.. line. This is the exact input that went into the device is below. If a user inputs a string without adding \n the config export will handle it and add it for you, same with special chars. Device will just escape them when exporting config.

This is a \"System Note\" for a Mikrotik router.\n\
It includes double quotes (\") and special characters such as:\n\
@, #, $, %, ^, &, *, (, ), _, +, [, ], {, }, |, ;, ',', ., /, <, >, and ?.\n\
\n\
Remember to escape any special characters with a backslash (\\) when necessary.\n\
\n\
This is a multiline note with several lines of text.

and when doing a terse config export, the following output appears ->

[admin@ag1.123site.nwk.nj] /system note> export terse
# apr/01/2023 17:16:02 by RouterOS 6.48.5
# software id =
#
#
#
/system note set note="This is a \\\"System Note\\\" for a Mikrotik router.\\n\\\
    \nIt includes double quotes (\\\") and special characters such as:\\n\\\
    \n@, #, \$, %, ^, &, *, (, ), _, +, [, ], {, }, |, ;, ',', ., /, <, >, and \?.\\n\\\
    \n\\n\\\
    \nRemember to escape any special characters with a backslash (\\\\) when necessary.\\n\\\
    \n\\n\\\
    \nThis is a multiline note with several lines of text.\\n\\\
    \nWow, what a great example of a note to ensure proper parsing by NetUtils!"
[admin@ag1.123site.nwk.nj] /system note>

hope this helps and captures what you asked for, LMK.

{"name": "routing-filter", "ordered": True, "section": ["/routing filter"]},
{"name": "ospf", "ordered": True, "section": ["/routing ospf"]},
{"name": "snmp", "ordered": True, "section": ["/snmp"]},
{"name": "ip-firewall", "ordered": True, "section": ["/ip firewall"]},
]
5 changes: 5 additions & 0 deletions ...mock/config/compliance/config_section_not_parsed/mikrotik_routeros/routeros_received.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
{
"remaining_cfg": "/system clock set time-zone-name=America/New_York\n/system identity set name=ag1.123site.nwk.nj\n/system logging add action=remote topics=error\n/system logging add action=remote topics=info",

"section_not_found": ["ip-firewall"]
}
10 changes: 10 additions & 0 deletions ...unit/mock/config/compliance/config_section_not_parsed/mikrotik_routeros/routeros_sent.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
/routing filter add action=discard chain=ibgp-no-default-out prefix=10.127.1.0/24
/routing filter add action=accept chain=ibgp-no-default-out prefix=10.64.0.0/10 prefix-length=22-32
/routing ospf interface add dead-interval=8s hello-interval=2s interface=vlan3049-Transit-to-85Presidential network-type=point-to-point
/routing ospf interface add cost=11 dead-interval=8s hello-interval=2s interface=vlan3166-Transit-HalseyCore-Agg1-to-Indigo-5 network-type=point-to-point
/routing ospf network add area=backbone network=10.126.0.16/29
/snmp set enabled=yes trap-community=somestring
/system clock set time-zone-name=America/New_York
/system identity set name=ag1.123site.nwk.nj
/system logging add action=remote topics=error
/system logging add action=remote topics=info
Loading