Mikrotik version > 6.48 = 'export' command incomplete

[prtomasi]

Description of Issue/Question

Note: Please check https://guides.github.com/features/mastering-markdown/
to see how to properly format your request.

Setup

Netmiko version

(Paste verbatim output from pip freeze | grep netmiko between quotes below)

netmiko==4.2.0

Netmiko device_type (if relevant to the issue)

(Paste device_type between quotes below)

    'device_type': 'mikrotik_routeros',

Steps to Reproduce the Issue

Error Traceback

(Paste the complete traceback of the exception between quotes below)

no error, just incomplete output from 'export' command

Relevant Python code

(Please try to essentialize your Python code to the minimum code needed to reproduce the issue)
(Paste the code between the quotes below)

from netmiko import ConnectHandler

mikrotik = {
    'device_type': 'mikrotik_routeros',
    'host':   '10.1.1.68',
    'port':     2222,
    'username': 'dude',
    'password': 'not_real_password'
}

mikrotik_connection = ConnectHandler(**mikrotik)
print(mikrotik_connection.send_command(f'export', cmd_verify=False))

Test

I noticed netmiko is breaking the output of 'export' command when the Mikrotik RouterOS version is > 6.48

The following was tested with 6.49.8 version
# mar/19/2024 11:25:34 by RouterOS 6.49.8

With the python code above, only 27 lines (from a total of 275 lines) were captured by netmiko, no errors were generated

line 22 --> /interface vlan
line 23 --> add disabled=yes interface=sfp-sfpplus1 name=v450 vlan-id=450
line 24 --> add interface=sfp-sfpplus1 name=v495 vlan-id=495
line 25 --> add interface=sfp-sfpplus1 name=v1904 vlan-id=1904
line 26 --> add interface=sfp-sfpplus1 name=v1906 vlan-id=1906
line 27 --> add interface=sfp-sfpplus1 name=v2056 vlan-id=2056

When I use PuTTY to run 'export' command in the same device, the output is complete and has 275 lines

...
line 28 --> /interface ethernet switch port
...
line 275 --> set default-group=deny use-radius=yes

line 278 --> prompt:

[user@RTR-FW-NNN-NNN-01] >

I asked for help at Mikrotik Forum, but I was ignored:

https://forum.mikrotik.com/viewtopic.php?t=205175

[ktbyers]

I would probably try on your send_command() statement to make it.

import re

# rest of your code
output = mikrotik_connection.send_command(
    f'export', 
    cmd_verify=False, 
    expect_string=re.escape("[user@RTR-FW-NNN-NNN-01] >")
)

Does that help?

[prtomasi]

Hi,

It still didn't work.
Every time I run the script the output has ~20 and ~30 lines (it's not breaking at any specific character or specific amount of lines)

I can't use a fixed expect_string, it must be a regex because the user and hostname will be variable in another complete script, that has a loop (for) to catch the running-config (export) from a list of devices.

[prtomasi]

Hi,

Additional information:

from netmiko import ConnectHandler
import re

mikrotik = {
    'device_type': 'mikrotik_routeros',
    'host':   '10.1.1.68',
    'port':     2222,
    'username': 'dude',
    'password': 'not_real_password'
}

mikrotik_connection = ConnectHandler(**mikrotik)

output = mikrotik_connection.send_command(
    f'export',
    cmd_verify=False,
    expect_string=re.escape("[user@RTR-FW-NNN-NNN-01] >")
)

print(output)
print(f"Command 'export' executed")

the amount of lines captured is not fixed, this time only 5 lines were captured before the command print(f"Command 'export' executed") was applied:

root@ubuntu:/# python3 run.py


# mar/20/2024 17:04:53 by RouterOS 6.49.8
# software id = 74TS-LPJG
#
# model = RB4011iGS+
# serial number = MC927J062DR <<< only 5 lines captured

Command 'export' executed <<< print(f"Command 'export' executed") applied
root@ubuntu:/#

no visible errors generated

[Tonygratta]

I've experienced a similar issue with my MikroTik. The problem seems to be that the router might repaint the input string after a command has been entered. When it causes the router captures the prompt string at the beginning of command execution, and then closes the connection.
Sometimes, Netmiko does manage to capture some output after repainting before exit, but it depends on the performance of the router. I have not seen any output from one of my devices (it gives a rather slow output). Others have shown some output results.
To solve this issue, I suggest using a regex string for search pattern that looks for not just the prompt string, but the prompt string exactly at the end of the output string, without any symbols after it. This prevents netmiko from early exit.
After testing on some real devices, I will send PR.

[ktbyers]

@Tonygratta and @prtomasi I wonder if we can do this?

/terminal set 0 type=dumb

In other words, configure the Mikrotik to be a dumb terminal so that a lot of the very problematic Mikrotik terminal behavior is bypassed.

Can anyone try this?

[ktbyers]

Related to this...it seems like we should be setting the e character here (which sets it to a "dumb" terminal):

https://github.com/ktbyers/netmiko/blob/develop/netmiko/mikrotik/mikrotik_ssh.py#L59

Once again does someone want to test this? Modified like would be:

self.username += "+cet511w4098h"

[Tonygratta]

self.username += "+cet511w4098h"

I've tried this method (adding "e" to the '_modify_connection_params' function), but it doesn't seem to be working. The router's output log slightly differs from the default, so I think the router has really been switched to a dumb mode. However, in both cases, the export doesn't work on my device.

[Tonygratta]

/terminal set 0 type=dumb

My router doesn't accept this command:

bad command name set (line 1 column 11)

Do you have any documentation related to this command? I couldn't find anything.

[prtomasi]

Hi,

The Mikrotik Help has some information about the parameters
https://help.mikrotik.com/docs/display/ROS/Command+Line+Interface

I tried to change 'username': 'dude' to 'username': 'dude' + '+e' but netmiko didn't accept the +e additional information
Result: Pattern not detected: '(?:\\].*>|Do you want to see the software license)' in output.

When using 'username': 'dude' + '+ct511w4098h' netmiko works, but RouterOS still breaks the output (export incomplete)

How do I insert self.username += "+cet511w4098h" to my script?

[ktbyers]

@Tonygratta I should have double checked it. It looks like ChatGPT made it up.

But we have that "e" option...which looks like it is not working anyways.

[ktbyers]

@prtomasi Easiest way is to modify the one line in the Netmiko code, but @Tonygratta already tested it and it didn't work :-(

[Tonygratta]

The more tests, the better.
@prtomasi you could try '+e' anyway. Or you may check the other solution:

import re
#  some code
output = mikrotik_connection.send_command(
    f'export', 
    expect_string=re.escape(mikrotik_connection.find_prompt()) + r'[ \t]*$')
)

[prtomasi]

Hello,

This code seems to be working:

from netmiko import ConnectHandler
import re

mikrotik = {
    'device_type': 'mikrotik_routeros',
    'host':   '10.1.1.68',
    'port':     2222,
    'username': 'dude' + '+cet',
    'password': 'not_real_password'
}

mikrotik_connection = ConnectHandler(**mikrotik)

output = mikrotik_connection.send_command(
    f'export',
    expect_string=re.escape(mikrotik_connection.find_prompt()) + r'[ \t]*$')

print(output)
print(f"#=== End of script ===#")

It seems now netmiko only stops when it sees the prompt [user@RTR-FW-NNN-NNN-01] >
The expect_string=re.escape(mikrotik_connection.find_prompt()) + r'[ \t]*$' regex seems to solve the issue
I ran the script several times and all of them got the complete RouterOS export config

I also added the +cet parameters to username: 'username': 'dude' + '+cet'
They're not mandatory but seems to help netmiko about the prompt issue

"c" = disable console colors ?
"t" = disable auto-detection of terminal capabilities ?
"e" = enables "dumb" terminal mode ?

[Tonygratta]

Great!
As for 'username': 'dude' + '+cet', it is not recommended to add this to the connection parameters, as netmiko will add some extra keys to the login string before the connection is made. The final login name will be dude+cet+ct511w4098h, which is not a valid login string. If you want to experiment with these parameters, you should correct the string self.username += "+ct511w4098h", which can be found in the mikrotik_ssh.py file in the netmiko package.

[amyasnikov]

Tried adding "e" to the login string by modifying this line. Although it helps for short export statements (e.g. /ip/address/export starts working correctly), the full export is still truncated.

In [15]: mkt.send_command('/export')
Out[15]: '\n\n\n# oct/02/2024 01:04:00 by RouterOS 7.1.5\n# software id = RC3X-6E1P\n#\n# model = RB952Ui-5ac2nD\n# serial number = CC3E0D10D0C5\n'

[Tonygratta]

@amyasnikov Did you try expect_string=re.escape(mikrotik_connection.find_prompt()) + r'[ \t]*$') in send_command like this #3405 (comment) ?

[prtomasi]

Hi,

I think this issue is solved.
After I changed the expect_string like @Tonygratta suggested I can now get the complete configuration using export command in RouterOS > 6.48

There was no need for me to change default username parameters in mikrotik_ssh.py file when using expect_string=re.escape(mikrotik_connection.find_prompt()) + r'[ \t]*$')

import re

mikrotik = {
    'device_type': 'mikrotik_routeros',
    'host':   '10.1.1.68',
    'port':     2222,
    'username': 'dude',
    'password': 'not_real_password'
}

mikrotik_connection = ConnectHandler(**mikrotik)

output = mikrotik_connection.send_command(
    f'export',
    expect_string=re.escape(mikrotik_connection.find_prompt()) + r'[ \t]*$')

print(output)
print(f"#=== End of script ===#")

[amyasnikov]

@Tonygratta it does help, but still not perfect.
With expect_string=re.escape(mkt.find_prompt()) + r'[ \t]*$') in action the full result of export comes out as the .send_command result, but it contains excess data at the end of the output: router prompt repeated several times:

/tool sniffer
set filter-interface=ether1 filter-ip-address=1.1.1.1/32




[admin@Router] >                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   
[admin@Router] >

@prtomasi in my opinion, the issue can't be considered solved unless plain connection.send_command('/export') starts to output full result of the "export" command without any additional params. The best thing we can say right now is that workaround is found.

[Tonygratta]

starts to output full result of the "export" command without any additional params

We already have pull request #3467 based on mentioned solving but it needs more testing.

router prompt repeated several times

That is unusual. There should be none of them at all because netmiko filters out the prompt in the .send_command output by default. Are you using strip_prompt = False parameter in the send_command function? Could you please share your relevant Python code?

[amyasnikov]

@Tonygratta I've found out it's "e" added to the username is causing these excessive prompts to appear in .send_command output.
The code is pretty straightforward:

output = conn.send_command('/export', expect_string=re.escape(conn.find_prompt()) + r'[ \t]*$')

With "e" previously added by me to the username this code causes prompts at the end of the output. Without this "e" everything works fine.