Netmiko dont detect mikrotik prompt but in session log it persist

[jpVm5jYYRE1VIKL]

Description of Issue/Question

Note: Please check https://guides.github.com/features/mastering-markdown/
to see how to properly format your request.

Setup

Netmiko version

(Paste verbatim output from pip freeze | grep netmiko between quotes below)

netmiko==4.4.0

Netmiko device_type (if relevant to the issue)

(Paste device_type between quotes below)

mikrotik_routeros

Steps to Reproduce the Issue

Error Traceback

(Paste the complete traceback of the exception between quotes below)

Traceback (most recent call last):
  File "/home/ktoto/PycharmProjects/MikrotikBackuper/main.py", line 36, in <module>
    print_hi('PyCharm')
  File "/home/ktoto/PycharmProjects/MikrotikBackuper/main.py", line 28, in print_hi
    output = net_connect.send_command(expect_string=prompt,command_string='export')
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ktoto/PycharmProjects/MikrotikBackuper/.venv/lib/python3.12/site-packages/netmiko/base_connection.py", line 111, in wrapper_decorator
    return_val = func(self, *args, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ktoto/PycharmProjects/MikrotikBackuper/.venv/lib/python3.12/site-packages/netmiko/utilities.py", line 596, in wrapper_decorator
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ktoto/PycharmProjects/MikrotikBackuper/.venv/lib/python3.12/site-packages/netmiko/base_connection.py", line 1826, in send_command
    raise ReadTimeout(msg)
netmiko.exceptions.ReadTimeout: 
Pattern not detected: '[exporter@LSPS2 Castor] >' in output.

Things you might try to fix this:
1. Explicitly set your pattern using the expect_string argument.
2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

Relevant Python code

(Please try to essentialize your Python code to the minimum code needed to reproduce the issue)
(Paste the code between the quotes below)

    net_connect = ConnectHandler(**mikrotik)
    prompt = net_connect.find_prompt()

    print("\""+prompt+"\"")
    output = net_connect.send_command(expect_string=prompt,command_string='export')

    net_connect.disconnect()

If i look to session log .
i see at the end

/user aaa
set accounting=no use-radius=yes




[exporter@LSPS2 Castor] >                                                      
[exporter@LSPS2 Castor] > 

last line without newline

[jpVm5jYYRE1VIKL]

i tried adjusted command which if to execute it in terminal generate just single empty line

   net_connect = ConnectHandler(**mikrotik)
    prompt = net_connect.find_prompt()
    output = net_connect.send_command(expect_string=prompt,command_string='/ip/packing/print')
    net_connect.disconnect()

result if to use inside normal ssh session command "/ip/packing/print":

[exporter@LSPS2 Castor] > /ip/packing/print                                    

[exporter@LSPS2 Castor] >               

output log if to use code with netmiko and still generated Pattern not detected

[exporter@LSPS2 Castor] >                                                      
[exporter@LSPS2 Castor] > /ip/packing/print
[exporter@LSPS2 Castor] > /ip/packing/print





[exporter@LSPS2 Castor] >                                                      
[exporter@LSPS2 Castor] > 

last line without newline

[ktbyers]

@jpVm5jYYRE1VIKL Try:

output = net_connect.send_command(expect_string=r">", command_string='export')

And see if that makes any difference (or not).

[jpVm5jYYRE1VIKL]

And see if that makes any difference (or not).

Hi i tried it . And not got any changes ie still issue persist. But i can say more
i just did a small test rewrite using pure paramiko ssh lib

with quick and dirty just to be sure that is is possible :

     input_stream , output_stream, error_stream = ssh_client.exec_command(command='export')
        while message_finished is False and counter < 200:
            line = output_stream.readline().strip()
            err_line = error_stream.readline().strip()
            print(counter , line)
            print(err_line)
            counter = counter + 1

and it is works quite good (but i not did shell activation) just used raw paramiko client. So i think it is something really wrong with function which expect prompt because on one of devices with another mikrotik version i got like this in session log.

[exporter@Mikrotik1 ] > 

[exporter@Mikrotik1 ] >                                                                                                                               

[exporter@Mikrotik1 ] > e

[exporter@Mikrotik1 ] > e                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     

[exporter@Mikrotik1 ] > ex

[exporter@Mikrotik1 ] > ex                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    

[exporter@Mikrotik1 ] > exp

Is it possible somehow to have own callback to catch lines to find moment when command execution finished ?

[ktbyers]

Mikrotik's are weird devices from an SSH automation perspective (as you can see from the constant repainting of the line containing the command).

You could try to enable Netmiko logging and post the log output here (to provide more information on why Netmiko is failing--as I don't see anything that is clearly wrong):

https://github.com/ktbyers/netmiko/blob/develop/COMMON_ISSUES.md#enable-netmiko-logging-of-all-reads-and-writes-of-the-communications-channel

Alternatively, you can use read_channel() and write_channel() methods in Netmiko and likely capture similar output as you have with Paramiko (but then you will have similar issues in determining when the output of the command is done).

Netmiko's send_command() has these mechanisms built-in, but obviously we are running into an issue.

[Tonygratta]

jpVm5jYYRE1VIKL did you try expect_string=re.escape(mikrotik_connection.find_prompt()) + r'[ \t]*$')?
See #3405 (comment)
You might also want to take a look at this pull request. #3467

[jpVm5jYYRE1VIKL]

i manually implemented(written on python) needed functionality using asyncssh and it works excellent. So most probably issue not in mikrotik. May be in paramiko lib itself

[Tonygratta]

@jpVm5jYYRE1VIKL Also did you try to increase read timeout by adding read_timeout=60 into send_command parameters? Some mikrotik routers are really slow at config export.
By default, read timeout is as small as 10 seconds

[jpVm5jYYRE1VIKL]

Well i don`t think that issue related to timeout . I have quite big network with hundreds CCR routers which are really fast and with fiber connections. In reality i think issue more related with bit weird terminal cli of mikrotik. For example they use dumb '\r' as end on line instead of normal linux '\n' and not even win '\r\n' combo.