Is there a way to keep reading output from stdout without channels closing?

[isamu-isozaki]

Hi! Sorry for all the qs and let me know if this question is better in the previous question #717 or standalone here. I think this question is a bit similar to #607 but the main distinction is I do not want my process to exit/channels to close.

Currently, I'm doing an approach suggested in the previous discussion where we can't expect a certain structure in output. So I wanted to input inputs from an user in one thread/asyncio task while in another thread/asyncio task I just keep reading anything stdout outputs. However, I noticed that at a certain point, asyncio.exceptions.IncompleteReadError gets raised because eof was received and I can basically not send any more commands since the stdin channel closes with error SSH connection failed: Channel not open for sending. A minimal example is as so

import asyncio
import asyncssh
import sys
import time
import logging
import re
# logging.basicConfig(level='DEBUG')
# asyncssh.set_debug_level(2)
async def run_client() -> None:
    async with asyncssh.connect('192.168.50.10', username='vagrant', password='vagrant', known_hosts=None) as conn:
        try:
            async with conn.create_process('echo hello', term_type='ansi') as proc:
                try:
                    while True:
                        output = await proc.stdout.readuntil(re.compile(r"[\s\S]*"))
                        print(output, "")

                except asyncio.exceptions.IncompleteReadError:
                    if proc.stdout._session._eof_received:
                        print("Eof recieved")
                    if proc.stdout._session._read_paused:
                        print("read paused")
                    print("error")
                print("Wrote to stdin")
                proc.stdin.write("echo hello2\n")
                print("Written")
                try:
                    while True:
                        output = await proc.stdout.readuntil(re.compile(r"[\s\S]*"))
                        print(output, "")

                except asyncio.exceptions.IncompleteReadError:
                    print("error")

        except asyncssh.ProcessError as exc:
            print(f"Error running su command: {exc.stderr}", file=sys.stderr)

# If you're in an environment where an event loop is already running (like Jupyter or IPython),
# you can simply use 'await' instead of 'run_until_complete'
try:
    # In Jupyter, use 'await' directly to run the coroutine
    asyncio.get_event_loop().run_until_complete(run_client())
except (OSError, asyncssh.Error) as exc:
    sys.exit('SSH connection failed: ' + str(exc))

Output:

hello

Eof recieved
error
Wrote to stdin
SSH connection failed: Channel not open for sending

I know the reason for this is because we read the eof by reading too much in the stdout but I was not sure if there is a way to know if we reach stdout beforehand. I know readline does avoid the error somewhat by just returning "" but I did notice that even then, stdin channel closes.

And I know that if we already knew the commands before hand this can be avoided/or if we allow this process to terminate and then restart a new process. But is there a way to keep the process/channels open and keep reading stdout without structure somehow? I was thinking of collect output but I do think the fundamental issue is channels closing when reading eof.

And again much appreciate the help. I am currently checking the code to see if I can achieve this but just wanted to hear your opinion first. Let me know if there's any part that sounds wrong etc!

[isamu-isozaki]

sorry I noticed my example was wrong. Updated. I'll investigate on my side too and write here if I can figure anything out

[isamu-isozaki]

I was wrong in my assumption above. The issue is not that I read the eof. It's that the command responds with an eof that the client catches. I think technically the connection can still be alive but this terminates the connection once it gets the eof which is a bit of an issue. I'm not sure if this is an asyncssh issue or just my design issue

[ronf]

An IncompleteReadError will only occur when EOF has been received. It's basically saying that the readuntil() didn't find a match in the remaining input and since EOF has been sent, there will be no possibility for more data to come in to change that.

Since you are running a command when you start the process rather than running a shell, EOF will be received just as soon as the "echo hello" command finishes running. That's what would trigger the EOF and IncompleteReadError.

If you want to run multiple commands, you need to not provide a command when starting the process, which will cause it to run an interactive shell. You could then use stdin.write() to write one or more commands to the shell. However, it will be up to you to parse the output coming back to know when one command finishes and another can be started.

[isamu-isozaki]

@ronf Thanks for the reply! This fixed the issue above. Let me confirm if just setting the process as sh will keep the eof from coming. For example, ex when writing su root to stdin and exiting etc

[isamu-isozaki]

At least in my tests seem to work! I was even able to get the one thread for stdin one thread for stdout idea above. Thanks a bunch!

[ronf]

Yes - you could pass in 'sh' if you like as the command if you wanted to force it to use that specific shell. You could also leave out that first argument, and it will default to whatever your normal login shell is. If the first command is going to be "su root", you could also pass that in as the command argument, as that will open a root shell that you can send multiple commands to, unlike your earlier example with "echo". That root shell will stay around until you explicitly send an "exit" command, or close the SSHClientProcess.

[ronf]

As you discovered, you can have two different parallel tasks handling stdin and stdout, but the stdin task won't be able to know anything about when the stdout process is done writing output for the command that was sent. As a result, it'll prompt you too early for the next command to send, before you get all the output from the previous one.