Execution stops after a few New-SSHShellStreams

[sola-tron]

Hey there,

I got a weird issue with a script.
Using Posh-SSH 3.0.8, Powershell 5.1 and 7.3

I'm connecting like this
$Session = New-SSHSession -ComputerName $IP -Credential $Cred -AcceptKey -ErrorAction Stop

then I run a bunch of Invoke-SSHCommand and a Invoke-SSHShellStreamCommand

$stream = New-SSHShellStream -SSHSession $Session
$cli = Invoke-SSHStreamShellCommand -ShellStream $stream -Command "cli json"

followed by another StreamShellCommand to get the required info from the device.
Every New-SSHShellStream is within a separate function and is executed on every beginning of the function.

After X Shell-Streams it stops working (currently trying to find out after how many)
The Powershell session keeps running, but the script stops working. Ctrl+C has no effect, I have to kill the powershell session.

Example:

write-host "leases 1"
$stream = New-SSHShellStream -SSHSession $Session
write-host "leases stream: $stream"
$cli = Invoke-SSHStreamShellCommand -ShellStream $stream -Command "cli json"
write-host "leases cli: $cli"

write-host "leases stream: $stream"
is never reached when it happens
the same function worked earlier in the script without problems.
(when I change the order of the functions it also stops at other functions using New-SSHShellStream)

Surrounded by try-catch I don't get any errors, it just stops and I can wait forever.

When I disconnect the SSH-session and reconnect somewhere in the middle of all those calls the script works till the end.

How to avoid the reconnect, since I never really know when it happens?
Do I have to "close" the streams somehow? Is there something like a connection limit I can increase?
Any ideas on debugging this issue further?

Thanks in advance!

[darkoperator]

That is an interesting one. Against the hat type of device/server is this against?Sent from my iPhoneOn Aug 15, 2023, at 3:50 AM, sola-tron ***@***.***> wrote: Hey there, I got a weird issue with a script. Using Posh-SSH 3.0.8, Powershell 5.1 and 7.3 I'm connecting like this $Session = New-SSHSession -ComputerName $IP -Credential $Cred -AcceptKey -ErrorAction Stop then I run a bunch of Invoke-SSHCommand and a Invoke-SSHShellStreamCommand $stream = New-SSHShellStream -SSHSession $Session $cli = Invoke-SSHStreamShellCommand -ShellStream $stream -Command "cli json" followed by another StreamShellCommand to get the required info from the device. Every New-SSHShellStream is within a separate function and is executed on every beginning of the function. After X Shell-Streams it stops working (currently trying to find out after how many) The Powershell session keeps running, but the script stops working. Ctrl+C has no effect, I have to kill the powershell session. Example: write-host "leases 1" $stream = New-SSHShellStream -SSHSession $Session write-host "leases stream: $stream" $cli = Invoke-SSHStreamShellCommand -ShellStream $stream -Command "cli json" write-host "leases cli: $cli" write-host "leases stream: $stream" is never reached when it happens the same function worked earlier in the script without problems. (when I change the order of the functions it also stops at other functions using New-SSHShellStream) Surrounded by try-catch I don't get any errors, it just stops and I can wait forever. When I disconnect the SSH-session and reconnect somewhere in the middle of all those calls the script works till the end. How to avoid the reconnect, since I never really know when it happens? Do I have to "close" the streams somehow? Is there something like a connection limit I can increase? Any ideas on debugging this issue further? Thanks in advance! —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[sola-tron]

It's a Securepoint UTM Appliance. Connecting via SSH gives you access to the CLI, not a "real" root shell:
https://wiki.securepoint.de/index.php?title=UTM/EXTRAS/CLI&uselang=en

My script was working fine till yesterday, where I added one more function with another New-SSHShellStream.
Then the issue appeared. I thought it's because of using PS 7.3, tried with 5.1 to get the same behavior.

After that I added the Disconnect & Connect right before the execution of the new function and it did work again

[sola-tron]

the magical number seems to be 11 :D

while ($true) {
    Write-Host $counter
    $stream = New-SSHShellStream -SSHSession $Session
    $cli = Invoke-SSHStreamShellCommand -ShellStream $stream -Command "cli json"
    $dhcpPools = ((Invoke-SSHStreamShellCommand -ShellStream $stream -Command "dhcp pool get") | ConvertFrom-Json).result.content

    $stream | ft *
    $counter++
}

1

SessionId Session                DataAvailable CanRead CanSeek CanWrite Length Position CanTimeout ReadTimeout WriteTimeout
--------- -------                ------------- ------- ------- -------- ------ -------- ---------- ----------- ------------
        0 Renci.SshNet.SshClient         False    True   False     True      0        0      False

2

SessionId Session                DataAvailable CanRead CanSeek CanWrite Length Position CanTimeout ReadTimeout WriteTimeout
--------- -------                ------------- ------- ------- -------- ------ -------- ---------- ----------- ------------
        0 Renci.SshNet.SshClient         False    True   False     True      0        0      False

3

SessionId Session                DataAvailable CanRead CanSeek CanWrite Length Position CanTimeout ReadTimeout WriteTimeout
--------- -------                ------------- ------- ------- -------- ------ -------- ---------- ----------- ------------
        0 Renci.SshNet.SshClient         False    True   False     True      0        0      False

4

SessionId Session                DataAvailable CanRead CanSeek CanWrite Length Position CanTimeout ReadTimeout WriteTimeout
--------- -------                ------------- ------- ------- -------- ------ -------- ---------- ----------- ------------
        0 Renci.SshNet.SshClient         False    True   False     True      0        0      False

5

SessionId Session                DataAvailable CanRead CanSeek CanWrite Length Position CanTimeout ReadTimeout WriteTimeout
--------- -------                ------------- ------- ------- -------- ------ -------- ---------- ----------- ------------
        0 Renci.SshNet.SshClient         False    True   False     True      0        0      False

6

SessionId Session                DataAvailable CanRead CanSeek CanWrite Length Position CanTimeout ReadTimeout WriteTimeout
--------- -------                ------------- ------- ------- -------- ------ -------- ---------- ----------- ------------
        0 Renci.SshNet.SshClient         False    True   False     True      0        0      False

7

SessionId Session                DataAvailable CanRead CanSeek CanWrite Length Position CanTimeout ReadTimeout WriteTimeout
--------- -------                ------------- ------- ------- -------- ------ -------- ---------- ----------- ------------
        0 Renci.SshNet.SshClient         False    True   False     True      0        0      False

8

SessionId Session                DataAvailable CanRead CanSeek CanWrite Length Position CanTimeout ReadTimeout WriteTimeout
--------- -------                ------------- ------- ------- -------- ------ -------- ---------- ----------- ------------
        0 Renci.SshNet.SshClient         False    True   False     True      0        0      False

9

SessionId Session                DataAvailable CanRead CanSeek CanWrite Length Position CanTimeout ReadTimeout WriteTimeout
--------- -------                ------------- ------- ------- -------- ------ -------- ---------- ----------- ------------
        0 Renci.SshNet.SshClient         False    True   False     True      0        0      False

10

SessionId Session                DataAvailable CanRead CanSeek CanWrite Length Position CanTimeout ReadTimeout WriteTimeout
--------- -------                ------------- ------- ------- -------- ------ -------- ---------- ----------- ------------
        0 Renci.SshNet.SshClient         False    True   False     True      0        0      False

11

this one runs forever:

while ($true) {
    Write-Host $counter
    $stream = New-SSHShellStream -SSHSession $Session
    $cli = Invoke-SSHStreamShellCommand -ShellStream $stream -Command "cli json"
    $dhcpPools = ((Invoke-SSHStreamShellCommand -ShellStream $stream -Command "dhcp pool get") | ConvertFrom-Json).result.content

    $stream | ft *

    if ($counter -eq 10) {
        $Session.Disconnect()
        $Session.Connect()
        $counter = 0
    }

    $counter++
}

The command I'm sending to the device in Invoke-SSHStreamShellCommand doesn't matter, tried various.

This is also running till infinity:

while ($true) {
    Write-Host $counter
    $info = Invoke-SSHCommand -SSHSession $Session -Command "dhcp pool get"
    $counter++
}