Destination connection state disconnected when a new session ssh connect to device

[royzhao7]

Describe the bug
Everything works fine at the beginning of opening tunnel but
Destination connection state will be disconnected when trying to create a new session ssh to my device
I had to open a new tunnel then works well again

To Reproduce

Steps to reproduce the behavior:

1. aws iotsecuretunneling open-tunnel --region cn-north-1 --endpoint https://api.tunneling.iot.cn-north-1.amazonaws.com.cn --destination-config thingName={thing_name},services=SSH --timeout-config maxLifetimeTimeoutMinutes=60
2. docker run --name {thing_name}-container -v /home/ec2-user/settings.json:/home/settings.json -d --network=host public.ecr.aws/aws-iot-securetunneling-localproxy/ubuntu-bin:amd64-latest --region cn-north-1 --settings-json /home/settings.json --destination-client-type V1 -v 6 -k -s 5555 -t $SourceAccessToken"

Expected behavior

Destination connection state should keep connected until timeout

Actual behavior

A clear and concise description of what actually happened.

Logs
2024-12-02 09:13:42.266807] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.266812] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.266818] (0x00007f98dd897840) [trace] Initiating tcp socket read
[2024-12-02 09:13:42.266847] (0x00007f98dd897840) [trace] Web socket write buffer drain for service id: SSH, connection id: 0
[2024-12-02 09:13:42.266854] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.266859] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.266867] (0x00007f98dd897840) [debug] Prepare to send data message: service id: SSH stream id: 4 connection id: 0
[2024-12-02 09:13:42.266900] (0x00007f98dd897840) [trace] Sending messages over web socket for service id: connection id: 0
[2024-12-02 09:13:42.266919] (0x00007f98dd897840) [trace] Current queue size: 0
[2024-12-02 09:13:42.266924] (0x00007f98dd897840) [trace] Put data 157 bytes into the web_socket_outgoing_message_queue for service id: connection id: 0
[2024-12-02 09:13:42.266939] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.266945] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.266953] (0x00007f98dd897840) [trace] Calling async_write with type: websocket_stream_single_ssl_type
[2024-12-02 09:13:42.267016] (0x00007f98dd897840) [debug] Write buffer has enough space, continue tcp read loop for SSH connection id: 0
[2024-12-02 09:13:42.267025] (0x00007f98dd897840) [trace] Begin tcp socket read loop for service id : SSH connection id : 0
[2024-12-02 09:13:42.267031] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.267036] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.267042] (0x00007f98dd897840) [debug] Not starting TCP read loop, socket is already reading
[2024-12-02 09:13:42.267055] (0x00007f98dd897840) [trace] Sent 157 bytes over websocket for service id: connection id: 0
[2024-12-02 09:13:42.267065] (0x00007f98dd897840) [trace] capturing after_send_message
[2024-12-02 09:13:42.267071] (0x00007f98dd897840) [trace] Web socket write buffer drain for service id: SSH, connection id: 0
[2024-12-02 09:13:42.267076] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.267082] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.267088] (0x00007f98dd897840) [debug] not writing, no buffer contents, skip straight to being done draining
[2024-12-02 09:13:42.267095] (0x00007f98dd897840) [trace] web_socket_outgoing_message_queue is empty, no more messages to send.
[2024-12-02 09:13:42.360926] (0x00007f98dd897840) [trace] Websocket read 44 bytes
[2024-12-02 09:13:42.360980] (0x00007f98dd897840) [trace] Message parsed successfully , type :1
[2024-12-02 09:13:42.360988] (0x00007f98dd897840) [trace] Processing data message
[2024-12-02 09:13:42.360992] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.360996] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.361001] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.361006] (0x00007f98dd897840) [trace] Using connection-specific data message handler for service id: SSH connection id: 0
[2024-12-02 09:13:42.361011] (0x00007f98dd897840) [trace] Forwarding message to tcp socket with connection id: 0
[2024-12-02 09:13:42.361015] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.361019] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.361023] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.361027] (0x00007f98dd897840) [trace] tcp write buffer drain for SSH, connection id: 0
[2024-12-02 09:13:42.361031] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.361035] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.361228] (0x00007f98dd897840) [trace] return continue_reading true
[2024-12-02 09:13:42.361244] (0x00007f98dd897840) [debug] Starting web socket read loop continue reading...
[2024-12-02 09:13:42.361248] (0x00007f98dd897840) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-12-02 09:13:42.361270] (0x00007f98dd897840) [trace] write done service id SSH, connection id: 0
[2024-12-02 09:13:42.361288] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.361292] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.361296] (0x00007f98dd897840) [trace] Wrote 36 bytes to tcp socket with service id: SSH, connection_id: 0
[2024-12-02 09:13:42.361306] (0x00007f98dd897840) [trace] TCP write buffer drain complete
[2024-12-02 09:13:42.361310] (0x00007f98dd897840) [trace] Done writing for: SSH, connection id: 0
[2024-12-02 09:13:42.361317] (0x00007f98dd897840) [trace] Handling read from tcp socket for service id SSH connection id 0
[2024-12-02 09:13:42.361325] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.361329] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.361332] (0x00007f98dd897840) [trace] TCP socket read 120 bytes for service id: SSH, connection id: 0
[2024-12-02 09:13:42.361337] (0x00007f98dd897840) [trace] Begin tcp socket read loop for service id : SSH connection id : 0
[2024-12-02 09:13:42.361341] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.361345] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.361348] (0x00007f98dd897840) [trace] Initiating tcp socket read
[2024-12-02 09:13:42.361353] (0x00007f98dd897840) [trace] Web socket write buffer drain for service id: SSH, connection id: 0
[2024-12-02 09:13:42.361357] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.361361] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.361365] (0x00007f98dd897840) [debug] Prepare to send data message: service id: SSH stream id: 4 connection id: 0
[2024-12-02 09:13:42.361379] (0x00007f98dd897840) [trace] Sending messages over web socket for service id: connection id: 0
[2024-12-02 09:13:42.361383] (0x00007f98dd897840) [trace] Current queue size: 0
[2024-12-02 09:13:42.361387] (0x00007f98dd897840) [trace] Put data 128 bytes into the web_socket_outgoing_message_queue for service id: connection id: 0
[2024-12-02 09:13:42.361392] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.361396] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.361401] (0x00007f98dd897840) [trace] Calling async_write with type: websocket_stream_single_ssl_type
[2024-12-02 09:13:42.361437] (0x00007f98dd897840) [debug] Write buffer has enough space, continue tcp read loop for SSH connection id: 0
[2024-12-02 09:13:42.361445] (0x00007f98dd897840) [trace] Begin tcp socket read loop for service id : SSH connection id : 0
[2024-12-02 09:13:42.361449] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.361452] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.361456] (0x00007f98dd897840) [debug] Not starting TCP read loop, socket is already reading
[2024-12-02 09:13:42.361464] (0x00007f98dd897840) [trace] Sent 128 bytes over websocket for service id: connection id: 0
[2024-12-02 09:13:42.361468] (0x00007f98dd897840) [trace] capturing after_send_message
[2024-12-02 09:13:42.361472] (0x00007f98dd897840) [trace] Web socket write buffer drain for service id: SSH, connection id: 0
[2024-12-02 09:13:42.361475] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.361479] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.361482] (0x00007f98dd897840) [debug] not writing, no buffer contents, skip straight to being done draining
[2024-12-02 09:13:42.361486] (0x00007f98dd897840) [trace] web_socket_outgoing_message_queue is empty, no more messages to send.
[2024-12-02 09:13:42.440239] (0x00007f98dd897840) [trace] Websocket read 637 bytes
[2024-12-02 09:13:42.440300] (0x00007f98dd897840) [trace] Message parsed successfully , type :1
[2024-12-02 09:13:42.440310] (0x00007f98dd897840) [trace] Processing data message
[2024-12-02 09:13:42.440316] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.440323] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.440329] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.440335] (0x00007f98dd897840) [trace] Using connection-specific data message handler for service id: SSH connection id: 0
[2024-12-02 09:13:42.440344] (0x00007f98dd897840) [trace] Forwarding message to tcp socket with connection id: 0
[2024-12-02 09:13:42.440350] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.440357] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.440364] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.440372] (0x00007f98dd897840) [trace] tcp write buffer drain for SSH, connection id: 0
[2024-12-02 09:13:42.440379] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.440384] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.440609] (0x00007f98dd897840) [trace] return continue_reading true
[2024-12-02 09:13:42.440630] (0x00007f98dd897840) [debug] Starting web socket read loop continue reading...
[2024-12-02 09:13:42.440636] (0x00007f98dd897840) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-12-02 09:13:42.440663] (0x00007f98dd897840) [trace] write done service id SSH, connection id: 0
[2024-12-02 09:13:42.440670] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.440677] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.440683] (0x00007f98dd897840) [trace] Wrote 628 bytes to tcp socket with service id: SSH, connection_id: 0
[2024-12-02 09:13:42.440697] (0x00007f98dd897840) [trace] TCP write buffer drain complete
[2024-12-02 09:13:42.440703] (0x00007f98dd897840) [trace] Done writing for: SSH, connection id: 0
[2024-12-02 09:13:42.801080] (0x00007f98dd897840) [trace] Websocket read 60 bytes
[2024-12-02 09:13:42.801143] (0x00007f98dd897840) [trace] Message parsed successfully , type :1
[2024-12-02 09:13:42.801153] (0x00007f98dd897840) [trace] Processing data message
[2024-12-02 09:13:42.801158] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.801164] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.801172] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.801178] (0x00007f98dd897840) [trace] Using connection-specific data message handler for service id: SSH connection id: 0
[2024-12-02 09:13:42.801185] (0x00007f98dd897840) [trace] Forwarding message to tcp socket with connection id: 0
[2024-12-02 09:13:42.801191] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.801197] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.801204] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.801211] (0x00007f98dd897840) [trace] tcp write buffer drain for SSH, connection id: 0
[2024-12-02 09:13:42.801219] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.801224] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.801276] (0x00007f98dd897840) [trace] return continue_reading true
[2024-12-02 09:13:42.801285] (0x00007f98dd897840) [debug] Starting web socket read loop continue reading...
[2024-12-02 09:13:42.801290] (0x00007f98dd897840) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-12-02 09:13:42.801311] (0x00007f98dd897840) [trace] write done service id SSH, connection id: 0
[2024-12-02 09:13:42.801318] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.801324] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.801331] (0x00007f98dd897840) [trace] Wrote 52 bytes to tcp socket with service id: SSH, connection_id: 0
[2024-12-02 09:13:42.801343] (0x00007f98dd897840) [trace] TCP write buffer drain complete
[2024-12-02 09:13:42.801348] (0x00007f98dd897840) [trace] Done writing for: SSH, connection id: 0
[2024-12-02 09:13:42.801394] (0x00007f98dd897840) [trace] Handling read from tcp socket for service id SSH connection id 0
[2024-12-02 09:13:42.801405] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.801410] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.801416] (0x00007f98dd897840) [trace] TCP socket read 408 bytes for service id: SSH, connection id: 0
[2024-12-02 09:13:42.801422] (0x00007f98dd897840) [trace] Begin tcp socket read loop for service id : SSH connection id : 0
[2024-12-02 09:13:42.801428] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.801445] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.801451] (0x00007f98dd897840) [trace] Initiating tcp socket read
[2024-12-02 09:13:42.801460] (0x00007f98dd897840) [trace] Web socket write buffer drain for service id: SSH, connection id: 0
[2024-12-02 09:13:42.801467] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.801473] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.801480] (0x00007f98dd897840) [debug] Prepare to send data message: service id: SSH stream id: 4 connection id: 0
[2024-12-02 09:13:42.801502] (0x00007f98dd897840) [trace] Sending messages over web socket for service id: connection id: 0
[2024-12-02 09:13:42.801509] (0x00007f98dd897840) [trace] Current queue size: 0
[2024-12-02 09:13:42.801515] (0x00007f98dd897840) [trace] Put data 417 bytes into the web_socket_outgoing_message_queue for service id: connection id: 0
[2024-12-02 09:13:42.801522] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.801529] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.801538] (0x00007f98dd897840) [trace] Calling async_write with type: websocket_stream_single_ssl_type
[2024-12-02 09:13:42.801588] (0x00007f98dd897840) [debug] Write buffer has enough space, continue tcp read loop for SSH connection id: 0
[2024-12-02 09:13:42.801596] (0x00007f98dd897840) [trace] Begin tcp socket read loop for service id : SSH connection id : 0
[2024-12-02 09:13:42.801613] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.801619] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.801626] (0x00007f98dd897840) [debug] Not starting TCP read loop, socket is already reading
[2024-12-02 09:13:42.801638] (0x00007f98dd897840) [trace] Sent 417 bytes over websocket for service id: connection id: 0
[2024-12-02 09:13:42.801645] (0x00007f98dd897840) [trace] capturing after_send_message
[2024-12-02 09:13:42.801651] (0x00007f98dd897840) [trace] Web socket write buffer drain for service id: SSH, connection id: 0
[2024-12-02 09:13:42.801657] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.801664] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.801670] (0x00007f98dd897840) [debug] not writing, no buffer contents, skip straight to being done draining
[2024-12-02 09:13:42.801676] (0x00007f98dd897840) [trace] web_socket_outgoing_message_queue is empty, no more messages to send.
[2024-12-02 09:13:42.850275] (0x00007f98dd897840) [trace] Websocket read 116 bytes
[2024-12-02 09:13:42.850337] (0x00007f98dd897840) [trace] Message parsed successfully , type :1
[2024-12-02 09:13:42.850348] (0x00007f98dd897840) [trace] Processing data message
[2024-12-02 09:13:42.850354] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.850360] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.850367] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.850375] (0x00007f98dd897840) [trace] Using connection-specific data message handler for service id: SSH connection id: 0
[2024-12-02 09:13:42.850382] (0x00007f98dd897840) [trace] Forwarding message to tcp socket with connection id: 0
[2024-12-02 09:13:42.850389] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.850395] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.850402] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.850409] (0x00007f98dd897840) [trace] tcp write buffer drain for SSH, connection id: 0
[2024-12-02 09:13:42.850416] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.850424] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.850471] (0x00007f98dd897840) [trace] return continue_reading true
[2024-12-02 09:13:42.850481] (0x00007f98dd897840) [debug] Starting web socket read loop continue reading...
[2024-12-02 09:13:42.850487] (0x00007f98dd897840) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-12-02 09:13:42.850509] (0x00007f98dd897840) [trace] write done service id SSH, connection id: 0
[2024-12-02 09:13:42.850515] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.850521] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.850527] (0x00007f98dd897840) [trace] Wrote 108 bytes to tcp socket with service id: SSH, connection_id: 0
[2024-12-02 09:13:42.850540] (0x00007f98dd897840) [trace] TCP write buffer drain complete
[2024-12-02 09:13:42.850545] (0x00007f98dd897840) [trace] Done writing for: SSH, connection id: 0
[2024-12-02 09:13:42.930387] (0x00007f98dd897840) [trace] Websocket read 349 bytes
[2024-12-02 09:13:42.930442] (0x00007f98dd897840) [trace] Message parsed successfully , type :1
[2024-12-02 09:13:42.930451] (0x00007f98dd897840) [trace] Processing data message
[2024-12-02 09:13:42.930457] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.930463] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.930476] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.930483] (0x00007f98dd897840) [trace] Using connection-specific data message handler for service id: SSH connection id: 0
[2024-12-02 09:13:42.930491] (0x00007f98dd897840) [trace] Forwarding message to tcp socket with connection id: 0
[2024-12-02 09:13:42.930498] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.930505] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.930510] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.930515] (0x00007f98dd897840) [trace] tcp write buffer drain for SSH, connection id: 0
[2024-12-02 09:13:42.930523] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.930529] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.930578] (0x00007f98dd897840) [trace] return continue_reading true
[2024-12-02 09:13:42.930602] (0x00007f98dd897840) [debug] Starting web socket read loop continue reading...
[2024-12-02 09:13:42.930607] (0x00007f98dd897840) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-12-02 09:13:42.930624] (0x00007f98dd897840) [trace] write done service id SSH, connection id: 0
[2024-12-02 09:13:42.930630] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.930636] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.930641] (0x00007f98dd897840) [trace] Wrote 340 bytes to tcp socket with service id: SSH, connection_id: 0
[2024-12-02 09:13:42.930653] (0x00007f98dd897840) [trace] TCP write buffer drain complete
[2024-12-02 09:13:42.930658] (0x00007f98dd897840) [trace] Done writing for: SSH, connection id: 0
[2024-12-02 09:13:42.930665] (0x00007f98dd897840) [trace] Websocket read 205 bytes
[2024-12-02 09:13:42.930675] (0x00007f98dd897840) [trace] Message parsed successfully , type :1
[2024-12-02 09:13:42.930682] (0x00007f98dd897840) [trace] Processing data message
[2024-12-02 09:13:42.930687] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.930692] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.930698] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.930704] (0x00007f98dd897840) [trace] Using connection-specific data message handler for service id: SSH connection id: 0
[2024-12-02 09:13:42.930710] (0x00007f98dd897840) [trace] Forwarding message to tcp socket with connection id: 0
[2024-12-02 09:13:42.930715] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.930720] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.930724] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.930730] (0x00007f98dd897840) [trace] tcp write buffer drain for SSH, connection id: 0
[2024-12-02 09:13:42.930735] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.930741] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.930770] (0x00007f98dd897840) [trace] return continue_reading true
[2024-12-02 09:13:42.930777] (0x00007f98dd897840) [debug] Starting web socket read loop continue reading...
[2024-12-02 09:13:42.930782] (0x00007f98dd897840) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-12-02 09:13:42.930802] (0x00007f98dd897840) [trace] write done service id SSH, connection id: 0
[2024-12-02 09:13:42.930808] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.930815] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.930821] (0x00007f98dd897840) [trace] Wrote 196 bytes to tcp socket with service id: SSH, connection_id: 0
[2024-12-02 09:13:42.930831] (0x00007f98dd897840) [trace] TCP write buffer drain complete
[2024-12-02 09:13:42.930837] (0x00007f98dd897840) [trace] Done writing for: SSH, connection id: 0
[2024-12-02 09:13:42.959736] (0x00007f98dd897840) [trace] Websocket read 60 bytes
[2024-12-02 09:13:42.959796] (0x00007f98dd897840) [trace] Message parsed successfully , type :1
[2024-12-02 09:13:42.959807] (0x00007f98dd897840) [trace] Processing data message
[2024-12-02 09:13:42.959813] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.959818] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.959825] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.959832] (0x00007f98dd897840) [trace] Using connection-specific data message handler for service id: SSH connection id: 0
[2024-12-02 09:13:42.959839] (0x00007f98dd897840) [trace] Forwarding message to tcp socket with connection id: 0
[2024-12-02 09:13:42.959844] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.959850] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.959857] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.959864] (0x00007f98dd897840) [trace] tcp write buffer drain for SSH, connection id: 0
[2024-12-02 09:13:42.959872] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.959880] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.959929] (0x00007f98dd897840) [trace] return continue_reading true
[2024-12-02 09:13:42.959940] (0x00007f98dd897840) [debug] Starting web socket read loop continue reading...
[2024-12-02 09:13:42.959945] (0x00007f98dd897840) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-12-02 09:13:42.959962] (0x00007f98dd897840) [trace] write done service id SSH, connection id: 0
[2024-12-02 09:13:42.959968] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.959974] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.959979] (0x00007f98dd897840) [trace] Wrote 52 bytes to tcp socket with service id: SSH, connection_id: 0
[2024-12-02 09:13:42.959991] (0x00007f98dd897840) [trace] TCP write buffer drain complete
[2024-12-02 09:13:42.959996] (0x00007f98dd897840) [trace] Done writing for: SSH, connection id: 0
[2024-12-02 09:13:42.960003] (0x00007f98dd897840) [trace] Websocket read 124 bytes
[2024-12-02 09:13:42.960013] (0x00007f98dd897840) [trace] Message parsed successfully , type :1
[2024-12-02 09:13:42.960020] (0x00007f98dd897840) [trace] Processing data message
[2024-12-02 09:13:42.960025] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.960031] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.960036] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.960042] (0x00007f98dd897840) [trace] Using connection-specific data message handler for service id: SSH connection id: 0
[2024-12-02 09:13:42.960048] (0x00007f98dd897840) [trace] Forwarding message to tcp socket with connection id: 0
[2024-12-02 09:13:42.960053] (0x00007f98dd897840) [debug] reverting to v2 message format
[2024-12-02 09:13:42.960059] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.960064] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.960069] (0x00007f98dd897840) [trace] tcp write buffer drain for SSH, connection id: 0
[2024-12-02 09:13:42.960074] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.960080] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.960109] (0x00007f98dd897840) [trace] return continue_reading true
[2024-12-02 09:13:42.960116] (0x00007f98dd897840) [debug] Starting web socket read loop continue reading...
[2024-12-02 09:13:42.960121] (0x00007f98dd897840) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-12-02 09:13:42.960140] (0x00007f98dd897840) [trace] write done service id SSH, connection id: 0
[2024-12-02 09:13:42.960146] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:13:42.960153] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:13:42.960159] (0x00007f98dd897840) [trace] Wrote 116 bytes to tcp socket with service id: SSH, connection_id: 0
[2024-12-02 09:13:42.960169] (0x00007f98dd897840) [trace] TCP write buffer drain complete
[2024-12-02 09:13:42.960175] (0x00007f98dd897840) [trace] Done writing for: SSH, connection id: 0
[2024-12-02 09:13:49.379748] (0x00007f98dd897840) [trace] Sent ping data: 1733130829379
[2024-12-02 09:13:49.379784] (0x00007f98dd897840) [trace] Calling async_ping with type: websocket_stream_single_ssl_type
[2024-12-02 09:13:49.379871] (0x00007f98dd897840) [trace] Successfully sent websocket ping
[2024-12-02 09:13:49.386424] (0x00007f98dd897840) [debug] Control message received enum(close=0, ping=1, pong=2): 2
[2024-12-02 09:13:49.386467] (0x00007f98dd897840) [trace] Pong reply latency: 7 ms
[2024-12-02 09:14:09.379966] (0x00007f98dd897840) [trace] Sent ping data: 1733130849379
[2024-12-02 09:14:09.380000] (0x00007f98dd897840) [trace] Calling async_ping with type: websocket_stream_single_ssl_type
[2024-12-02 09:14:09.380082] (0x00007f98dd897840) [trace] Successfully sent websocket ping
[2024-12-02 09:14:09.383035] (0x00007f98dd897840) [debug] Control message received enum(close=0, ping=1, pong=2): 2
[2024-12-02 09:14:09.383067] (0x00007f98dd897840) [trace] Pong reply latency: 4 ms
[2024-12-02 09:14:17.645652] (0x00007f98dd897840) [debug] socket port 5555
[2024-12-02 09:14:17.645711] (0x00007f98dd897840) [debug] endpoint mapping:
[2024-12-02 09:14:17.645720] (0x00007f98dd897840) [debug] SSH = 5555
[2024-12-02 09:14:17.645728] (0x00007f98dd897840) [info] Falling back to older protocol, setting new connection id to 0
[2024-12-02 09:14:17.645734] (0x00007f98dd897840) [info] creating tcp connection id 0
[2024-12-02 09:14:17.646492] (0x00007f98dd897840) [info] Accepted tcp connection on port 5555 from 127.0.0.1:47128
[2024-12-02 09:14:17.646536] (0x00007f98dd897840) [debug] Sending stream start, setting new stream ID to: 5, service id: SSH
[2024-12-02 09:14:17.646561] (0x00007f98dd897840) [trace] Sending messages over web socket for service id: connection id: 0
[2024-12-02 09:14:17.646567] (0x00007f98dd897840) [trace] Current queue size: 0
[2024-12-02 09:14:17.646573] (0x00007f98dd897840) [trace] Put data 6 bytes into the web_socket_outgoing_message_queue for service id: connection id: 0
[2024-12-02 09:14:17.646579] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:14:17.646585] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:14:17.646593] (0x00007f98dd897840) [trace] Calling async_write with type: websocket_stream_single_ssl_type
[2024-12-02 09:14:17.646668] (0x00007f98dd897840) [trace] Handling read from tcp socket for service id SSH connection id 0
[2024-12-02 09:14:17.646676] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:14:17.646681] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:14:17.646687] (0x00007f98dd897840) [trace] received error code: system:125
[2024-12-02 09:14:17.646696] (0x00007f98dd897840) [debug] Handling tcp socket error for service id: SSH connection id: 0. error message: Operation canceled
[2024-12-02 09:14:17.646706] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:14:17.646711] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:14:17.646716] (0x00007f98dd897840) [info] Disconnected from: 127.0.0.1:47128
[2024-12-02 09:14:17.646726] (0x00007f98dd897840) [trace] Web socket write buffer drain for service id: SSH, connection id: 0
[2024-12-02 09:14:17.646732] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:14:17.646737] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:14:17.646743] (0x00007f98dd897840) [debug] not writing, no buffer contents, skip straight to being done draining
[2024-12-02 09:14:17.646749] (0x00007f98dd897840) [trace] on_web_socket_write_buffer_drain_complete callback
[2024-12-02 09:14:17.646755] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:14:17.646760] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:14:17.646788] (0x00007f98dd897840) [info] simultaneous connections are not enabled, sending stream reset
[2024-12-02 09:14:17.646796] (0x00007f98dd897840) [trace] Send reset stream for service id: SSH
[2024-12-02 09:14:17.646810] (0x00007f98dd897840) [trace] Sending messages over web socket for service id: connection id: 0
[2024-12-02 09:14:17.646816] (0x00007f98dd897840) [trace] Current queue size: 1
[2024-12-02 09:14:17.646821] (0x00007f98dd897840) [trace] Put data 6 bytes into the web_socket_outgoing_message_queue for service id: connection id: 0
[2024-12-02 09:14:17.646828] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:14:17.646833] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:14:17.646845] (0x00007f98dd897840) [trace] Sent 6 bytes over websocket for service id: connection id: 0
[2024-12-02 09:14:17.646852] (0x00007f98dd897840) [trace] capturing after_send_message
[2024-12-02 09:14:17.646858] (0x00007f98dd897840) [trace] Setting up bi-directional data transfer for service id: SSH connection id: 0
[2024-12-02 09:14:17.646864] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:14:17.646870] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:14:17.646875] (0x00007f98dd897840) [trace] Clearing tcp connection buffers
[2024-12-02 09:14:17.646882] (0x00007f98dd897840) [debug] Starting web socket read loop while web socket is already reading. Ignoring...
[2024-12-02 09:14:17.646888] (0x00007f98dd897840) [trace] Begin tcp socket read loop for service id : SSH connection id : 0
[2024-12-02 09:14:17.646895] (0x00007f98dd897840) [trace] Getting tcp connection with id: 0
[2024-12-02 09:14:17.646965] (0x00007f98dd897840) [trace] num active connections for service id SSH: 1
[2024-12-02 09:14:17.646975] (0x00007f98dd897840) [trace] socket for service id : SSH connection id: 0 is not open yet, skip reading
[2024-12-02 09:14:17.646983] (0x00007f98dd897840) [trace] Sending messages over web socket for service id: connection id: 0
[2024-12-02 09:14:17.646989] (0x00007f98dd897840) [trace] Current queue size: 1
[2024-12-02 09:14:17.646995] (0x00007f98dd897840) [trace] Calling async_write with type: websocket_stream_single_ssl_type
[2024-12-02 09:14:17.647033] (0x00007f98dd897840) [trace] Sent 6 bytes over websocket for service id: connection id: 0
[2024-12-02 09:14:17.647040] (0x00007f98dd897840) [trace] capturing after_send_message
[2024-12-02 09:14:17.647046] (0x00007f98dd897840) [trace] Setting up tcp socket for service id: SSH
[2024-12-02 09:14:17.647076] (0x00007f98dd897840) [debug] Resolving bind address host: localhost
[2024-12-02 09:14:17.647089] (0x00007f98dd897840) [debug] Port to connect 5555
[2024-12-02 09:14:17.647103] (0x00007f98dd897840) [trace] web_socket_outgoing_message_queue is empty, no more messages to send.
[2024-12-02 09:14:17.647112] (0x00007f98dd897840) [error] Could not listen/accept incoming connection on localhost:5555 -- Operation canceled
[2024-12-02 09:14:17.647400] (0x00007f98dd897840) [debug] Resolved bind IP: 127.0.0.1
[2024-12-02 09:14:17.647451] (0x00007f98dd897840) [info] Listening for new connection on port 5555
[2024-12-02 09:14:29.380254] (0x00007f98dd897840) [trace] Sent ping data: 1733130869380
[2024-12-02 09:14:29.380295] (0x00007f98dd897840) [trace] Calling async_ping with type: websocket_stream_single_ssl_type
[2024-12-02 09:14:29.380401] (0x00007f98dd897840) [trace] Successfully sent websocket ping

Environment (please complete the following information):

• OS: [e.g. Ubuntu]
• Version [e.g. 16]
• Architecture: [e.g. x86-64 or x86-32]
• Localproxy commit: [e.g. 8980ea8e0190c7e8fd942f2bce8bfc01aa6a6a52]

Additional context

Add any other context about the problem here.

[ig15]

Hi @royzhao7 . Thanks for bringing up this issue. We are looking into it and will get back to you.

[dbouras]

I was about to open an issue on this as well. In my case it is triggered unintentionally when attempting to tunnel HTTP/HTTPS connections to services on the edge device by making use of SSH forwarding. For example, when opening an SSH connection without logging in, simply by forwarding a port of interest:

ssh -N -p 8940 -L 127.0.0.1:3000:172.18.0.3:3000 user@localhost

and then connecting using a local browser to 127.0.0.1:3000.
When using older versions of localProxy, concurrent connections attempted by the browser would just hang until being serviced in sequence (since only a single connection is possible when using v1).
In the current version, localProxy complains and drops the connection:

[2024-12-03 15:33:51.696872] (0x00007ff84abcbf80) [trace] Forwarding message to tcp socket with connection id: 0
[2024-12-03 15:33:51.696891] (0x00007ff84abcbf80) [debug] reverting to v2 message format
[2024-12-03 15:33:51.696909] (0x00007ff84abcbf80) [trace] Getting tcp connection with id: 0
[2024-12-03 15:33:51.697059] (0x00007ff84abcbf80) [trace] num active connections for service id SSH: 1
[2024-12-03 15:33:51.697218] (0x00007ff84abcbf80) [trace] tcp write buffer drain for SSH, connection id: 0
[2024-12-03 15:33:51.697255] (0x00007ff84abcbf80) [trace] Getting tcp connection with id: 0
[2024-12-03 15:33:51.697278] (0x00007ff84abcbf80) [trace] num active connections for service id SSH: 1
[2024-12-03 15:33:51.697396] (0x00007ff84abcbf80) [trace] return continue_reading true
[2024-12-03 15:33:51.697435] (0x00007ff84abcbf80) [debug] Starting web socket read loop continue reading...
[2024-12-03 15:33:51.697457] (0x00007ff84abcbf80) [trace] Calling async_read_some with type: websocket_stream_single_ssl_type
[2024-12-03 15:33:51.697558] (0x00007ff84abcbf80) [trace] write done service id SSH, connection id: 0
[2024-12-03 15:33:51.697589] (0x00007ff84abcbf80) [trace] Getting tcp connection with id: 0
[2024-12-03 15:33:51.697610] (0x00007ff84abcbf80) [trace] num active connections for service id SSH: 1
[2024-12-03 15:33:51.697694] (0x00007ff84abcbf80) [trace] Wrote 420 bytes to tcp socket with service id: SSH, connection_id: 0
[2024-12-03 15:33:51.697748] (0x00007ff84abcbf80) [trace] TCP write buffer drain complete
[2024-12-03 15:33:51.697762] (0x00007ff84abcbf80) [trace] Done writing for: SSH, connection id: 0
[2024-12-03 15:33:51.698313] (0x00007ff84abcbf80) [trace] Handling read from tcp socket for service id SSH connection id 0
[2024-12-03 15:33:51.698361] (0x00007ff84abcbf80) [trace] Getting tcp connection with id: 0
[2024-12-03 15:33:51.698398] (0x00007ff84abcbf80) [trace] num active connections for service id SSH: 1
[2024-12-03 15:33:51.698430] (0x00007ff84abcbf80) [trace] received error code: system:54
[2024-12-03 15:33:51.698924] (0x00007ff84abcbf80) [debug] Handling tcp socket error for service id: SSH connection id: 0. error message: Connection reset by peer
[2024-12-03 15:33:51.698947] (0x00007ff84abcbf80) [trace] Getting tcp connection with id: 0
[2024-12-03 15:33:51.698963] (0x00007ff84abcbf80) [trace] num active connections for service id SSH: 1
[2024-12-03 15:33:51.713321] (0x00007ff84abcbf80) [info] Disconnecting... remote endpoint not found due to TCP connection already terminated

Looking at the log a bit more closely it seems that localProxy may not be entirely to blame -- maybe it is the remote end: in this case, device client version v1.9.0-da10323 deployed by Greengrass component aws.greengrass.SecureTunneling version 1.0.19.

Interestingly enough, the same spurious disconnection also happens when attempting to transfer (usually large) binary files (typically gzip compressed TAR files) from or to the edge device using scp. At the SSH level the result is the same as when the browser attempts concurrent connections: an error message complaining about data corruption:

Bad packet length 2037139303.
ssh_dispatch_run_fatal: Connection to ::1 port 8940: Connection corrupted

[ig15]

Seems you have missed updating the Environment section of the issue, please do so.

And just to reiterate the flow for your initial issue:
1- You have Greengrass component aws.greengrass.SecureTunneling running on the destination device
2- You open a tunnel to that device and destination conncetion state is connected now
3- You run latest docker image of local proxy in source mode and source also reflects connected
4- You try to ssh through this tunnel but the destination connection breaks
5 You create new tunnel and try ssh again and it works

Let me know if the flow is inaccurate.

[ig15]

Also, can you tell what is the size range of these large binary files whose transfer fails when using the scp command?

[dbouras]

Hi Ishan @ig15,

I didn't mean to "hijack" @royzhao7 's issue, but the behavior I have been observing very closely matches what he is reporting (in the first comment in this thread).

All the problems I've mentioned started when switching from aws.greengrass.SecureTunneling v1.0.18 and localProxy commit d3150e0 to SecureTunneling v1.0.19 and localProxy commit 9eace74 and beyond. The issue is quite severe in that I can no longer connect reliably to remote device web interfaces (using SSH port forwarding) or transfer files using scp.

To your question: I can't seem to pin down some kind of correlation with size, apart from the fact that the file must be relatively large. It has happened to me sporadically, with sizes from a few hundred MB to several GB. This issue never occured when I was using the older versions.

[royzhao7]

Seems you have missed updating the Environment section of the issue, please do so.

And just to reiterate the flow for your initial issue: 1- You have Greengrass component aws.greengrass.SecureTunneling running on the destination device 2- You open a tunnel to that device and destination conncetion state is connected now 3- You run latest docker image of local proxy in source mode and source also reflects connected 4- You try to ssh through this tunnel but the destination connection breaks 5 You create new tunnel and try ssh again and it works

Let me know if the flow is inaccurate.

Hi @ig15

The flow is not fully accurate. The flow should be:
1- You have Greengrass component aws.greengrass.SecureTunneling running on the destination device
2- You open a tunnel to that device and destination conncetion state is connected now
3- You run latest docker image of local proxy in source mode and source also reflects connected
4- Try to ssh through this tunnel works well
5 -At the same time I create a new session like webssh connect to device but the destination connection breaks
6 I create new tunnel and try ssh again and it works

[ig15]

@dbouras Thanks for your inputs. @royzhao7 Thanks for your reply. We are working to find the reason for this behavior. Can you share the environment details in the ticket in the mean time.

[ig15]

@royzhao7 Thing is that what you are experiencing here is the expected behavior of the tunnel. You are using aws.greengrass.SecureTunneling as your destination device which only supports V1 protocol of Local Proxy (reference). Whereas in order to multiplex multiple data streams over the same tunnel, both your source and destination devices should support the V3 protocol along with appropriate tunnel configurations. You can refer to this section for more information on the same.

[royzhao7]

@royzhao7 Thing is that what you are experiencing here is the expected behavior of the tunnel. You are using aws.greengrass.SecureTunneling as your destination device which only supports V1 protocol of Local Proxy (reference). Whereas in order to multiplex multiple data streams over the same tunnel, both your source and destination devices should support the V3 protocol along with appropriate tunnel configurations. You can refer to this section for more information on the same.

Does it mean like if I wanna to implement multiplex multiple data streams .Dest device must run a V2 localproxy ?

[ig15]

@royzhao7 Yes that is right. Currently, the only secure tunneling client that supports V2 or V3 protocol is local proxy and you'll have to use that as your destination device for this setup to work