port forward ends unexpectedly

[aviramha]

user is running 3 mirrord port forwarding sessions, each one different app - example json + commandline. They're using operator.

the sessions drop at different times (not at the same time) but can happen to each one.

{
  "target": {
    "path": "deployment/...",
    "namespace": "..."
  },
  "feature": {
    "env": {
      "override": { 
      }
    },
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "header_filter": "x-user-id: {{ get_env(name='USER') }}",
          "ports": [8080]
        }
      },
    "outgoing": false
   },
   "fs": false
  },
  "operator": true
}

mirrord port-forward -f "${something}/mirrord/mirrord-admin.json" -R 80:9007

with verbose logging:

a13c-9f3e0cf585da] mirrord-agent loaded. 3/3 mirrord-agents alive
2024-12-05T04:07:47.267751Z  WARN handle_msg_from_agent{LogMessage(LogMessage { message: "[7b9a454c-7bbf-433a-a13c-9f3e0cf585da] mirrord-agent failed: agent was terminated: agent failed ping pong check. 2/3 mirrord-agents alive", level: Warn })}: mirrord::port_forward: agent log: [7b9a454c-7bbf-433a-a13c-9f3e0cf585da] mirrord-agent failed: agent was terminated: agent failed ping pong check. 2/3 mirrord-agents alive
2024-12-05T04:07:47.269771Z  WARN handle_msg_from_agent{LogMessage(LogMessage { message: "[3a5d7c13-3ec5-46b9-aa7f-b02193f32f5d] mirrord-agent connection was lost. 1/3 mirrord-agents alive", level: Warn })}: mirrord::port_forward: agent log: [3a5d7c13-3ec5-46b9-aa7f-b02193f32f5d] mirrord-agent connection was lost. 1/3 mirrord-agents alive
2024-12-05T04:07:47.282634Z  WARN handle_msg_from_agent{LogMessage(LogMessage { message: "[aacfd31d-f4c3-4177-b928-4aa72d1076c9] mirrord-agent connection was lost. 0/3 mirrord-agents alive", level: Warn })}: mirrord::port_forward: agent log: [aacfd31d-f4c3-4177-b928-4aa72d1076c9] mirrord-agent connection was lost. 0/3 mirrord-agents alive
2024-12-05T04:07:54.140660Z ERROR handle_msg_from_agent{LogMessage(LogMessage { message: "[3a5d7c13-3ec5-46b9-aa7f-b02193f32f5d] failed to load mirrord-agent: agent creation timed out", level: Error })}: mirrord::port_forward: agent log: [3a5d7c13-3ec5-46b9-aa7f-b02193f32f5d] failed to load mirrord-agent: agent creation timed out
2024-12-05T04:07:54.162035Z ERROR handle_msg_from_agent{LogMessage(LogMessage { message: "[7b9a454c-7bbf-433a-a13c-9f3e0cf585da] failed to load mirrord-agent: agent creation timed out", level: Error })}: mirrord::port_forward: agent log: [7b9a454c-7bbf-433a-a13c-9f3e0cf585da] failed to load mirrord-agent: agent creation timed out
2024-12-05T04:07:54.262154Z ERROR handle_msg_from_agent{LogMessage(LogMessage { message: "[aacfd31d-f4c3-4177-b928-4aa72d1076c9] failed to load mirrord-agent: agent creation timed out", level: Error })}: mirrord::port_forward: agent log: [aacfd31d-f4c3-4177-b928-4aa72d1076c9] failed to load mirrord-agent: agent creation timed out
2024-12-05T04:08:01.162505Z ERROR handle_msg_from_agent{LogMessage(LogMessage { message: "[7b9a454c-7bbf-433a-a13c-9f3e0cf585da] failed to load mirrord-agent: agent creation timed out", level: Error })}: mirrord::port_forward: agent log: [7b9a454c-7bbf-433a-a13c-9f3e0cf585da] failed to load mirrord-agent: agent creation timed out
2024-12-05T04:08:01.185896Z ERROR handle_msg_from_agent{LogMessage(LogMessage { message: "[3a5d7c13-3ec5-46b9-aa7f-b02193f32f5d] failed to load mirrord-agent: agent creation timed out", level: Error })}: mirrord::port_forward: agent log: [3a5d7c13-3ec5-46b9-aa7f-b02193f32f5d] failed to load mirrord-agent: agent creation timed out
2024-12-05T04:08:01.262064Z ERROR handle_msg_from_agent{LogMessage(LogMessage { message: "[aacfd31d-f4c3-4177-b928-4aa72d1076c9] failed to load mirrord-agent: agent creation timed out", level: Error })}: mirrord::port_forward: agent log: [aacfd31d-f4c3-4177-b928-4aa72d1076c9] failed to load mirrord-agent: agent creation timed out
2024-12-05T04:08:06.980560Z ERROR mirrord_operator::client::conn_wrapper: Operator connection failed error=WebSocket protocol error: Connection reset without closing handshake
2024-12-05T04:08:06.981734Z TRACE run: mirrord_intproxy::proxies::incoming: message bus closed, exiting
Error:   × An error occurred in the port-forwarding process: agent closed connection with error: `unexpected end of connection with agent`
  ╰─▶ agent closed connection with error: `unexpected end of connection with agent`

[linear]

MBE-559 port forward ends unexpectedly