Multi-node environment does not show up in main screen

[gsilvapt]

Hi again,

Apologies if this is not the best approach to get in touch with you but for documentation sake, thought I'd raise this here and anyone else that may encounter similar problems. I apologise in advance if this is the best approach to ask questions. Kindly let me know where can I get help in that case.

After some time struggling with a swarm deployment of Isaiah, I'm having issues understanding why the other nodes do not show up in the main screen. Initially I struggled to get the agents to reach the master node because I'm running traefik for HTTP endpoints (not TCP) and the initial checks failed. Instead of patching the check, I came up with the idea of creating a network for the stack and specify the service name under MASTER_HOST. After some restarts (because quite often agents can't connect because of reasons beyond my understanding), I finally got to a state where both agents mention they are connected in the logs: | 2024/09/05 13:21:24 Connection with master node is established.

For context, I have a master node in one VM and 2 agents in 2 other VMs, separately. I'm using swarm to orchestrate the deployment but I really think this has nothing to do with swarm. Here is the redacted config:

version: '3.2'

networks:
  base:
    external: true
  isaiah:

services:
  isaiah-master:
    image: will-moss/isaiah:latest
    restart: on-failure:3
    environment:
      AUTHENTICATION_SECRET: "xxx"
    networks:
      - base
      - isaiah
    ports:
      - "3000:3000"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    deploy:
      labels:
        - "traefik.docker.network=base_monitoring"
        - "traefik.http.routers.isaiah.rule=Host(`isaiah.my.tld`)"
        - "traefik.http.services.isaiah.loadbalancer.server.port=3000"
      placement:
        constraints:
          - node.labels.node_id == 01

  isaiah-agent1:
    image: will-moss/isaiah:latest
    restart: on-failure:3
    networks:
      - isaiah
    depends_on:
      - isaiah-master
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      SERVER_ROLE: "Agent"
      AUTHENTICATION_SECRET: "xxx"

      MASTER_HOST: "isaiah-master:3000"
      MASTER_SECRET: "xxx"
      AGENT_NAME: "Isaiah Agent 02"
    deploy:
      placement:
        constraints:
          - node.labels.node_id == 02

### There's an identical isaiah-agent2 configuration for node_id == 03

In the main screen however, I can only see master node. I was expecting to have two other nodes, one for each VM where the agents is installed.

I'm sure the websockets are spun because I shut down the master node and both agents immediately failed with | 2024/09/05 13:14:52 websocket: close 1006 (abnormal closure): unexpected EOF. Starting the websockets does not really mean they are working though, but I unfortunately cannot find any tips or any logs leading to malfunctions.

Where can I look to debug this? The agents and master logs seem all normal but it's not working as expected.

[will-moss]

Just a very quick idea before I reply further, have you tried entering the UI, and push the A key, or in the lower right corner, click on "Master"? This should make the Agent Picker panel appear.

I think there's a little UX improvement to be made, but agents won't ever show in the overview before you connect to them (from the browser) at least once. This is because Agents have their own authentication, so the client (browser) can't display anything before you actually connect to every agent and provide their secret.

[gsilvapt]

Now I feel dumb :) Thanks, that worked. It wasn't clear in the UI and I didn't have the A option in the bottom bar to switch agent.
I was expecting to not have to authenticate too but it still prompts for authentication when I change agent 🤔

[will-moss]

Re,

Thanks for explaining everything and for providing all these details.

First, regarding the Master-Agent communication, it uses TCP because Master and Agent nodes communicate over a Web Socket. I make the assumption that, in your setup, Traefik may not be configured to allow direct Web Socket between nodes. Bu all in all, the most important thing is, this doesn't work over HTTP(S) at all. Really, the communication between the nodes is 100% TCP-based via the Web Sockets. Ultimately, I think that putting every container on the same network and connecting with Master using the container's hostname fixes the issue, indeed, because there is no proxy between the nodes anymore, and they can communicate directly.

Second, you may want to disable the authentication on your agents, except if you have a use case for it. Given that all the communications between the client (your browser) and the Agent nodes is proxied by the Master node, you can enable authentication on Master, and disable it on the Agent nodes, because they are never really exposed to the internet (because they communicate with Master via a private Web Socket).

Third, I will definitely improve that part. The UI is misleading, and you struggling with it is a consequence. I have just added that to my roadmap, and I hope to publish a new release in the coming days.

Bonus : If you use the same authentication secret on Master and on the Agents, there is no authentication prompt anymore. The client will transparently attempt to connect to the agents using the already-known secret used to connect to the Master node.

Hope it helps!