Investigate sbot discovery over 802.11s mesh point interfaces

[benhylau]

The issue, as we have discovered earlier, is that sbot nodes mutually discover each other when connected to the same LAN through a router, but not over the wireless mesh point interfaces even though they have distinct local IP addresses.

eth0 peer

root@bloor:~# docker exec -it ssb sbot gossip.peers
...
  {
    "host": "192.168.1.141",
    "port": 8008,
    "key": "@KtcWUFe7CaStdfrvQG30vjNYLn7rstd6qERC5s46qSQ=.ed25519",
    "source": "local",
    "announcers": 1,
    "duration": null,
    "client": false,
    "state": "connected",
    "stateChange": 1527099875872
  }

192.168.1.141 is bathurst, connected to the router over ethernet cable as bloor is.

wlan1 no-peer

But, the broadcasts are being received, at least for @benhylau running sbot in dockers with host networking:

root@bloor:~# tcpdump -i wlan1 | grep 8008                                                                     
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wlan1, link-type EN10MB (Ethernet), capture size 262144 bytes

18:49:45.786481 IP union.8008 > 255.255.255.255.8008: UDP, length 66
18:49:46.786370 IP union.8008 > 255.255.255.255.8008: UDP, length 66
18:49:47.788036 IP union.8008 > 255.255.255.255.8008: UDP, length 66
18:49:48.789712 IP union.8008 > 255.255.255.255.8008: UDP, length 66
18:49:49.790210 IP union.8008 > 255.255.255.255.8008: UDP, length 66
18:49:50.791620 IP union.8008 > 255.255.255.255.8008: UDP, length 66

but not for @darkdrgn2k's setup.

union is connected to bloor and bathurst only via wlan1 which is mesh point.

[darkdrgn2k]

After a TON of digging I think I have figured it out.

I boiled it right down to a C example to try to figure out the Broadcast issue on a simple debuggable platform. However I had a hard time getting to work with the error sendto: Network is unreachable. Looking up the error this mean that the IP address was not on the local network. This was odd because the target IP address was 255.255.255.255

What is happening is the network stack (linux) does not know WHERE to send 255.255.255.255 since there are multiple interfaces on the network. In C there is an option called SO_BINDTODEVICE that will tie the socket to the interface, however node.js does not seem to have that option.

After reverse engineering and poking at the protocol some more I found we can generate our OWN packets and lay them on the network wire correctly. (Another issue I noticed was that when sbot generated the packets, it generated them with the access point's ip address (one of the interfaces) regardless which interface it was sent out on. Since all nodes have this ip address locally to them they tried to connect to themselves and failed. In my case net:10.0.0.1:8008~shs:z0oUH/HFCzhWW2G07jE4tYOUF6JHyHjRDpr/Em9MOl8=)

The packets can be constructed as such

echo -n "net:192.168.0.3:8008~shs:Lg9wNa6sut/H99aHl2PplHdwpxa2RcQgvUslkT/1qV0o=" |  \
socat - UDP4-DATAGRAM:255.255.255.255:8008,broadcast,so-bindtodevice=wlan1

Where the IP address it the IP address you want the remote device to "call you back" (its your IP address) along with the port. ~shs means "secret hand shake" with the pubkey (from sbot whoami truncated) that follows. Finally wlan1 is the interface to lay these packets on (prevent the unreachable error)

You will not see the peers in sbot gossip.peers but if you run sbot gossip.changes on the remote machine you will see it connect.

A script can be run to iterate through the interfaces and produce the required packets are regular intervals as a work around.

Sample script

id=$(sbot whoami | grep id | awk -F "\"" '{print $4}' | sed 's/.ed25519//' | sed 's/@//')
for int in $(ls -1Atu /sys/class/net ); do
    ip=$(ip addr show $int | grep -v inet6 | grep -v '127.0.0.1' |grep inet | head -n 1 | awk '{print $2}' | awk -F "/" '{print $1}')
    if ! [ -z "$ip" ]; then
        echo -n "net:$ip:8008~shs:$id" |  socat - UDP4-DATAGRAM:255.255.255.255:8008,broadcast,so-bindtodevice=$int
        echo "net:$ip:8008~shs:$id:$int"
    fi
done

---

Edited for horrible grammer

[darkdrgn2k]

Additional debugging tools for UDP Broadcasts

Listen for UDP traffic
socat - udp4-listen:8008

Generate UDP Traffic
socat STDIO UDP4-DATAGRAM:255.255.255.255:8008,broadcast,so-bindtodevice=wlan1

[benhylau]

So bloor actually sees union with key @AYcVzgNQzhjNgIcywOPq7KGOP0GWwIiipe8n8Vx5B4s=.ed25519:

root@bloor:~# docker exec -it ssb sbot gossip.peers
...
  {
    "host": "10.0.0.1",
    "port": 8008,
    "key": "@AYcVzgNQzhjNgIcywOPq7KGOP0GWwIiipe8n8Vx5B4s=.ed25519",
    "source": "local",
    "announcers": 1,
    "duration": {
      "mean": 0.029072306444569217,
      "stdev": 3.241505619533097,
      "count": 16029,
      "sum": 466,
      "sqsum": 168436
    },
    "stateChange": 1527134364100,
    "state": "connected",
    "error": "Error: hello not accepted\n    at abort (/usr/lib/node_modules/scuttlebot/node_modules/secret-handshake/protocol.js:36:45)\n    at Object.cb (/usr/lib/node_modules/scuttlebot/node_modules/secret-handshake
/protocol.js:50:26)\n    at drain (/usr/lib/node_modules/scuttlebot/node_modules/pull-reader/index.js:42:23)\n    at /usr/lib/node_modules/scuttlebot/node_modules/pull-reader/index.js:59:18\n    at /usr/lib/node_module
s/scuttlebot/node_modules/pull-reader/index.js:20:7\n    at /usr/lib/node_modules/scuttlebot/node_modules/pull-stream/throughs/through.js:19:9\n    at drain (/usr/lib/node_modules/scuttlebot/node_modules/stream-to-pull
-stream/index.js:141:18)\n    at Socket.<anonymous> (/usr/lib/node_modules/scuttlebot/node_modules/stream-to-pull-stream/index.js:158:5)\n    at emitNone (events.js:111:20)\n    at Socket.emit (events.js:208:7)",
    "failure": 16027,
    "client": false
  }

But the host should be 10.0.1.36:

root@union:~# networkctl status
...
       Address: 10.0.0.1 on wlan0
                172.17.0.1 on docker0
                10.0.1.36 on wlan1
...

[darkdrgn2k]

I have see this as well, when packets do go through the ip address is incorrect making the remote device unable to connect or connects to the wrong device.

[benhylau]

This script from @darkdrgn2k makes peering work by sending out manually constructed broadcasts:

#!/bin/sh
id=$(sbot whoami | grep id | awk -F "\"" '{print $4}' | sed 's/.ed25519//' | sed 's/@//')
while true; do
    for int in $(ls -1Atu /sys/class/net ); do
        ip=$(ip addr show $int | grep -v inet6 | grep -v '127.0.0.1' |grep inet | head -n 1 | awk '{print $2}' | awk -F "/" '{print $1}')
        if ! [ -z "$ip" ]; then
            echo -n "net:$ip:8008~shs:$id" |  socat - UDP4-DATAGRAM:255.255.255.255:8008,broadcast,so-bindtodevice=$int
            echo "net:$ip:8008~shs:$id:$int"
    
        fi
    done
    sleep 5
done

If running sbot in a docker, replace sbot whoami with docker exec -it ssb sbot whoami.

Then run:

root@union:~# ./peercast
net:172.17.0.1:8008~shs:AYcVzgNQzhjNgIcywOPq7KGOP0GWwIiipe8n8Vx5B4s=:docker0
net:10.0.0.1:8008~shs:AYcVzgNQzhjNgIcywOPq7KGOP0GWwIiipe8n8Vx5B4s=:wlan0
net:169.254.139.163:8008~shs:AYcVzgNQzhjNgIcywOPq7KGOP0GWwIiipe8n8Vx5B4s=:wlan1
net:172.17.0.1:8008~shs:AYcVzgNQzhjNgIcywOPq7KGOP0GWwIiipe8n8Vx5B4s=:docker0
net:10.0.0.1:8008~shs:AYcVzgNQzhjNgIcywOPq7KGOP0GWwIiipe8n8Vx5B4s=:wlan0
net:169.254.139.163:8008~shs:AYcVzgNQzhjNgIcywOPq7KGOP0GWwIiipe8n8Vx5B4s=:wlan1
...

Now you get peering over mesh point wlan1's link-local:

root@bloor:~# docker exec -it ssb sbot gossip.peers
...
  {
    "host": "169.254.139.163",
    "port": 8008,
    "key": "@AYcVzgNQzhjNgIcywOPq7KGOP0GWwIiipe8n8Vx5B4s=.ed25519",
    "source": "local",
    "announcers": 1,
    "duration": null,
    "stateChange": 1527135373056,
    "state": "connected",
    "failure": 0
  }