Weighted loadbalancing (client steering), WG peer config push to clients

[DasSkelett]

In-progress work to add weighted loadbalancing between workers/gateways to wgkex. This allows for steering clients between gateways for better load distribution. Relatedly, parts of the WireGuard config (endpoint address+port, pubkey, gateway wg interface address) is now pushed to clients in the exchange API response.

See #39 and #49 and freifunkMUC/site-ffm#142

Overview

Each worker periodically scans for the number of connected peers for each interface/domain, and publishes the value through MQTT.
Each worker publishes its WireGuard PubKey, WireGuard listening port, WireGuard interface address and externally resolvable hostname/public IP address (read from config) through MQTT.

The broker stores the metrics and worker data. When a client hits the exchange endpoint, the best worker based on a simple weighting algorithm is selected.
The PubKey, interface address and external address+port for this worker is returned in the response to the client, like:

{
  "Endpoint": {
    "Address": "gw04.ext.ffmuc.net",
    "Port": "40011",
    "AllowedIPs": [
      "fe80::27c:16ff:fec0:6c74"
    ],
    "PublicKey": "TszFS3oFRdhsJP3K0VOlklGMGYZy+oFCtlaghXJqW2g="
  }
}

The weighting algorithm

Based on the total number of connected peers the should-be value for each worker is calculated like target = (worker_weight / sum_of_weights) * total_peers.
Then the difference with the actual number of connections is calculated: diff = (actual_peers - target)
The worker list is then sorted by the difference values; the worker with the lowest diff value is chosen (usually below 0, i.e. "clients missing").

MQTT topics

Publishing keys broker->worker: wireguard/{domain}/{worker}
Publishing metrics worker->broker: wireguard-metrics/{domain}/{worker}/connected_peers
Publishing worker status: wireguard-worker/{worker}/status
Publishing worker data: wireguard-worker/{worker}/{domain}/data

Other cleanup

• README additions & build fixes #89
• The tests now use the absolute import pathes wgkex.{worker,config,common,broker}.*, which allows running most tests with python3 -m unittest as well. The exception is netlink_test.py, as somehow the mocking doesn't work 100% there yet.
  This required changes in the BUILD files, and the mocking code.
• The config is now loaded and required in many more places. In order to avoid performance regressions through constant disk re-reads and reparsing through load_config() the config system has been refactored.
  The config is read and parsed and converted into a Config object only once at first use.
  The Config class is now used as primary access to the config values, instead of using the bare dict.

TODO:

• More testing, multiple interfaces
• Code cleanup & quality improvements & documentation
• Unit tests
• Split parts into separate PRs for easier review

Possible after merge in future iterations:

• Set client.suppress_exceptions on worker to avoid crashes when errors are raised.
• Monitor performance and making sure there are no performance regressions with all the new loops and dicts
• Writing the client code for this in gluon-mesh-wireguard-vxlan, so the clients actually choose the returned gateway. This can be done afterwards, though.
• Sign JSON responses, verify signatures in client code

Closes #39

[DasSkelett]

Ugh, I might have to redesign the worker background threads for metrics etc.
Judging from eclipse/paho.mqtt.python#354, the library is not threadsafe, calling client.publish simultaneously causes a deadlock.

Although at the bottom it says that version 1.6 has improved in this regard.

[awlx]

In the future we should add another json response called Signature to verify the reply originated an authoritative broker.

[GoliathLabs]

@DasSkelett would it be possible to improve the test coverage?

[awlx]

I think we should test it soon.

[grische]

@DasSkelett can you check the conflicts?

[DasSkelett]

Even more bugs fixed from the old code, pretty happy with it now. A test deploy of the broker on one of our hosts was fine.