AutoInterface not discovering any peers between Manjaro and PostmarketOS

[klundry]

I have "enable_node = yes" in my config on both clients on my LAN. But when I go to Network and view the Announce Stream there is nothing there on either one. Even when selecting Announce Now again nothing shows up so I can't get my nodes/clients connected to each other.

Have I missed something?

[markqvist]

I don't think you have missed anything, this should definitely work out of the box. Would you mind posting the Reticulum config from the system hosting a nomadnet node, and also from one of the clients?

[markqvist]

Also, can you report what operating systems you are using on these machines?

[klundry]

Both of these are running linux. One Manjaro and one postmarketOS (alpine). I didn't do anything with reticulum manually. I just set enable_node to yes in the nomadnet config on both. But here is the reticulum config.

# This is the default Reticulum config file.
# You should probably edit it to include any additional,
# interfaces and settings you might need.

# Only the most basic options are included in this default
# configuration. To see a more verbose, and much longer,
# configuration example, you can run the command:
# rnsd --exampleconfig


[reticulum]
  
  # If you enable Transport, your system will route traffic
  # for other peers, pass announces and serve path requests.
  # This should only be done for systems that are suited to
  # act as transport nodes, ie. if they are stationary and
  # always-on. This directive is optional and can be removed
  # for brevity.
  
  enable_transport = True
  
  
  # By default, the first program to launch the Reticulum
  # Network Stack will create a shared instance, that other
  # programs can communicate with. Only the shared instance
  # opens all the configured interfaces directly, and other
  # local programs communicate with the shared instance over
  # a local socket. This is completely transparent to the
  # user, and should generally be turned on. This directive
  # is optional and can be removed for brevity.
  
  share_instance = Yes
  
  
  # If you want to run multiple *different* shared instances
  # on the same system, you will need to specify different
  # shared instance ports for each. The defaults are given
  # below, and again, these options can be left out if you
  # don't need them.
  
  shared_instance_port = 37428
  instance_control_port = 37429
  
  
  # You can configure Reticulum to panic and forcibly close
  # if an unrecoverable interface error occurs, such as the
  # hardware device for an interface disappearing. This is
  # an optional directive, and can be left out for brevity.
  # This behaviour is disabled by default.
  
  panic_on_interface_error = No


[logging]
  # Valid log levels are 0 through 7:
  #   0: Log only critical information
  #   1: Log errors and lower log levels
  #   2: Log warnings and lower log levels
  #   3: Log notices and lower log levels
  #   4: Log info and lower (this is the default)
  #   5: Verbose logging
  #   6: Debug logging
  #   7: Extreme logging
  
  loglevel = 4


# The interfaces section defines the physical and virtual
# interfaces Reticulum will use to communicate on. This
# section will contain examples for a variety of interface
# types. You can modify these or use them as a basis for
# your own config, or simply remove the unused ones.

[interfaces]
  
  # This interface enables communication with other
  # link-local Reticulum nodes over UDP. It does not
  # need any functional IP infrastructure like routers
  # or DHCP servers, but will require that at least link-
  # local IPv6 is enabled in your operating system, which
  # should be enabled by default in almost any OS. See
  # the Reticulum Manual for more configuration options.
  
  [[Default Interface]]
    type = AutoInterface
    enabled = Yes

[markqvist]

Thanks! Can you also run rnsd -vvvv on both machines before running nomadnet? This will start up the reticulum stack as a foreground daemon with debug logging enabled. If you can paste the log output from both machines to here, it would help immensely. I just need the output up until, and a few seconds after, an announce has been sent from the nomadnet node.

[klundry]

[2022-08-24 16:03:02] [Debug] Started shared instance interface: Shared Instance[37428]
[2022-08-24 16:03:02] [Extra] Cleaning resource and packet caches...
[2022-08-24 16:03:02] [Verbose] Bringing up system interfaces...
[2022-08-24 16:03:02] [Extra] AutoInterface[Default Interface] No link-local IPv6 address configured for lo, skipping interface
[2022-08-24 16:03:02] [Extra] AutoInterface[Default Interface] Selecting link-local address fe80::b84e:1b01:e370:5166%wlp4s0 for interface wlp4s0
[2022-08-24 16:03:02] [Extra] AutoInterface[Default Interface] Creating multicast discovery listener on wlp4s0 with address ff12:0:d70b:fb1c:16e4:5e39:485e:31e1
[2022-08-24 16:03:02] [Verbose] AutoInterface[Default Interface] discovering peers for 1.5 seconds...
[2022-08-24 16:03:04] [Verbose] System interfaces are ready
[2022-08-24 16:03:04] [Verbose] Configuration loaded from /home/kelbot/.reticulum/config
[2022-08-24 16:03:04] [Verbose] Loaded 1 known destination from storage
[2022-08-24 16:03:04] [Verbose] Loaded Transport Identity from storage
[2022-08-24 16:03:04] [Verbose] Loaded 0 path table entries from storage
[2022-08-24 16:03:04] [Verbose] Loaded 0 tunnel table entries from storage
[2022-08-24 16:03:04] [Verbose] Transport instance <596a9c59bb5dc1b243ac9aba88b0fdd6> started
[2022-08-24 16:03:04] [Notice] Started rnsd version 0.3.11
[2022-08-24 16:03:11] [Extra] Accepting new connection to shared instance: LocalInterface[39706]
[2022-08-24 16:03:11] [Extra] Valid announce for <58861e76b135d3bc6679c2928a6e7b0e> 0 hops away, received on LocalInterface[39706]
[2022-08-24 16:03:11] [Debug] Destination <58861e76b135d3bc6679c2928a6e7b0e> is now 0 hops away via <58861e76b135d3bc6679c2928a6e7b0e> on LocalInterface[39706]
[2022-08-24 16:03:11] [Extra] Valid announce for <d9480a34d9dd9da3c5299aa8b33db321> 0 hops away, received on LocalInterface[39706]
[2022-08-24 16:03:11] [Debug] Destination <d9480a34d9dd9da3c5299aa8b33db321> is now 0 hops away via <d9480a34d9dd9da3c5299aa8b33db321> on LocalInterface[39706]
[2022-08-24 16:03:11] [Extra] Valid announce for <c3bdaf015606af519611a9cb27ecef07> 0 hops away, received on LocalInterface[39706]
[2022-08-24 16:03:11] [Debug] Destination <c3bdaf015606af519611a9cb27ecef07> is now 0 hops away via <c3bdaf015606af519611a9cb27ecef07> on LocalInterface[39706]
[2022-08-24 16:03:14] [Extra] Valid announce for <d9480a34d9dd9da3c5299aa8b33db321> 0 hops away, received on LocalInterface[39706]
[2022-08-24 16:03:14] [Debug] Destination <d9480a34d9dd9da3c5299aa8b33db321> is now 0 hops away via <d9480a34d9dd9da3c5299aa8b33db321> on LocalInterface[39706]
[2022-08-24 16:03:15] [Debug] Rebroadcasting announce for <d9480a34d9dd9da3c5299aa8b33db321> with hop count 0
[2022-08-24 16:03:16] [Extra] Completed announce processing for <d9480a34d9dd9da3c5299aa8b33db321>, retry limit reached
[2022-08-24 16:03:17] [Extra] Valid announce for <c3bdaf015606af519611a9cb27ecef07> 0 hops away, received on LocalInterface[39706]
[2022-08-24 16:03:17] [Debug] Destination <c3bdaf015606af519611a9cb27ecef07> is now 0 hops away via <c3bdaf015606af519611a9cb27ecef07> on LocalInterface[39706]
[2022-08-24 16:03:18] [Debug] Rebroadcasting announce for <c3bdaf015606af519611a9cb27ecef07> with hop count 0
[2022-08-24 16:03:19] [Extra] Completed announce processing for <c3bdaf015606af519611a9cb27ecef07>, retry limit reached
[2022-08-24 16:03:20] [Debug] AutoInterface[Default Interface] added peer fe80::6eb4:f709:3a59:af2 on wlp4s0
[2022-08-24 16:03:31] [Extra] Valid announce for <58861e76b135d3bc6679c2928a6e7b0e> 0 hops away, received on LocalInterface[39706]
[2022-08-24 16:03:31] [Debug] Destination <58861e76b135d3bc6679c2928a6e7b0e> is now 0 hops away via <58861e76b135d3bc6679c2928a6e7b0e> on LocalInterface[39706]
[2022-08-24 16:03:32] [Debug] Rebroadcasting announce for <58861e76b135d3bc6679c2928a6e7b0e> with hop count 0
[2022-08-24 16:03:33] [Extra] Completed announce processing for <58861e76b135d3bc6679c2928a6e7b0e>, retry limit reached
[2022-08-24 16:03:37] [Extra] Valid announce for <58861e76b135d3bc6679c2928a6e7b0e> 0 hops away, received on LocalInterface[39706]
[2022-08-24 16:03:37] [Debug] Destination <58861e76b135d3bc6679c2928a6e7b0e> is now 0 hops away via <58861e76b135d3bc6679c2928a6e7b0e> on LocalInterface[39706]
[2022-08-24 16:03:38] [Debug] Rebroadcasting announce for <58861e76b135d3bc6679c2928a6e7b0e> with hop count 0
[2022-08-24 16:03:39] [Extra] Completed announce processing for <58861e76b135d3bc6679c2928a6e7b0e>, retry limit reached
[2022-08-24 16:04:33] [Extra] Valid announce for <d9480a34d9dd9da3c5299aa8b33db321> 0 hops away, received on LocalInterface[39706]
[2022-08-24 16:04:33] [Debug] Destination <d9480a34d9dd9da3c5299aa8b33db321> is now 0 hops away via <d9480a34d9dd9da3c5299aa8b33db321> on LocalInterface[39706]
[2022-08-24 16:04:33] [Debug] Rebroadcasting announce for <d9480a34d9dd9da3c5299aa8b33db321> with hop count 0
[2022-08-24 16:04:34] [Extra] Completed announce processing for <d9480a34d9dd9da3c5299aa8b33db321>, retry limit reached

[2022-08-24 16:09:13] [Debug] Started shared instance interface: Shared Instance[37428]
[2022-08-24 16:09:13] [Extra] Cleaning resource and packet caches...
[2022-08-24 16:09:13] [Verbose] Bringing up system interfaces...
[2022-08-24 16:09:13] [Extra] AutoInterface[Default Interface] No link-local IPv6 address configured for lo, skipping interface
[2022-08-24 16:09:13] [Extra] AutoInterface[Default Interface] Selecting link-local address fe80::6eb4:f709:3a59:af2%wlan0 for interface wlan0
[2022-08-24 16:09:13] [Extra] AutoInterface[Default Interface] Creating multicast discovery listener on wlan0 with address ff12:0:d70b:fb1c:16e4:5e39:485e:31e1
[2022-08-24 16:09:13] [Verbose] AutoInterface[Default Interface] discovering peers for 1.5 seconds...
[2022-08-24 16:09:15] [Verbose] System interfaces are ready
[2022-08-24 16:09:15] [Verbose] Configuration loaded from /home/kelbot/.reticulum/config
[2022-08-24 16:09:15] [Verbose] Loaded 3 known destination from storage
[2022-08-24 16:09:16] [Verbose] Loaded Transport Identity from storage
[2022-08-24 16:09:16] [Verbose] Loaded 0 path table entries from storage
[2022-08-24 16:09:16] [Verbose] Loaded 0 tunnel table entries from storage
[2022-08-24 16:09:16] [Verbose] Transport instance <55dae89899b6d3f28044a7b321776c6b> started
[2022-08-24 16:09:16] [Notice] Started rnsd version 0.3.11
[2022-08-24 16:09:23] [Extra] Accepting new connection to shared instance: LocalInterface[34786]
[2022-08-24 16:09:24] [Extra] Valid announce for <f0deae1906c97b77c324b72b2e92acfe> 0 hops away, received on LocalInterface[34786]
[2022-08-24 16:09:24] [Debug] Destination <f0deae1906c97b77c324b72b2e92acfe> is now 0 hops away via <f0deae1906c97b77c324b72b2e92acfe> on LocalInterface[34786]
[2022-08-24 16:09:24] [Extra] Valid announce for <40839d2cb9aa9cdf3b49aae9f87050c4> 0 hops away, received on LocalInterface[34786]
[2022-08-24 16:09:24] [Debug] Destination <40839d2cb9aa9cdf3b49aae9f87050c4> is now 0 hops away via <40839d2cb9aa9cdf3b49aae9f87050c4> on LocalInterface[34786]
[2022-08-24 16:09:24] [Extra] Valid announce for <b9bd7b8721c46b26ddcfc18388ea61a1> 0 hops away, received on LocalInterface[34786]
[2022-08-24 16:09:24] [Debug] Destination <b9bd7b8721c46b26ddcfc18388ea61a1> is now 0 hops away via <b9bd7b8721c46b26ddcfc18388ea61a1> on LocalInterface[34786]
[2022-08-24 16:09:27] [Extra] Valid announce for <40839d2cb9aa9cdf3b49aae9f87050c4> 0 hops away, received on LocalInterface[34786]
[2022-08-24 16:09:27] [Debug] Destination <40839d2cb9aa9cdf3b49aae9f87050c4> is now 0 hops away via <40839d2cb9aa9cdf3b49aae9f87050c4> on LocalInterface[34786]
[2022-08-24 16:09:28] [Debug] Rebroadcasting announce for <40839d2cb9aa9cdf3b49aae9f87050c4> with hop count 0
[2022-08-24 16:09:29] [Extra] Completed announce processing for <40839d2cb9aa9cdf3b49aae9f87050c4>, retry limit reached
[2022-08-24 16:09:30] [Extra] Valid announce for <b9bd7b8721c46b26ddcfc18388ea61a1> 0 hops away, received on LocalInterface[34786]
[2022-08-24 16:09:30] [Debug] Destination <b9bd7b8721c46b26ddcfc18388ea61a1> is now 0 hops away via <b9bd7b8721c46b26ddcfc18388ea61a1> on LocalInterface[34786]
[2022-08-24 16:09:31] [Debug] Rebroadcasting announce for <b9bd7b8721c46b26ddcfc18388ea61a1> with hop count 0
[2022-08-24 16:09:32] [Extra] Completed announce processing for <b9bd7b8721c46b26ddcfc18388ea61a1>, retry limit reached
[2022-08-24 16:09:44] [Extra] Valid announce for <f0deae1906c97b77c324b72b2e92acfe> 0 hops away, received on LocalInterface[34786]
[2022-08-24 16:09:44] [Debug] Destination <f0deae1906c97b77c324b72b2e92acfe> is now 0 hops away via <f0deae1906c97b77c324b72b2e92acfe> on LocalInterface[34786]
[2022-08-24 16:09:45] [Debug] Rebroadcasting announce for <f0deae1906c97b77c324b72b2e92acfe> with hop count 0
[2022-08-24 16:09:46] [Extra] Completed announce processing for <f0deae1906c97b77c324b72b2e92acfe>, retry limit reached
[2022-08-24 16:09:51] [Extra] Valid announce for <f0deae1906c97b77c324b72b2e92acfe> 0 hops away, received on LocalInterface[34786]
[2022-08-24 16:09:51] [Debug] Destination <f0deae1906c97b77c324b72b2e92acfe> is now 0 hops away via <f0deae1906c97b77c324b72b2e92acfe> on LocalInterface[34786]
[2022-08-24 16:09:51] [Debug] Rebroadcasting announce for <f0deae1906c97b77c324b72b2e92acfe> with hop count 0
[2022-08-24 16:09:52] [Extra] Completed announce processing for <f0deae1906c97b77c324b72b2e92acfe>, retry limit reached
^C[2022-08-24 16:10:09] [Debug] Detaching LocalInterface[34786]

[markqvist]

Thanks for providing the logs. That is indeed strange, the AutoInterfaces are not even seeing each other. Do you have client isolation or something similar enabled on the WiFi APs? Or maybe multicast traffic is filtered?

[klundry]

No. I am able to ssh between the two just fine.

[markqvist]

Thanks for all the details. I'll try and replicate the issue to get it fixed. I'll post here when I know something :)

[klundry]

Ok, if there is anything else I need to look into that may be causing issues I am willing to. My setup is nothing unusual though so I'm not sure what it would be.

[markqvist]

What hardware are you running Postmarketos on? Is it a device with a non-mainline kernel? I am just guessing here, but maybe it is using an android-based kernel with no IPv6 multicast support. I have not been able to reproduce this on Manjaro alone, and unfortunately, I don't have access to any devices that will run postmarketos right now.

I don't know if it is possible for you to do a wireshark capture at both ends for the wifi interfaces for the announce, and send me those. Is there an option in your WiFi AP or router to block multicast traffic? If so, try turning it off. The AutoInterface in Reticulum relies on multicast to discover peers.

[klundry]

This is on a Pinephone. The kernel package is linux-postmarketos-allwinner-5.19.2 and it's not android based I don't think. I believe it's pretty close to mainline. I will try another device and see if I have any luck.

[markqvist]

Thanks for the details. It would be very nice to have it confirmed working on the PinePhone, but unfortunately I don't have one available to test with. Could I ask you to try and set up a TCP server/client connection between the two devices? Just to rule out that it is not something else entirely. It would also be quite strange though, if they had disabled multicast support in the kernel.

[klundry]

What do you mean by TCP server client connection?

[markqvist]

Configuring the interfaces manually between the two devices, and seeing if that works. So, for example, set up a TCPServerInterface on the Manjaro system, and configure a TCPClientInterface on the PinePhone. Both are added by editing the Reticulum config file. This is just to check that communication is actually working if the two devices have been manually configured to connect to each other. If that test succeeds, we can be pretty sure it has something to do with multicast traffic not being processed as intended.

[klundry]

This worked and they are now connected.

[markqvist]

Ok, thanks for trying it out. Then it is almost certainly an issue with multicast traffic not being processed correctly. The question remains whether this lies in PostmarketOS, in the PinePhone kernel, or in the WiFi AP(s) you are using.

If you are willing to share it, could you let me know what WiFi access points you are using?

[klundry]

I use one of these. https://freemeshwireless.com/ It runs openWRT.

[markqvist]

Ok, I am pretty sure it is not going to be blocking multicast then :)

Do you have another computer running Linux you could test it with? As in, trying out using only the AutoInterface between the Manjaro system and another Linux system? Just to confirm completely that the automatic peering actually works over your WiFi.

[klundry]

Alright, I'm trying to install nomadnet on two different machines with pip but I'm having the same issue with both. Failures pointing at cffi. I successfully installed nomadnet on two different machines a couple weeks ago no problem.

[markqvist]

If it is on ARM machines, you might need to look here: https://markqvist.github.io/Reticulum/manual/gettingstartedfast.html#reticulum-on-arm64

There's not always precompiled Python wheels for the dependencies of RNS on ARM, and on some distros, the build tools/headers are not installed by default.

Alternatively, just for testing, you can try with the rnspure package from pip, which has no binary dependencies. It is much slower though, since all crypto primitives are in pure Python.

[klundry]

I installed python3-dev and got farther. Now it says "error: can't find Rust compiler". It must be debian doing things differently than alpine/pmOS because I had none of these issues when installing it on the Pinephone and it is arm64 as well.

[klundry]

Ok, it's cryptography that is failing because of debian buster not having a new enough rust package I think. I need to upgrade this this machine to bullseye anyways so I'll do that soon and give it another try. I think I have one other computer I can try on in the mean time.

[markqvist]

Thanks for all your effort in debugging this :)

[klundry]

Ok, still failing to install with pip on an arm v7 device running debian bullseye. Suppose I should probably open a separate issue for the issues getting it installed. But so far I haven't been able to get it installed on anything else I have to be able to test the auto interface discovery further.

[markqvist]

The ARM Debian build scripts mentioned here might help you out:
#97

[markqvist]

I'm closing this as an issue now, and migrating it to the discussion forum, since I am quite certain it is not an issue with nomadnet/rns, but a problem with multicast on the pinephone. Hopefully more info crops up, and if there is a way to implement a workaround, we definitely should.