Support routing .onion traffic over tor

[richvdh]

It would be nice to be able to configure synapse so that it can federate both with servers which are on .onion addresses, and those which are on regular clearnet addresses. Currently this is tricky, because there is no way to tell synapse to route some hostnames via a tor proxy whilst routing others as normal

[vsatmydynipnet]

IMHO there would be need to:

• enable Tor Y/N
• settings for IP and Port of Tor socks Proxy. IP because it must not run on localhost
• disable valid cert for Tor or accept clearnet cert for xyz1234.onion
• alias xyz1234.onion for mydomain.com as user alias if routed through tor

[vsatmydynipnet]

Remark: Tor is actually the only stable solution for real federation. There are alot of systems able to run synapse out there, but people have connectivity problems. IPv6 will not be available to the masses within the next 10 years, so Tor is the only chance for all people to run their own Synapse Homeserver.

[cyphar]

I was just about to open an issue for this. There are a few things that would be important to implement:

• Support using a proxy for all outgoing connections as a homeserver. This would be required for the .onion Matrix server to have any hope of being anonymous -- otherwise all federation out of the Matrix server (or through the Matrix server) would trivially reveal its IP address.

• There is an argument that it should be possible to set (in a room's options) that only users which are on .onion homeservers are allowed to join (sort of like the "this room is not federated" option today) and for the server to have a configuration which only allows federation with .onion homeservers (but I think you can do that already with the federation whitelist). This way you can set up conversations that explicitly only will be shared by anonymous homeserver users.

• Get most (if not all) homeservers to route .onion addresses. While I believe messages transit through federation as a broadcast, you don't want there to be only a few hot servers which route .onion traffic (not just because it becomes a pretty serious single point of failure but also for performance reasons). However, we should probably allow for .onion routing to be disabled since some users might not feel comfortable running a Tor client (even though there should be no reason to worry about this).

And it would be nice for clients to indicate when they are in a .onion room or if a .onion user is talking. IRC clients have in the past had issue with Tor users (effectively) social-engineering conversations to go to certain topics, but marking users as being anonymous would be useful. Clients should also tell users to use Tor if they're trying to log into a .onion homeserver.

[Thatoo]

Is there any news on this topic?

[sneak]

I'm also interested in running a .onion homeserver - am happy to contribute on this topic in any way I can.

[dr-bonez]

I'm also looking into how to do this. Main features necessary for implementation seem to be adding SOCKS5 proxy support and disabling SSL for .onion urls. I'm not a huge python guy so I'm not sure, but it looks like changes for this could be confined to this file. Help from anyone with a better orientation of this codebase would be greatly appreciated with respect to integrating this.

[jh72de]

This would de-anonymize each server that is set-up to do this (bridge traffic between .onion and clearnet) as well as expose metadata of all the clients of each of these servers.

[dr-bonez]

Anyone who is bridging traffic between .onion and clearnet I'm assuming doesn't care about anonymity and probably is running their homeserver on clearnet. It would not reveal the IP addresses of .onion homeservers that route all traffic over Tor.

[cyphar]

The simplest solution would be to just add Tor (as a client) to every synapse server which would avoid any privacy issues, but this might not be practical (some people may have concerns about running Tor on their homeservers -- in some countries, running Tor at all may result in a visit from the authorities). And if not all servers run Tor, then we should avoid fragmenting the federation (it probably isn't good for the Matrix ecosystem if you end up in a situation where publicly-federatable servers can only be accessed by users on some homeservers).

[dr-bonez]

it probably isn't good for the Matrix ecosystem if you end up in a situation where publicly-federatable servers can only be accessed by users on some homeservers

why would this be an issue? especially since the alternative is those homeservers just not existing? Pretty sure matrix will handle this pretty gracefully. Ideally, big homeservers like matrix.org would have tor support, and if a user is running against a small homeserver that isn't running tor, and can't connect to rooms hosted on a .onion, the error would be pretty expected. Perhaps there is something about how matrix federation works that I'm missing.

[vsatmydynipnet]

as mentioned in may 2019 above i still think matrix has massiv need for Tor routing. Especially where Dendrite will be able to run a homeserver as single binary on all OS soon, Tor is THE ONLY solution which allows connects to the HS without port forwarding, NAT and dynamic IP problems.

The is NO practicable soltuion for normal users to run their HS without Tor. And please do not name strange UDP something P2P stuff which is not even fully understood by profis. For non techies this will never be a replacement and these ones are also to easy blockable by providers aso. . Especially where Tor Libs exists for Go and others. So it would be possible to create a single static binary without extra Tor setup needed.

Federation as I see it is the possibility for EVERYBODY to run his own HS for himself or the familiy/friends. This needs Tor otherwise it will not work for non techies.

And in that case, anonymisation in not priority 1, it is the possibility to get a jfkslkjflks.onion address which is reachable without any extra setup. And it works everywhere, kinda unblockable compared to UDP stuff which is blocked in probably 90% of the firewalls anyway.

Would love to see this would be built in.

[xshadow]

I would be interested as well :)

[ghost]

Interested a lot. We need that feature

[garlicgambit]

Not sure if this is the right place to ask.

Is it currently possible to setup a (non-anonymous) Tor onion service for Synapse? The location of the server doesn't need to be protected and server-to-server traffic doesn't need to be routed over Tor. Only the Matrix clients need to be able access the server via a .onion service.

[xshadow]

Not sure if this is the right place to ask.

Is it currently possible to setup a (non-anonymous) Tor onion service for Synapse? The location of the server doesn't need to be protected and server-to-server traffic doesn't need to be routed over Tor. Only the Matrix clients need to be able access the server via a .onion service.

Except cloudflares global onion service, which is enabled for matrix.org, I haven't seen any implementation in the wild, where you have an additional onion service next to your public matrix service.

[rwjack]

Let's make this happen.

It is pretty pointless having a secret server for people to chat on, without the server being anonymous.

TOR is the ideal option.

[wi18b088]

Is there any progress on this? Are SOCKS5 proxies already supported in either Synapse or Dendrite? This would be the only missing piece right now for federating .onion homeservers without clearnet federation. TLS checks can be disabled already on both.

[jeanlucmongrain]

@wi18b088 #10475

[w1ldrabb1t]

Still looking for this in 2022. Any news/updates about supporting routing .onion over tor?

[Polve]

Support for I2P would be even better!

[wi18b088]

For those of you who are happy to tinker, you could use an HTTP to SOCKS proxy wrapper like polipo (no longer maintained) for a proof of concept. Haven't tried it myself but there are some guides out there.
Both Synapse and Dendrite support outgoing HTTP proxies.
https://www.marcus-povey.co.uk/2016/03/24/using-tor-as-a-http-proxy/

[redragonx]

Please add tor support 👍

[Bertrand256]

EmbassyOS has been supporting the Synapse Matrix server over Tor for quite some time now. I tested it a bit and it works fine.
Their GitHub page: https://github.com/Start9Labs/embassy-os

[ghost]

I give you a federation-over-Tor proof-of-concept.

It wasn't easy, takes three times more work than a standard instance, but let's try to enumerate the steps and propose further improvements to make the process easier for others.

1. Obviously, we need at least two homeservers. Both will need tor client and privoxy. For rev-proxy I used caddy, for resolving DNS I simply fiddled with Tor and resolv.conf, but you can set up bind or even a whole separate machine.
2. In Synapse config, these are the interesting parts, let's discuss them:

server_name: "X.onion"
listeners:
  - port: 8008
    tls: false
    type: http
    x_forwarded: true
    bind_addresses: ['::1', '127.0.0.1']
    resources:
      - names: [client, federation]
        compress: false
trusted_key_servers:
  - server_name: "matrix.org"
    accept_keys_insecurely: true

allow_public_rooms_over_federation: true
use_insecure_ssl_client_just_for_testing_do_not_use: true
federation_verify_certificates: false
tls_certificate_path: /etc/.../cert.pem
tls_private_key_path: /etc/.../key.pem

The domain name (X) comes from tor client, it's a well-documented process, I'll skip that here. The server backend runs on unencrypted and uncompressed at 127.0.0.1:8080 as it usually does with Reverse Proxy, perhaps you can expose it directly to Tor, good luck. I had to turn on some generally insecure options, namely accept_keys_insecurely, use_insecure_ssl_client_just_for_testing_do_not_use and federation_verify_certificates as explained below.
3. The certificates I had to generate myself:
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes -keyout key.pem -out cert.pem -subj "/CN=example.com" -addext "subjectAltName=DNS:X.onion"
using, of course, the X domain name returned by Tor. Why use these at all? Well, it's a good question. The first Synapse suggestion would be to stop relying on https in federation. There is not much point in it since I use Tor addresses anyways and certs being self-signed yield various complications:
a) To connect to my node with Element (or any electron-based client), I need to run it with --ignore-certificate-errors. I can, however, run the client-facing site on plain http, but it would be silly since I have to encrypt :8448 traffic anyway.
b) Caddy won't just let me use any self-signed certificate, the SAN list needs to match my domain. It also won't auto-create it with zerossl, as they don't support .onion domains.
c) Synapse also checks certificates and I had to flip those unsafe options. For non-over-Tor federations that would be a risk, so at minimum, Synapse should recognize .onion addresses and allow unencrypted/unverified-certificate connections with them.
4. Tor configuration seems pretty straightforward:

HiddenServiceDir /somewhere
HiddenServicePort 80 127.0.0.1:80
HiddenServicePort 443 127.0.0.1:443
HiddenServicePort 8448 127.0.0.1:8448
SOCKSPort 127.0.0.1:9050
AutomapHostsOnResolve 1
DNSPort 127.0.0.2:53

Obviously, after first run it creates the hostname file in /somewhere which you can use to configure the rest. I open three ports here: 80 for normal Tor traffic, 443 so that Element can connect to my site without prepending the address with "http://" and 8448 for federations. Socks is necessary for privoxy, and thus, for federations. I'm not sure if AutomapHostsOnResolve is now required, there were some DNS resolution errors from Synapse when I experimented, but at least I won't forget it now. And the most interesting one: DNSPort 127.0.0.2:53 - you can ask: why not 127.0.0.1:9053? Well, because you cannot use 127.0.0.1#9053 as a parameter to one of the python libraries used in Synapse, that's why.
5. Privoxy is easy:

listen-address  127.0.0.1:8118
forward-socks5t / 127.0.0.1:9050 .

What it does is using tors SOCKS proxy and wraps it into a nice http proxy that Synapse can understand. Just remember to export:

export http_proxy=http://127.0.0.1:8118
export https_proxy=http://127.0.0.1:8118

in your system service definition when running Synapse so that it can connect to your overseas homeservers. Normal, non-over-Tor federations are easy without it, but the receiving party needs to understand that the inbound connection is coming from .onion server, otherwise you will get a 401 error in Synapse logs.
6. Finally, Caddyfile:

X.onion:8448 {
	tls /etc/.../cert.pem /etc/.../key.pem
	reverse_proxy http://localhost:8008
}

X.onion:80 {
	redir https://X.onion
}

X.onion:443 {
	tls /etc/.../cert.pem /etc/.../key.pem
	encode gzip
	reverse_proxy /_matrix/* http://localhost:8008
	reverse_proxy /_synapse/* http://localhost:8008
	handle_path /.well-known/matrix/* {
		respond /client ` {
			"m.homeserver": {
				"base_url": "https://X.onion"
			}
		} `
		respond /server ` {
			"m.server": "X.onion:443"
		} `
		respond /support ` { ... }`
	}
}

It handles a few things. Since we are forced to run at port 443 (Element will let you register/login to a non-https homeserver if you prepend the address with http:// but it will malform a http://X.onion address into matrix:httpX.onion when you try to discover rooms on a homeserver with such address), let's just redirect all :80 connections to :443. On :8448 we handle federation traffic and it's encrypted since Synapse requires it. The main configuration needs to include .well-known delegation since we don't have SRV records in Tor. And since federation failed on this lack of SRV support in Tor, I added a small patch to Synapse to make it not fail the connection over tor: #15523
7. To allow Synapse resolving .onion domains, the DNS must support it. Since we enabled it in torrc, just let the system resolver know it by putting nameserver 127.0.0.2 in /etc/resolv.conf (works with Windows too). There is some redundancy of this step with step 5 but I didn't really get deep into that, it seems plausible to assume that privoxy should provide some mechanism for this.
9. What works, what doesn't? I was able to create a room, join it from another homeserver, chat a little. I didn't try VoIP (no TURN server) but I expect it to be possible over Tor in the foreseeable future. I didn't try integrations and bots, as I don't expect it to be any different. The performance is not great, not terrible. Obviously, double traffic encryption doesn't help. I didn't profile CPU usage neither.

I'm not saying this guide is the state-of-art thing. Perhaps I've been working with Synapse instances suboptimally, maybe some config is redundat or plainly wrong. More ideas are welcome, I will be monitoring this thread as I expect this to be a game-changing feature. Maybe someone can now create a Dockerfile based on my hints and publish it here, I will be happy to connect with more instances from my machine.

Edit: changed SOCKSPort and privoxy socks target to 127.0.0.1 so it's not exposed to the network.

[k0gen]

EmbassyOS has been supporting the Synapse Matrix server over Tor for quite some time now. I tested it a bit and it works fine. Their GitHub page: https://github.com/Start9Labs/embassy-os

That’s correct, the Synapse over Tor wrapper repository is available at https://github.com/Start9Labs/synapse-wrapper

I give you a federation-over-Tor proof-of-concept.

@zdzichu-rks great work!
We have a Start9 - Tor Party Room that you can use to test federation over tor with your instance:
#tor-party:matrix.privacy34kn4ez3y3nijweec6w4g54i3g54sdv7r5mr6soma3w4begyd.onion

[ghost]

We have a Start9 - Tor Party Room that you can use to test federation over tor with your instance:

Yup, it does work, however the amount of Synapse Errors is worrysome:

      4 synapse.federation.sender.per_destination_queue
     10 synapse.handlers.federation
     22 synapse.replication.tcp.handler
     34 synapse.http.matrixfederationclient
     55 synapse.media.media_repository
     93 twisted
    131 synapse.handlers.federation_event
    139 synapse.crypto.keyring
    334 synapse.federation.federation_client

For all I know, most homeservers expect inbound SSL connections from the address they can resolve and fetch server keys. Privoxy connection limit gets exhausted pretty fast and it took over 10 minutes to load room history. I suppose Synapse should at least gracefully acknowledge some other server can't understand Tor and ignore it.

[ghost]

My patch is now live in https://github.com/matrix-org/synapse/releases/tag/v1.84.0, perhaps anyone could now give feedback if this is working for them without hiccups.

Not saying this ticket should be closed yet, but perhaps a definitive TODO list could be now compiled and tracked.

[ghost]

For a not-just-Tor instance, which doesn't care about its own anonymity, the following differences make sense:

1. In privoxy.conf:

forward-socks5t .onion  127.0.0.1:9050 .

so that non-.onion traffic is not routed trough Tor, just making it faster.
2. In torrc we skip all the HiddenService... lines, as we don't host any hidden site. Obviously, we still need some DNS domain, certificates and such, just as regular instance should already have set.
3. In homeserver.yaml, still need:

trusted_key_servers:
    accept_keys_insecurely: true
use_insecure_ssl_client_just_for_testing_do_not_use: true
federation_verify_certificates: false
suppress_key_server_warning: true

Which is the main issue here - as we break SSL so that MITM attacks are now possible. It's still encrypted and will work just fine until someone wants to impersonate another instance, but Synapse should allow finer configuration, e.g. just limiting this to .onion connections.
4. Env vars are still useful, on systemd-based systems just add these lines to /etc/systemd/system/multi-user.target.wants/matrix-synapse.service in [Service] section:

Environment="export http_proxy=http://127.0.0.1:8118"
Environment="export https_proxy=http://127.0.0.1:8118"

5. /etc/resolv.conf is in fact frequently overwritten. Just run chattr +i /etc/resolv.conf after your changes. This could also be handled better by Synapse - if run with Tor, just use it for name resolution instead of system-wide DNS settings.
6. With these changes, I observed that for some reason, connections to 127.x.y.z are being blacklisted by Synapse, perhaps under some assumption it is not an actual network connection. No worries, in homeserver.yaml:

ip_range_whitelist:
  - '127.0.0.0/8'
  - 'fe80::/10'

7. Now, I'm not sure why, I still get tls errors while either trying to list another homeserver's rooms, or just joining a room there. My hybrid node still cannot connect to X.onion:8448/ : An error occurred while connecting: 22: Invalid argument.. It comes from synapse.http.federation.matrix_federation_agent, the _do_connect function, which might be another candidate for my patches, e.g. if it's not respecting ssl certificate ignorance flags set above. But after all, my T2T (tor2tor) node is working fine, so I'm not sure why it only occurs now. The only real difference is privoxy setting and I already checked if using the / as target pattern helps, it doesn't.

[ghost]

Scratch 4. above, it should be (no exports):

Environment="http_proxy=http://127.0.0.1:8118"
Environment="https_proxy=http://127.0.0.1:8118"

Other than this, pay attention to what your rev-proxy (Caddy in my case) certs configuration is on your .onion service. In general, I'm not saying you can't run with http-only, something like:

http://X.onion:8448 {
        reverse_proxy http://localhost:8008
}

X.onion:443 {
       redir http://X.onion
}

X.onion:80 {
...

But this probably will fail on the redir part, as Synapse's federation agent will simply fail when presented with Caddy's default or no certificate. Instead, my self-signed config from the first post above seems to do just fine.

There is also a possibility to configure client and federation listeners separately, e.g. by letting clients to use non-encrypted site and avoid running Element with --ignore-certificate-errors but I haven't got that much free time.

It seems that both use cases (T2T and hybrid) work fine now and now I really see why don't you try and run it yourselves. It really does bring Matrix experience to the new level, especially if you own a vanity .onion domain.

[ghost]

My further experiments led to two new points:

1. Synapse documentation is useful. One can use:

federation_verify_certificates: true
federation_certificate_verification_whitelist:
  - "*.onion"

Seems pretty self-explanatory. Useful both for T2T and hybrid instances. Completely fixes my headaches regarding federation MITM attacks.

2. Synapse documentation is inaccurrate. https://matrix-org.github.io/synapse/latest/setup/forward_proxy.html doesn't mention, that we can also add:

Environment="matrix_proxy=http://127.0.0.1:8118"

To actually use proxy (and Tor!) while collecting keys from other instances via :8448. It seems to use urllib.request.getproxies_environment underneath which will collect any env var and try to match it with schemes like http, https, and matrix. Since I didn't have matrix proxy set up, my federations failed in synapse.http.federation.matrix_federation_agent, now they're fine.

Seems I can now join, browse public rooms, pull media in both directions and none of my instances yields any warnings related to the other one. Will monitor these further.

[ghost]

There is some issue with joining some spaces based on hybrid instance from my T2T instance if there are users from another not-tor-aware instances already in that space, effectively blocking me from joining it.

On hybrid instance the log shows:

2023-06-07 XX:XX:XX,XXX - synapse.federation.sender.transaction_manager - 178 - WARNING - federation_transaction_transmission_loop-6 - TX [foreign.nontor.instance] {X} Remote returned error for $some_event_id: {'error': "ERROR 403: unable to verify signature for sender domain mytorinstance.onion: 401: Failed to find any key to satisfy: _FetchKeyRequest(server_name='mytorinstance.onion', minimum_valid_until_ts=X, key_ids=['ed25519:a_bRKp'])"}

And then after a timeout, my T2T client gets:

MatrixError: [403] Auth events could not be found (https://mytorinstance.onion/_matrix/client/r0/join/%23some-room%3Amy.hybrid.instance?server_name=my.hybrid.instance)

I suppose there is a bug in spaces synapse code that fails the join if some foreign host doesn't acknowledge the event, while with joining regular rooms there is no such issue.

It might help if the foreign instance had my hybrid instance set as a trusted key server, but nobody is going to do that.

[ajohns92a]

Following the steps above, it seems largish media (6mb+) shared on one TOR server is not able to be retrieved by another federated tor server. Smaller media federates just fine between the two. The larger file can be successfully downloaded by users on the same HS that the uploader used.

Are there any tweaks that could be implemented to get around this?

Environment:
Synapse 1.86
Debian
TOR
Privoxy

Clients used:
Element (Web)
Schildi (Web)
Element (Desktop)

Tweaks so far:

Homeserver.yaml:

federation:
  client_timeout: 10m
  max_short_retry_delay: 5m
  max_long_retry_delay: 10m
  max_short_retries: 5
  max_long_retries: 20

Nginx conf:

    proxy_request_buffering off;
     sendfile on;
     tcp_nopush on;
     tcp_nodelay on;
     keepalive_timeout  65;
     client_body_timeout 600;
     send_timeout 600;
     proxy_read_timeout 600;

Error received
Either 62b file download - or -
{"errcode":"M_UNKNOWN","error":"Failed to fetch remote media"}

[Buttars]

Anyone who's backing a TOR only solution remember that TOR has the possibility of being compromised. We should be embracing a network agnostic approach. It seems like there are several distinct but nonexclusive use cases that need to be addressed individually.

1. Keeping homeserver private
2. Keeping users connecting to the homeserver private
3. Restricting the homeserver to only users on private networks

The above POC shows potential.

I'd like an option that allows administrators to:

1. Require clients to use TOR exclusively, or
2. Accept both TOR and clearnet clients at the same time.

And

An option to

1. Force all non-client traffic to go through the TOR network, or
2. Map non-client TOR traffic to TOR and clearnet traffic to clearnet (enabling communication to anonymous only homeservers i.e. servers that only exist as a hidden service while retaining the ability to sync with clearnet servers)

And

An option to block all traffic not coming from certain domains/protocols (TOR/I2P) to enforce anonymous only homeservers. Maybe this should be left to the server admin to configure on their network and not a consideration of the Synapse server at all?

It would also be really nice to be able to make the implementation agnostic to the anonymization network and even be able to run both TOR and I2P for larger shared homeservers. There are valid criticisms of all of the available anonymization networks and the end users should be allowed to choose which works best for them based on their requirements and threat model.

Since this issue is specifically about TOR there is no reason to discuss I2P specifically but please take other anonymization networks into consideration when coming up with an approach.