not all punycode-encoded domains work

[Sorunome]

Description

If you have a domain with only emoji and want to set your server_name to the punycode-encoded versoin of it (e.g. xn--9s9hnf.ws) the server is not able to federate.

This is because twisted.web.client uses the python package idna over here to check if a domain is valid. Editing that to use text.encode("idna") and text.decode("idna") fixes this issue.

Said idna package has an issue here about not supporting emoji domains, and from the sounds of it they do not plan to add it.

Soru would open this issue with the twisted package, but she couldn't find any way to open non-security-critical bug reports with them. This issue also affects synapse as, well, that does not allow you to have all punycode-encoded domains as your server_name, even though they are valid according to the spec etc.

[clokep]

If you have a domain with only emoji and want to set your server_name to the punycode-encoded versoin of it (e.g. xn--9s9hnf.ws) the server is not able to federate.

From reading through the linked issue it seems this is an invalid domain name, but domains similar to this exist "due to other non-standards compliant software".

Soru would open this issue with the twisted package, but she couldn't find any way to open non-security-critical bug reports with them.

See https://twistedmatrix.com/trac/newticket

FTR, xn--9s9hnf.ws converts to 🧡🦊.ws.

It is unclear to me whether these should be supported or not.

[Sorunome]

From reading through the linked issue it seems this is an invalid domain name, but domains similar to this exist "due to other non-standards compliant software".

Well, they work in normal webbrowsers, many even convert emojis to punycode for you. The fact that you can register such domains should make them valid, in sorus opinion.

See https://twistedmatrix.com/trac/newticket

Thank you!

[auscompgeek]

I'm seeing a UnicodeEncodeError for that particular server name as well.

https://sentry.io/share/issue/46729566e17f4ce19b1b9af751888b68/

The offending line (on HEAD as of writing) is:

synapse/synapse/http/matrixfederationclient.py

Line 129 in a802606

destination_bytes = self.destination.encode("ascii")

I'm not sure what the consequences of changing the encoding to utf-8 here would be though.

[Sorunome]

Soru experimented with putting the actual emoji in the server_name, which would result in the error you just mentioned. This issue is about the punycode-encoded server_name, though, and is thus something totally separate expecting servers to punycode-encode the server_name on'the-fly would likely need an msc. 29. 12. 2020 8:13:24 David Vo <notifications@github.com>:

…

I'm seeing a UnicodeEncodeError for that particular server name as well. https://sentry.io/share/issue/46729566e17f4ce19b1b9af751888b68/ The offending line (on HEAD as of writing) is: https://github.com/matrix-org/synapse/blob/a8026064755362fe3e5dc00f537606d340ce242a/synapse/http/matrixfederationclient.py#L129 I'm not sure what the consequences of changing the encoding to utf-8 here would be though. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub[#8991 (comment)], or unsubscribe[https://github.com/notifications/unsubscribe-auth/AASSEVHAEBWU2JVDF6L4GLTSXF6RFANCNFSM4VKEV4AQ]. [data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAEgAAABICAYAAABV7bNHAAAAAXNSR0IArs4c6QAAAARzQklUCAgICHwIZIgAAAArSURBVHic7cEBDQAAAMKg909tDjegAAAAAAAAAAAAAAAAAAAAAAAAAAA+DFFIAAEctgHwAAAAAElFTkSuQmCC###24x24:true###][Sledovací obrázek][https://github.com/notifications/beacon/AASSEVAVKOGNFJH5H2V5ETLSXF6RFA5CNFSM4VKEV4A2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOFTJD3DQ.gif]

[Sorunome]

Managed to create a ticket by now: https://twistedmatrix.com/trac/ticket/10078#ticket

[clokep]

It seems kjd/idna#18 is the eventual upstream for this.

[clokep]

Managed to create a ticket by now: twistedmatrix.com/trac/ticket/10078#ticket

For cross-linking, this is now twisted/twisted#10078.