Support for SMTPUTF8

[wez]

This issue is tracking what would be necessary to implement SMTPUTF8 support.

• 8BITMIME is necessary in order to receive 8-bit content.
  • receive 8-bit DATA without complaining (I think this is already the case, but need to verify)
  • at send time check whether the content is 8-bit and either:
    • Augment MAIL FROM with the BODY=8BITMIME parameter if the destination also supports 8BITMIME
    • or: transform via equivalent to check-fix-conformance to 7-bit. This transformation will invalidate signatures.
    • or: fail the message
• Augment the SMTP command parser to allow:
  • 8-bit local parts in mailboxes
  • 8-bit domain parts. Probably should implicitly convert these via IDNA to 7-bit form and carry that through
• Trace headers must 7-bit encode any envelope information
• Change the SMTP client to check whether the envelope contains 8-bit and:
  • augment the MAIL FROM command to add SMTPUTF8 as a parameter if the destination supports both 8BITMIME and SMTPUTF8
  • otherwise: fail the message

Note that #198 may be necessary for both 8BITMIME and SMTPUTF8 in the case that the content cannot be transformed on the fly for the next SMTP hop and a bounce message is required to be generated.

It is valid for a server to choose not to implement the send-time conversion of a message. That neatly avoids the case where we could silently mangle signed content, and instead generate a bounce/permfail stating explicitly what the problem is. We already provide the ability to transform at reception time via msg:check_fix_conformance() which gives the operator a high degree of control over whether the transformation takes place.

[wez]

The current state here is:

• We will accept 8-bit envelopes, converting the domains to 7-bit form. Local parts are left as-is.
• We will accept 8-bit data, and your local policy can then choose to call message:check_fix_conformance() to convert it to 7-bit
• At send time:
  • if we still have an 8-bit envelope (eg: local part is 8-bit), then if the destination advertises SMTPUTF8, we will add that extension to the MAIL FROM command
  • if we still have 8-bit data, then if the destination advertises 8BITMIME, we will add that extension to the MAIL FROM command

We are not strictly conforming to either 8BITMIME or SMTPUTF8, so we do not in turn advertise either of these extensions.

The conformance gap is:

• At send time, if we have 8-bit data and the remote system doesn't advertise the corresponding SMTP extension, the RFC says that we must not try to send the data. What we should do in that situation is either:
  • automatically down-convert (which isn't possible for the envelope, and which will invalidate DKIM signatures for the body)
  • fail the message send with an appropriate error message

We're punting on closing that gap for the moment until we get some more feedback on this.

Pragmatically speaking, we want to avoid an automatic conversion at send time because it isn't guaranteed to be 100% content-preserving. In terms of generating an error message, the destination will generate one of its own if it doesn't like what we sent.

What we have in place right now is probably close enough for most purposes, so if you wish to advertise these extensions in your specific instance you can use the following snippet in your init.lua:

    kumo.on('smtp_server_ehlo', function(domain, conn_meta, extensions)
      table.insert(extensions, 'SMTPUTF8')
      table.insert(extensions, '8BITMIME')
      return extensions
    end)