Investigate why smtps fails but smtp does not

[aeneasr]

Describe the bug

https://community.ory.sh/t/unable-to-send-email-using-smtp-connection/2252/12

[johanforssell]

I can verify that smtps against https://sparkpost.com/ does not work, but plaintext smtp does. I'm getting this same error

level=error 
msg=Unable to send email using SMTP connection.
...
error=map[message:tls: first record does not look like a TLS handshake] 
...

My demo system is reverse proxied behind a Nginx, if that make any difference.

I also verified with swaks that I can actually use smtps with StartTLS against Sparkpost

swaks -server smtp.eu.sparkpostmail.com:587 -tls \
  --auth-user SMTP_Injection --auth-password bla-bla-bla \
  --to my@email.com --from my@email.com

[CMCDragonkai]

I'm getting no errors, but no email at all:

[nix-shell:~/Projects/formbay-auth]$ echo $COURIER_SMTP_CONNECTION_URI 
smtp://127.0.0.1:1025

With that environment variable after completing the self-service registration flow I get this log:

INFO[2021-06-01T12:10:21+10:00] Sending out verification email with verification link.  audience=audit email_address=Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". identity_id=c7f1d082-cc98-456c-82ef-df0ace695f2e service_name=Ory Kratos service_version=master verification_link_id=87526b72-be85-4063-a203-5acaabfc6479 verification_link_token=Value is sensitive and has been redacted. To see the value set config key "log.leak_sensitive_values = true" or environment variable "LOG_LEAK_SENSITIVE_VALUES=true". via=email

But no errors at all, and nothing.

I'm using MailHog instead of MailSlurper, and I tested it with mailsend and it does work.

mailsend \
    -to target@target.com \
    -from me@me.com \
    -port 1025 \
    -smtp 127.0.0.1 \
    -sub test \
    -M "test message"

[CMCDragonkai]

There's also no logs on the MailHog, whereas usually there would be logs if I access it from another system.

Also if I configure it from the kratos.yaml, still nothing either.

[StarAurryon]

Hello there,

I believe that the tag "legacy_ssl" is a bit misleading and refers more to postfix description (and support in kratos could be potentially dropped). The current trend in CyberSecurity is to avoid STARTTLS when possible and use "Implicit TLS", meaning that the case of SMTP dealing between two domains that are potentially not supporting encryption should be the only one that uses STARTTLS.

In my opinion it would be best to use protocol scheme:

• smtp:// for unencrypted and encrypted smtp with starttls
  • with an option force_starttls as all companies do not have this yet
  • with skip_ssl_verify (already present)
• smtps:// for implicit TLS
  • with skip_ssl_verify (already present)

All of that is described in the RFC at https://datatracker.ietf.org/doc/html/rfc8314
I can provide patches if needed

Best regards

[StarAurryon]

Just as example OVH Cloud, Orange and GMail all use 465 with implicit TLS and do not allow STARTTLS.

[aeneasr]

Thank you for the ideas! Adding legacy to the query param might indeed be misleading - I added it as having this set to true was breaking lots of clients trying to connect to AWS, GCP, ...

I find having smtp:// misleading if we use an encrypted connection. Not sure if that is described in the RFC (couldn't find anything). It probably makes more sense to have explicit naming around force_start_tls. At the same time, the current set up works fine and is documented and breaking the behavior is worse than having a not so nice name!

I would be open however to accept a PR which doesn't break the query param but adds it as a deprecated option (with log output) and that makes it obvious what SSL/TLS methodology is used.

[StarAurryon]

The way SMTP is handled by the RFCs history is quite misleading. For the protocols IMAP and POP for example we have:
pop/imap: cleartext protocol (ports 110/587)
pop/imap with STARTTLS: encrypted protocol after the initial cleartext (ports 110/587) but the initial payload is a STARTTLS keyword defined in the imap protocol.
pops/imaps: implicit encrypted TLS protocol (ports 995/993)

"At the same time, the current set up works fine and is documented and breaking the behavior is worse than having a not so nice name!" => We have full TLS smtp in our production, and when we upgraded from 0.6 to 0.7 it broke.... XD.
So to my opinion the point is to have something you agree before the 1.0 release. Documenting the options for people that don't have or want good encryption is the way to go.

The ideal situation to me and, I made a mistake in my previous post, is to enforce starttls by default on smtp and allow with an option to disable it and use another scheme for smtp with tls as protocols are different (same as http and https). But please note that using cleartext with smtps is also quite misleading and might be skipped during a security review of the configuration.

I worked for three years as a Pentester and a Security consultant and having something that is simple to understand without the documentation is the way to go as administrators do not read the doc so often. So it is best to enforce, like Kratos is doing now, security by default and have naming that is corresponding to conventions or it becomes a mess: first in the mind of the people and then in the security implementation (i.e. configuration).

Happy to discuss,

Regards,

[StarAurryon]

To be more concise i would suggest:

• smtp://server:xxx/?skip_starttls=true&skip_ssl_verify=true
• smtps://server:xxx/?skip_ssl_verify=true

with skip_starttls and skip_ssl_verify at false by default for security.

Or if you want to enforce encryption at all costs and have sth more explicit:

• smtp+starttls://server:xxx/?skip_ssl_verify=true
• smtp+tls://server:xxx/?skip_ssl_verify=true

https://www.ietf.org/rfc/rfc1738.txt

[aeneasr]

To be more concise i would suggest:

• smtp://server:xxx/?skip_starttls=true&skip_ssl_verify=true
• smtps://server:xxx/?skip_ssl_verify=true

with skip_starttls and skip_ssl_verify at false by default for security.

That looks like a good proposal in my opinion! Would you be open to contribute this with:

1. Tests ensuring the options work as advertised
2. An option to habe fully unencrypted SMTP for localhost as supported by the Go stdlib to keep tests working
3. Documentation including explanation given in this thread
4. The current behavior as a deprecated option which be removed once we reach incubation

? :)

[StarAurryon]

I believe we have a deal :)
I will work on this ASAP.

[aeneasr]

Nice :)

[mooijtech]

Hello, also note if you are not seeing any output that SMTP has sent the message make sure you have started the courier via --watch-courier when starting Kratos. See the docs.
I thought there was a problem with my SMTP configuration but turns out I didn't start the courier.