SSL lib does not handle trailing dot (period) in hostname or certificate

[samiam]

BPO	31997
Nosy	@tiran, @alex, @hynek, @dstufft, @samiam
Files	ssl_trailing_dot.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = <Date 2018-02-26.08:19:05.387>
created_at = <Date 2017-11-10.01:54:14.439>
labels = ['expert-SSL', 'type-bug', '3.7', 'invalid']
title = 'SSL lib does not handle trailing dot (period) in hostname or certificate'
updated_at = <Date 2018-02-26.08:19:05.385>
user = 'https://github.com/samiam'

bugs.python.org fields:

activity = <Date 2018-02-26.08:19:05.385>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = True
closed_date = <Date 2018-02-26.08:19:05.387>
closer = 'christian.heimes'
components = ['SSL']
creation = <Date 2017-11-10.01:54:14.439>
creator = 'samiam'
dependencies = []
files = ['47257']
hgrepos = []
issue_num = 31997
keywords = ['patch']
message_count = 8.0
messages = ['306004', '306007', '306008', '306080', '306082', '306238', '306240', '312882']
nosy_count = 6.0
nosy_names = ['janssen', 'christian.heimes', 'alex', 'hynek', 'dstufft', 'samiam']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue31997'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7']

[samiam]

I recently came across an issue in the ssl library and have a simple fix to address it.

When doing hostname verification against an X.509 certificate, a trailing dot (period) in the hostname is matched against the certificate. But the trailing dot should only be applied to the DNS lookup not the certificate match.

Conversely, a certificate that has a trailing dot in its commonName (probably rare but allowed) should match a hostname without the trailing dot. As the ssl library is written now, both cases fail.

The truth table below shows the current python ssl DNS matching behavior.

+----------------------------------------+
| # hostname certificate MATCH |
| +------------------------------------+ |
| | dns dns. cname cname. | |
| +------------------------------------+ |
| 1 x x TRUE |
| 2 x x FALSE |
| 3 x x FALSE |
| 4 x x TRUE |
+----------------------------------------+

Case 1 and 4 currently match as both hostname and certificate strings match exactly when the trailing dot is either present or not.

Case 2 is unlikely, as certificates are rarely signed with a trailing dot in the subject commonName and if they were, clients would ALWAYS have to enter the hostname with a trailing dot to get a match.

Case 3 is more likely where the hostname has a trailing dot, but the certificate does not. For example, "www.example.com." is used for the DNS lookup, but then, "www.example.com." will not match the certificate due to the trailing dot missing from the certificate.

I propose the truth table should be true in all cases and just ignore the trailing dot in both the hostname and certificate.

As best I can tell, the RFCs [1] are silent on this issue. Although the hostname and commonName strings currently must match, there are a couple of precedents where ignoring the trailing dot is done in practice.

Web browsers allow a trailing dot in the URI and will accept a certificate even when the certificate doesn't have a trailing dot. For example, visit Google with trailing dot (https://www.google.com./) from a browser of your choice and check certificate. It should check out as valid. [2]

Also, at least two language SSL libraries, Ruby [3] and Go [4], match certificates when hostnames contain a trailing dot. Lastly cURL [5] ignores trailing dots in certs and hostnames.

In summary, I don't feel the current python ssl library is wrong - it is following an interpretation of the RFC. But I think it can be more permissive, following the spirit of the RFC without sacrificing security.

Patch attached with code change. If accepted, I can do a more formal PR, backport to 2.7 and add tests.

Thoughts?
-Sam

Example of issue
================

Python 3.7.0a2+ (heads/master-dirty:cbe1756e3e, Nov  3 2017, 15:56:14)
[Clang 8.1.0 (clang-802.0.42)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> cert = {'subject': ((('commonName', 'example.com'),),)}
>>> ssl.match_hostname(cert, "example.com")    ## Case 1 from truth table
>>> ssl.match_hostname(cert, "example.com.")   ## Case 3 from truth table
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../cpython/Lib/ssl.py", line 330, in match_hostname
    % (hostname, dnsnames[0]))
ssl.CertificateError: hostname 'example.com.' doesn't match 'example.com'

>>> cert = {'subject': ((('commonName', 'example.com.'),),)}
>>> ssl.match_hostname(cert, "example.com.")   ## Case 4 from truth table
>>> ssl.match_hostname(cert, "example.com")    ## Case 2 from truth table
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../cpython/Lib/ssl.py", line 330, in match_hostname
    % (hostname, dnsnames[0]))
ssl.CertificateError: hostname 'example.com' doesn't match 'example.com.'

References
==========

[1] RFCs - there may be other RFCs addressing X.509 certificates

https://tools.ietf.org/html/rfc6125
https://tools.ietf.org/html/rfc5280
https://tools.ietf.org/html/rfc3986

[2] Old Mozilla thread discussing the trailing dot:

https://bugzilla.mozilla.org/show_bug.cgi?id=134402#c36

I quote:

Yes, it's ok to match "www.example.com." (trailing) to the cert with "www.example.com" (no trailing)
It's also OK to match "www.example.com" (no trailing) to the cert with "www.example.com." (trailing)

[3] Ruby

Ironically Ruby doesn't even take the trailing dot into consideration as it splits the strings using dot as the delimiter.

irb> "www.example.com".split('.') == "www.example.com.".split('.')
=> true

https://github.com/ruby/openssl/blob/master/lib/openssl/ssl.rb#L295

[4] Go

The trailing dot is explicitly removed in Go TLS library.

https://github.com/golang/go/blob/master/src/crypto/x509/verify.go#L913

[5] cURL

cURL strips trailing dot following behavior in browsers.

https://github.com/curl/curl/blob/master/lib/hostcheck.c#L51

The cURL author discusses the trailing dot in a couple threads.

curl/curl#716
https://lists.w3.org/Archives/Public/ietf-http-wg/2016JanMar/0430.html

[tiran]

In the future Python will no longer use its own hostname verification code. Instead we are going to rely on OpenSSL to verify the hostname for us. A trailing dot also affects SNI. How do OpenSSL's functions SSL_set_tlsext_host_name() and X509_VERIFY_PARAM_set1_host() deal with a trailing dot?

How do TLS servers such as Apache mod_ssl, Apache mod_nss, nginx, Go's TLS server, ... deal with trailing dot in SNI?

[tiran]

Trailing dots in hostname seem to be protocol specific, e.g. SMTP does not allow them. Unless you find a RFC that mandates support for trailing dots in TLS, I'm against a change in Python's TLS stack. It's too risky to mess up SNI, too.

I'd rather follow RFC 5890, make the caller deal with FQDN + trailing dot and require libraries to pass in a DNS Domain Names (a fully qualified domain name without a trailing dot) to server_hostname.

https://tools.ietf.org/html/rfc6125#section-2.2
https://tools.ietf.org/html/rfc5890#section-2.2

(The complete name convention using a trailing dot described
in RFC 1123 [RFC1123], which can be explicit as in "www.example.com."
or implicit as in "www.example.com", is not considered in this
specification.)

[tiran]

After more investigation and discussion with Daniel Stenberg, I'm considering to close the issue as WONTFIX + documentation update. The issue cannot be addressed in the SSL/TLS layer. I'm waiting for a reply from Ryan Sleevi on CAB baseline requirements. BR 1.5.1 does not state if trailing dots are allowed.

The trailing dot issue affects more than just hostname matching. For HTTPS, server name indication (SNI) and HTTP "Host" header play an important role, too. In general the SNI TLS header and HTTP Host header must match. In case the HTTP header is missing or deviates from the SNI header, web servers like Apache fail with Bad Request error. In general SNI must also match a SAN dNSName extension.

Apache with mod_ssl strips off trailing dots internally. Daniel pointed out that other webservers (IIS) do not handle trailing dots correctly. Some protocols like SMTP do not allow trailing dot in FQDN.

IMO the problem should be handled in high level libraries such as urllib. urllib should use the FQDN with trailing dot for DNS resolution, then strip off the trailing dot and use the FQDN for HTTP Host header and server_hostname.

[tiran]

Ryan said:
Chrome will match both trailing dot and non-trailing dot in URL against non-trailing dot in cert. trailing dot in cert is 5280 violation by not being in preferred name syntax

https://twitter.com/sleevi_/status/929305281405833216

[samiam]

Sorry I wasn't able to get back to you sooner.

If having a trailing dot in the cert is an RFC violation, then case 2 can be left alone.

As for case 3, we can be more explicit: if hostname ends in a dot AND cert does not end in a dot, strip dot from hostname. This seems to be what Ryan was saying Chrome does.

I did a test using s_client in openssl. Testing all 4 cases in the truth table returned 200s.

$ openssl s_client -connect www.google.com.:443
...
# Enter next two lines and press return twice
HEAD / HTTP/1.0
Host: www.google.com.

# Returns 200
HTTP/1.0 200 OK
Date: Sat, 11 Nov 2017 21:20:44 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
...

So it would appear openssl against Google handles dots ok, but I could be wrong. I don't know what server software they are running.

As for testing other server ssl implementations what are you looking for?

I found a small C openssl client implementation.

https://ubuntuforums.org/showthread.php?t=2217101&p=12989750#post12989750

Compiling that code with some minor tweaks against openssl and testing it with different hostnames and Host headers (dot and no dot), the ssl connection was established and data read. Invalid constructs led to errors.

Yes, you could move the logic to urllib, but I'm not sure it's practical as many folks just expect the ssl lib to handle the nuances. If users have to handle it themselves or include urllib, it seems like an extra lift.

I appreciate you taking the time to consider the issue.

[tiran]

As I explained before, the ssl module is the wrong place to address the issue. You *must* keep SNI TLS extension, HTTP Host header, and hostname for SAN matching in sync. Python uses the server_hostname argument for both SNI and hostname verification.

The issue must be solved in HTTP layer because the HTTP layer is the only place that can affect the HTTP Host header and SNI.

OpenSSL and NSS (Firefox's crypto and TLS lib) agree with me. Both don't like trailing dots in hostname either. BoringSSL's hostname verification code is based on OpenSSL's code. I'm pretty sure that Chrome handles trailing dot in a different layer, not in the actual TLS and X.509 handler. Ryan merely said that Chrome supports hostnames with trailing dot, not BoringSSL.

$ /usr/lib64/nss/unsupported-tools/vfyserv www.python.org
Connecting to host www.python.org (addr 151.101.112.223) on port 443
Handshake Complete: SERVER CONFIGURED CORRECTLY
   bulk cipher AES-128-GCM, 128 secret key bits, 128 key bits, status: 1
   subject DN:
 CN=www.python.org,O=Python Software Foundation,L=Wolfeboro,ST=New Hampshire,C=US,postalCode=03894-4801,STREET=16 Allen Rd,serialNumber=3359300,incorporationState=Delaware,incorporationCountry=US,businessCategory=Private Organization
   issuer  DN:
 CN=DigiCert SHA2 Extended Validation Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
   0 cache hits; 0 cache misses, 0 cache not reusable
***** Connection 1 read 518 bytes total.

$ /usr/lib64/nss/unsupported-tools/vfyserv www.python.org.
Connecting to host www.python.org. (addr 151.101.112.223) on port 443
Error in function PR_Write: -12276
 - Unable to communicate securely with peer: requested domain name does not match the server's certificate.


$ openssl s_client -servername www.python.org -verify_hostname www.python.org -connect www.python.org:443
...
SSL handshake has read 4204 bytes and written 403 bytes
Verification: OK
Verified peername: www.python.org
...

$ openssl s_client -servername www.python.org. -verify_hostname www.python.org. -connect www.python.org.:443
...
SSL handshake has read 4204 bytes and written 404 bytes
Verification error: Hostname mismatch
...

[tiran]

I'm closing this bug as "not a bug" because it works as intended. The trailing dot has to be handled in the application layer.