[CVE-2024-11168] urlparse incorrectly retrieves IPv4 and regular name hosts from inside of brackets

[JohnJamesUtley]

Background

RFC 3986 defines a host as follows

host = IP-literal / IPv4address / reg-name

Where

IP-literal = "[" ( IPv6address / IPvFuture  ) "]"
reg-name = *( unreserved / pct-encoded / sub-delims )
IPv4address = dec-octet "." dec-octet "." dec-octet "." dec-octet

WhatWG says that "A valid host string must be a valid domain string, a valid IPv4-address string, or: U+005B ([), followed by a valid IPv6-address string, followed by U+005D (])."

The Bug

This is code from Lib/urllib/parse.py:196-208 used for retrieving the hostname from the netloc

    @property
    def _hostinfo(self):
        netloc = self.netloc
        _, _, hostinfo = netloc.rpartition('@')
        _, have_open_br, bracketed = hostinfo.partition('[')
        if have_open_br:
            hostname, _, port = bracketed.partition(']')
            _, _, port = port.partition(':')
        else:
            hostname, _, port = hostinfo.partition(':')
        if not port:
            port = None
        return hostname, port

It will incorrectly retrieve IPv4 addresses and regular name hosts from inside brackets. This is in violation of both specifications.

Minimally reproducible example:

from urllib.parse import urlsplit

parsedURL = urlsplit('scheme://user@[regname]/Path')
print(parsedURL.hostname) # Prints 'regname'

Your environment

• CPython versions tested on:
  • 3.12a7 (23cf1e2)
  • 3.10.10
• Operating system and architecture:
  • Arch Linux x86_64

Linked PRs

• gh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format #103849
• [3.11] gh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format (GH-103849) #104349
• [3.10] gh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format (#103849) #126975
• [3.9] gh-103848: Adds checks to ensure that bracketed hosts found by urlsplit are of IPv6 or IPvFuture format (#103849) #126976

[mjpieters]

As implemented, the URL scheme://foobar.[::1]/ is considered correct, but results in the hostname value of ::1. The hostname portion of the netloc is invalid however, as it doesn't start with [. Shouldn't the parser at least validate that the [ bracket is the first character or directly following a @?

[MarcoGlauser]

This change caused an issue on our system. We have brackets in a password. Since 3.11.4, urlparse tries to parse an ip address if it finds brackets. The brackets don't even have to be in the correct order.

Running this code:

from urllib.parse import urlparse
urlparse("https://user:some]password[@host.com")

In 3.11.3 it works as expected:

ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')

In 3.11.4 it throws an exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/urllib/parse.py", line 395, in urlparse
    splitresult = urlsplit(url, scheme, allow_fragments)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/urllib/parse.py", line 500, in urlsplit
    _check_bracketed_host(bracketed_host)
  File "/usr/local/lib/python3.11/urllib/parse.py", line 446, in _check_bracketed_host
    ip = ipaddress.ip_address(hostname) # Throws Value Error if not IPv6 or IPv4
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/ipaddress.py", line 54, in ip_address
    raise ValueError(f'{address!r} does not appear to be an IPv4 or IPv6 address')
ValueError: '@host.com' does not appear to be an IPv4 or IPv6 address

[mjpieters]

This change caused an issue on our system. We have brackets in a password. Since 3.11.4, urlparse tries to parse an ip address if it finds brackets. The brackets don't even have to be in the correct order.

Running this code:

from urllib.parse import urlparse
urlparse("https://user:some]password[@host.com")

In 3.11.3 it works as expected:

ParseResult(scheme='https', netloc='user:some]password[@host.com', path='', params='', query='', fragment='')

That this worked before was a bug. urlparse() was violating the various and miriad URL standards before; [ and ] are reserved characters with meaning in the netloc portion of a URL (the part between the scheme:// and /path) and must be URL encoded. See the WhatWG living URL standard percent-encoding section; for the userinfo part (username + password), you need to encode:

• U+002F (/), U+003A (:), U+003B (;), U+003D (=), U+0040 (@), U+005B ([) to U+005E (^), inclusive, and U+007C (|).
• U+003F (?), U+0060 (`), U+007B ({), and U+007D (}) (path percent-encode set).
• U+0020 SPACE, U+0022 ("), U+0023 (#), U+003C (<), and U+003E (>) (query percent-encode set).
• U+0000 NULL to U+001F INFORMATION SEPARATOR ONE, inclusive, and all codepoints past U+007E (~) (C0 control percent-encode set)

So this works:

>>> from urllib.parse import urlparse
>>> urlparse("https://user:some%5Dpassword%5B@host.com")
ParseResult(scheme='https', netloc='user:some%5Dpassword%5B@host.com', path='', params='', query='', fragment='')
>>> _.password
'some%5Dpassword%5B'

The password is still URL encoded, use urllib.parse.unquote() for that.

Further references:

• Wikipedia: URL encoding
• RFC 3986, section 2.2 Reserved Characters

[MarcoGlauser]

Thank you for the clarification! This makes a lot of sense :)

[Jc2k]

I'm also hitting "ValueError: 'xxxxx' does not appear to be an IPv4 or IPv6 address" on code that was working.

In my case I have no control over the input data, it's background noise from the internet that i'm parsing and analyzing. Some of the crap in my dataset is very much not standards compliant. None the less, as it was working and now isn't i figured i should 👍 before i start vendoring old versions of the stdlib into my project 😆

[gpshead]

The original issue was fixed in May... it sounds like there may be some new issues that unfortunately arose? People with new parsing problems, please file new issues to track those.

[mcepl]

My backport of fix for Python 3.6 (which with minimal changes applies to 3.4 as well, and with a lot of changes, including injecting ipaddress from https://github.com/phihag/ipaddress into the Python standard library, into 2.7) is at openSUSE-Python@4f2496b