Linkify incorrectly parses array arguments

[M1ha-Shvn]

Hi.
Library version up to 3.1.0 incorrectly parses array and object url parameters:

from bleach import DEFAULT_CALLBACKS, Linker
text= 'http://test.com?array[]=1&params_in[]=2'
linker = Linker(url_re=linkifier.URL_RE, callbacks=DEFAULT_CALLBACKS, skip_tags=None, parse_email=False)
print(linker.linkify(text))
# prints: <a href="http://test.com?array" rel="nofollow">http://test.com?array</a>[]=1¶ms_in[]=2

As you see, url is split by [], loosing part of the link.

[willkg]

The url matching code is here:

bleach/bleach/linkifier.py

Lines 32 to 53 in 2f210e0

	def build_url_re(tlds=TLDS, protocols=html5lib_shim.allowed_protocols):
	"""Builds the url regex used by linkifier
	If you want a different set of tlds or allowed protocols, pass those in
	and stomp on the existing ``url_re``::
	from bleach import linkifier
	my_url_re = linkifier.build_url_re(my_tlds_list, my_protocols)
	linker = LinkifyFilter(url_re=my_url_re)
	"""
	return re.compile(
	r"""\(* # Match any opening parentheses.
	\b(?<![@.])(?:(?:{0}):/{{0,3}}(?:(?:\w+:)?\w+@)?)? # http://
	([\w-]+\.)+(?:{1})(?:\:[0-9]+)?(?!\.\w)\b # xx.yy.tld(:##)?
	(?:[/?][^\s\{{\}}\|\\\^\[\]`<>"]*)?
	# /path/zz (excluding "unsafe" chars from RFC 1738,
	# except for # and ~, which happen in practice)
	""".format('|'.join(protocols), '|'.join(tlds)),
	re.IGNORECASE | re.VERBOSE | re.UNICODE)

The comment there suggests that it'll match characters except those denoted as unsafe characters in RFC 1738. Unsafe characters like [ and ] characters need to be encoded. So that's what's going on here.

RFC 3986 updates RFC 1738 and has a set of reserved characters which includes [ and ]. I think we need to update the url regex to RFC 3986.

I'm pretty busy for a while. I'll accept a PR if someone wants to do one.

[mastizada]

@willkg Removing \[\] escape in regex gave the expected result, but I'm wondering why &params is converted to ¶ms. Is it an issue or an expected behavior for &para?

[filak]

The &para issue

from bleach import Linker
linker = Linker()
text= 'http://test.com?&params=2'
print(linker.linkify(text))
## prints:   <a href="http://test.com?¶ms=2" rel="nofollow">http://test.com?¶ms=2</a>

I believe this happens somewhere in BleachHTMLSerializer class:

bleach/bleach/html5lib_shim.py

Line 661 in ed06d4e

class BleachHTMLSerializer(HTMLSerializer):

[willkg]

The &para thing should be a separate issue. This issue is covering array arguments.

[filak]

@willkg I am sorry, please see #670

[willkg]

Thank you! I appreciate it!

[willkg]

Thank you for writing this up!