reserved characters are treated inconsistently and not sensibly preserved

[glyph]

This has been a design flaw since the inception of the library, so, mea culpa on that.

Fundamentally, preserving, escaping, and encoding "reserved" characters is entirely the URL object's job, and it's failing at that. Possibly the most succinct demonstration of the problem is this:

>>> u = URL()
>>> u = u.child(u'/')
>>> u = u.asIRI()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    u = u.asIRI()
  File
"/Users/glyph/Library/Python/2.7/lib/python/site-packages/hyperlink/_url.py",
line 1116, in to_iri
    fragment=_percent_decode(self.fragment))
  File
"/Users/glyph/Library/Python/2.7/lib/python/site-packages/hyperlink/_url.py",
line 861, in replace
    userinfo=_optional(userinfo, self.userinfo),
  File
"/Users/glyph/Library/Python/2.7/lib/python/site-packages/hyperlink/_url.py",
line 606, in __init__
    for segment in path))
  File
"/Users/glyph/Library/Python/2.7/lib/python/site-packages/hyperlink/_url.py",
line 606, in <genexpr>
    for segment in path))
  File
"/Users/glyph/Library/Python/2.7/lib/python/site-packages/hyperlink/_url.py",
line 410, in _textcheck
    % (''.join(delims), name, value))
ValueError: one or more reserved delimiters /?# present in path segment: u'/'
>>>

This is - obviously I hope - the wrong place to be failing with an error like this.

There was previously some attempt to preserve these characters in the data model and escape them only upon stringification, but d26814c wrecked these semantics. (In fairness: the attempt to do this was broken, and there are some places, like the scheme, where certain characters indeed cannot be represented, so this direction isn't entirely wrong.)

Fundamentally if a user wants to encode slashes, question marks, hash signs or whatever else that a human might, for example, type into a text field, then it should be possible to do that.

We could fix this obvious manifestation of the problem by just putting back the escape-only-on-asText logic, but that still leaves an even more pernicious problem:

>>> u = URL(path=tuple([u'%2525']))
>>> u.asText()
u'%2525'
>>> u.asIRI().asText()
u'%25'
>>> u.asIRI().asIRI().asText()
u'%'
>>> 

Clearly, multiple trips through asIRI should not be un-escaping the escape character - the idea is that .asIRI() is a normalization step, that should be idempotent upon subsequent calls.

For the moment, I'm not sure exactly what the correct fix is here, but the property I'd really like to preserve is that for any x,

URL.fromText(URL().child(x).<as many asIRI()s or asURI()s as you want>.asText()).<as many .asIRI()s as you want, although possibly not .asURI()s>.segments[0] == x

[mahmoud]

Agreed on the invariant. It's quite late, but I have an intuition I'd like to put to paper, as it were.

asIRI has this excessive escaping behavior, but I haven't seen excessive unescaping from asURI, and I wonder if it's as simple as providing specialized decoding functions for each part, to match the specialized encoding functions (e.g., _encode_query_part). The decoding felt a bit simplistic, and I suspect this is why.

[nigelmegitt]

It isn't obvious that this was broken before, but it definitely is now (since d26814c). The library cannot know if a path segment is supposed to include relative sub-parts like outer/middle/inner or if that should be % encoded as outer%2fmiddle%2finner - it can only deal with what it is given.

The repeat encoding/decoding functionality is not a bug - it is correct implementation of the percent encoding and decoding rules. Only the calling code can know how many times it has been called. I think it is putting too much cleverness into URL to try to make it idempotent in the way this issue calls for, though I understand that it seems like an elegant solution. Instead I think if an application is calling u.asIRI().asIRI()... that is almost certainly a usage error, and if it is not a usage error, it's in the calling code's domain to know that.

Quite separately, I believe the effect of _textcheck() here, which throws if path segments include percent encoded reserved characters is a bug. There's simply no problem having percent encoded characters in a URI or IRI. That's what they're for.

Apologies in advance if I've misunderstood the intent of this issue, it's a complex area and I've probably made some mistakes understanding it all!

[mahmoud]

Hey @nigelmegitt! I think you've got the right idea as to the intent of the issue, that .asIRI be idempotent. As for _textcheck, it's totally percent-encoding agnostic, so I'm pretty sure the problem lies between there and the top level _create_client method you linked from here.

I'll delve into this particular case and see if I can diagnose the issue. Might get a good test case out of it, too :) An independent issue and stack trace might speed that along, if you wouldn't mind.

[glyph]

I think that the right thing to do for now might actually be refuse to ever decode these meta-characters; i.e. URL.from_text('%25%2F').asIRI() → URL.from_text('%25%2F').

Right now, .path and .query are literal text as they appear in the URI itself. This makes sense as a data structure, although it is somewhat limiting. It's also a necessary piece of structure to preserve, because the conversion to "decoded" characters is not information-preserving: since a percent-encoded sequence might represent an invalid UTF-8 sequence of octets, for example. But it also means that reserved characters can never appear in them without escaping.

The thing to do for segments that encode slashes, question marks and percent signs, I believe, would be to have a separate attribute without the exception-raising restriction that we currently have; decoded_path for a punching-bag name suggestion.

[glyph]

The library cannot know if a path segment is supposed to include relative sub-parts like outer/middle/inner or if that should be % encoded as outer%2fmiddle%2finner - it can only deal with what it is given.

I just wanted to respond to this claim in particular - the library can totally know :). You just need to have attributes which unambiguously represent the data according to a particular perspective. If .path is "the segments as they appear between slashes" then it's easy to establish the invariant that the strings in it can't contain slashes.

[glyph]

I think #23 fixed the "defect" part of this, but there's still the "feature" part - we need some way to shove a "/" into a path segment and get it back out again if an application wants to do that.

[mahmoud]

I think "need" is a strong word here. I'd like to see such a need demonstrated before putting the defect at the same level as the feature. For instance, can you get werkzeug or anything else to even have a rule to match a slash in a path segment?

I'd term it an advanced use case at the very least. Maybe open a broader discussion to making the _encode_* and _decode_* functions public in a sort of hazmat way. Now that I understand those semantics, I think they're pretty stable (and way better than the anemic footgusn in the stdlib).

[glyph]

One could quibble about path segments but certainly query parameters need this treatment; they are often used by applications as a map for storing quasi-arbitrary data.

[mahmoud]

Sure, that's how a lot of PayPal's structured logging worked. Based on my experience, I'd've preferred a dedicated object for that and/or encode/decode functions with clear uses and explanations.

Another approach that occurs to me, but I'm only mentioning for brainstorming purposes, is to have a single proxy object wrapper utility that marks a value as unescaped, and have each field check for that and apply the respective encoding (query encoding for query string parts, path encoding for path parts, etc.).

u = URL()
u.replace(path=escape('a', '///', 'b'))

I don't know why, but it feels kind of Django-y. :P

[glyph]

I really don't like the idea of another top-level name, especially not the heavily fraught escape.

Along the lines of a dedicated object here, what if there were a wrapper type, like this:

u = URL()
u.replace(path=["%25"]) → URL.fromText('%25')
u.decoded.replace(path=["%25"]) → URL.fromText('%2525')
URL.fromText('%25').decoded.path[0] → '%'

The two problems I see here are:

1. not super in love with the name .decoded
2. it might be a little confusing that .decoded gives you a _DecodedURL or whatever, but .replace gives you back a regular URL again.

But I think I prefer this general idea, if those can be addressed / made clear somehow.

[nigelmegitt]

@glyph wrote:

If .path is "the segments as they appear between slashes" then it's easy to establish the invariant that the strings in it can't contain slashes.

OK, with this approach I cannot add a sub-path to the segment as some/path/to in one step, I have to add multiple path segments in turn. Let's trace through the impacts of allowing adding such sub-paths or not allowing it:

If you allow adding a sub-path then you need to accept that sometimes it will have slashes in and sometimes it might have %-encoded slashes that are intended to be retained as such. In which case you really have to trust that the providing code was written by someone who knew what they were doing. And then when you pass that string on somewhere and it is decoded then any sub-paths with slashes in will be itemised separately. On the plus side you don't need to check the string for slashes. If they're there, it's fine.

Alternatively if you do require each segment to be appended separately then the API needs to be clear about what it is going to do if there is a reserved character in it - will it always percent encode the provided string, or will it never percent encode it but always reject an incorrectly encoded string?

[glyph]

OK, with this approach I cannot add a sub-path to the segment as some/path/to in one step, I have to add multiple path segments in turn. Let's trace through the impacts of allowing adding such sub-paths or not allowing it:

Indeed, this is the way that FilePath works, and URL intentionally mirrors it, for security (and other) reasons.

[glyph]

If you allow adding a sub-path then you need to accept that sometimes it will have slashes in and sometimes it might have %-encoded slashes that are intended to be retained as such. In which case you really have to trust that the providing code was written by someone who knew what they were doing. And then when you pass that string on somewhere and it is decoded then any sub-paths with slashes in will be itemised separately.

The approach right now is to just raise an exception, which seems like it neatly short-circuits all these issues.

If we add a .decoded object, then the answer in that case will be "always escape".

[glyph]

Alternatively if you do require each segment to be appended separately then the API needs to be clear about what it is going to do if there is a reserved character in it - will it always percent encode the provided string, or will it never percent encode it but always reject an incorrectly encoded string?

In the case of URL itself: always reject, never encode.

In the case of .decoded: always encode, never reject.

[mahmoud]

just as a point of order, navigation and appending path segments is meant to be done with .sibling() and .child(), not .replace().

Also, one may say "escape" is fraught, but looking at Python code these days, one sees a lot more encode() and decode() than escape, quote, and a lot else :P

[nigelmegitt]

OK, with this approach I cannot add a sub-path to the segment as some/path/to in one step, I have to add multiple path segments in turn. Let's trace through the impacts of allowing adding such sub-paths or not allowing it:

Indeed, this is the way that FilePath works, and URL intentionally mirrors it, for security (and other) reasons.

What about when we just create a URL from a unicode string including all the parts? Are you saying that functionality is no longer supported?

[nigelmegitt]

The approach right now is to just raise an exception, which seems like it neatly short-circuits all these issues.

Well it bumps the issues down the line by throwing an exception if an otherwise perfectly reasonably %-encoded /, ? or # is present in the segment. This didn't happen before hyperlink release 17.2.0, so I'm stuck with the previous release until I can construct a URI from a string that includes those (encoded) entities in the path. Maybe I'm missing some newly added functionality for doing this?

[glyph]

What about when we just create a URL from a unicode string including all the parts? Are you saying that functionality is no longer supported?

No, that functionality is fromText, and is irrelevant to this conversation, because the way you tell fromText about new path segments is that you put a / into the path :).

[glyph]

Well it bumps the issues down the line by throwing an exception if an otherwise perfectly reasonably %-encoded /, ? or # is present in the segment.

OK, I'm missing something then. As far as I can tell, at no point has hyperlink ever raised an exception when given e.g. URL(path=[u'%25%2F%23%3F']). In 17.2.1, that.asIRI() will raise an exception, which is definitely a bug, but that bug has been fixed in #23.

This didn't happen before hyperlink release 17.2.0, so I'm stuck with the previous release until I can construct a URI from a string that includes those (encoded) entities in the path. Maybe I'm missing some newly added functionality for doing this?

URL.fromText(u'%25%2F%23%3F') works on 17.2.1 as well, although the fact that you can't normalize it is definitely a bummer; again, that'll be fixed in the next release.

[nigelmegitt]

Here's the issue as simply as I can make it in 17.2.1:

$ pip show hyperlink
Name: hyperlink
Version: 17.2.1
Summary: A featureful, correct URL for Python.
Home-page: https://github.com/python-hyper/hyperlink
Author: Mahmoud Hashemi and Glyph Lefkowitz
Author-email: mahmoud@hatnote.com
License: BSD
Location: /Users/megitn02/Code/ebu/ebu-tt-live-toolkit/venv/lib/python2.7/site-packages
Requires: 
$ python
Python 2.7.10 (default, Oct 23 2015, 19:19:21) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from hyperlink import URL
>>> url = URL.from_text(u'ws://localhost:9006/sequence%2F%C3%BCnic%C3%B6d%C3%A9%3F%2FName/subscribe')
>>> iri = url.to_iri()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "*****/venv/lib/python2.7/site-packages/hyperlink/_url.py", line 1116, in to_iri
    fragment=_percent_decode(self.fragment))
  File "*****/venv/lib/python2.7/site-packages/hyperlink/_url.py", line 861, in replace
    userinfo=_optional(userinfo, self.userinfo),
  File "*****/venv/lib/python2.7/site-packages/hyperlink/_url.py", line 606, in __init__
    for segment in path))
  File "*****/venv/lib/python2.7/site-packages/hyperlink/_url.py", line 606, in <genexpr>
    for segment in path))
  File "*****/venv/lib/python2.7/site-packages/hyperlink/_url.py", line 410, in _textcheck
    % (''.join(delims), name, value))
ValueError: one or more reserved delimiters /?# present in path segment: u'sequence/\xfcnic\xf6d\xe9?/Name'
>>> quit()

Now here's the same code working on an older version:

$ pip install 'hyperlink<17.2.0'
Collecting hyperlink<17.2.0
  Using cached hyperlink-17.1.1-py2.py3-none-any.whl
Installing collected packages: hyperlink
  Found existing installation: hyperlink 17.2.1
    Uninstalling hyperlink-17.2.1:
      Successfully uninstalled hyperlink-17.2.1
Successfully installed hyperlink-17.1.1
EBU mc-s092695:ebu-tt-live-toolkit megitn02$ python
Python 2.7.10 (default, Oct 23 2015, 19:19:21) 
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from hyperlink import URL
>>> url = URL.from_text(u'ws://localhost:9006/sequence%2F%C3%BCnic%C3%B6d%C3%A9%3F%2FName/subscribe')
>>> iri = url.to_iri()
>>> print iri.to_text()
ws://localhost:9006/sequence%2Fünicödé%3F%2FName/subscribe
>>> quit()

Apologies again if this is actually fixed by #23 but it explains my earlier comment.

[mahmoud]

Yep, fixed on trunk, don't you fret!

>>> from hyperlink import URL
>>> url = URL.from_text(u'ws://localhost:9006/sequence%2F%C3%BCnic%C3%B6d%C3%A9%3F%2FName/subscribe')
>>> iri = url.to_iri()
>>> print iri.to_text()
ws://localhost:9006/sequence%2Fünicödé%3F%2FName/subscribe

[glyph]

Here's the issue as simply as I can make it in 17.2.1

Yep, as @mahmoud said, that's exactly what #23 fixed. Sorry we were talking past each other a little bit - that what I meant by "that.asIRI() will raise an exception".

[nigelmegitt]

Great, thank you!

[mahmoud]

The release published last night (17.3.0) fixes all this behavior. Give it a spin, let me know what you think. Thanks again for the report!