reserved characters are treated inconsistently and not sensibly preserved

[glyph]

This has been a design flaw since the inception of the library, so, mea culpa on that.

Fundamentally, preserving, escaping, and encoding "reserved" characters is entirely the URL object's job, and it's failing at that. Possibly the most succinct demonstration of the problem is this:

>>> u = URL()
>>> u = u.child(u'/')
>>> u = u.asIRI()
Traceback (most recent call last):
  File "<input>", line 1, in <module>
    u = u.asIRI()
  File
"/Users/glyph/Library/Python/2.7/lib/python/site-packages/hyperlink/_url.py",
line 1116, in to_iri
    fragment=_percent_decode(self.fragment))
  File
"/Users/glyph/Library/Python/2.7/lib/python/site-packages/hyperlink/_url.py",
line 861, in replace
    userinfo=_optional(userinfo, self.userinfo),
  File
"/Users/glyph/Library/Python/2.7/lib/python/site-packages/hyperlink/_url.py",
line 606, in __init__
    for segment in path))
  File
"/Users/glyph/Library/Python/2.7/lib/python/site-packages/hyperlink/_url.py",
line 606, in <genexpr>
    for segment in path))
  File
"/Users/glyph/Library/Python/2.7/lib/python/site-packages/hyperlink/_url.py",
line 410, in _textcheck
    % (''.join(delims), name, value))
ValueError: one or more reserved delimiters /?# present in path segment: u'/'
>>>

This is - obviously I hope - the wrong place to be failing with an error like this.

There was previously some attempt to preserve these characters in the data model and escape them only upon stringification, but d26814c wrecked these semantics. (In fairness: the attempt to do this was broken, and there are some places, like the scheme, where certain characters indeed cannot be represented, so this direction isn't entirely wrong.)

Fundamentally if a user wants to encode slashes, question marks, hash signs or whatever else that a human might, for example, type into a text field, then it should be possible to do that.

We could fix this obvious manifestation of the problem by just putting back the escape-only-on-asText logic, but that still leaves an even more pernicious problem:

>>> u = URL(path=tuple([u'%2525']))
>>> u.asText()
u'%2525'
>>> u.asIRI().asText()
u'%25'
>>> u.asIRI().asIRI().asText()
u'%'
>>> 

Clearly, multiple trips through asIRI should not be un-escaping the escape character - the idea is that .asIRI() is a normalization step, that should be idempotent upon subsequent calls.

For the moment, I'm not sure exactly what the correct fix is here, but the property I'd really like to preserve is that for any x,

URL.fromText(URL().child(x).<as many asIRI()s or asURI()s as you want>.asText()).<as many .asIRI()s as you want, although possibly not .asURI()s>.segments[0] == x

[mahmoud]

Agreed on the invariant. It's quite late, but I have an intuition I'd like to put to paper, as it were.

asIRI has this excessive escaping behavior, but I haven't seen excessive unescaping from asURI, and I wonder if it's as simple as providing specialized decoding functions for each part, to match the specialized encoding functions (e.g., _encode_query_part). The decoding felt a bit simplistic, and I suspect this is why.

[nigelmegitt]

It isn't obvious that this was broken before, but it definitely is now (since d26814c). The library cannot know if a path segment is supposed to include relative sub-parts like outer/middle/inner or if that should be % encoded as outer%2fmiddle%2finner - it can only deal with what it is given.

The repeat encoding/decoding functionality is not a bug - it is correct implementation of the percent encoding and decoding rules. Only the calling code can know how many times it has been called. I think it is putting too much cleverness into URL to try to make it idempotent in the way this issue calls for, though I understand that it seems like an elegant solution. Instead I think if an application is calling u.asIRI().asIRI()... that is almost certainly a usage error, and if it is not a usage error, it's in the calling code's domain to know that.

Quite separately, I believe the effect of _textcheck() here, which throws if path segments include percent encoded reserved characters is a bug. There's simply no problem having percent encoded characters in a URI or IRI. That's what they're for.

Apologies in advance if I've misunderstood the intent of this issue, it's a complex area and I've probably made some mistakes understanding it all!

[mahmoud]

Hey @nigelmegitt! I think you've got the right idea as to the intent of the issue, that .asIRI be idempotent. As for _textcheck, it's totally percent-encoding agnostic, so I'm pretty sure the problem lies between there and the top level _create_client method you linked from here.

I'll delve into this particular case and see if I can diagnose the issue. Might get a good test case out of it, too :) An independent issue and stack trace might speed that along, if you wouldn't mind.

[glyph]

I think that the right thing to do for now might actually be refuse to ever decode these meta-characters; i.e. URL.from_text('%25%2F').asIRI() → URL.from_text('%25%2F').

Right now, .path and .query are literal text as they appear in the URI itself. This makes sense as a data structure, although it is somewhat limiting. It's also a necessary piece of structure to preserve, because the conversion to "decoded" characters is not information-preserving: since a percent-encoded sequence might represent an invalid UTF-8 sequence of octets, for example. But it also means that reserved characters can never appear in them without escaping.

The thing to do for segments that encode slashes, question marks and percent signs, I believe, would be to have a separate attribute without the exception-raising restriction that we currently have; decoded_path for a punching-bag name suggestion.

[glyph]

The library cannot know if a path segment is supposed to include relative sub-parts like outer/middle/inner or if that should be % encoded as outer%2fmiddle%2finner - it can only deal with what it is given.

I just wanted to respond to this claim in particular - the library can totally know :). You just need to have attributes which unambiguously represent the data according to a particular perspective. If .path is "the segments as they appear between slashes" then it's easy to establish the invariant that the strings in it can't contain slashes.

[glyph]

I think #23 fixed the "defect" part of this, but there's still the "feature" part - we need some way to shove a "/" into a path segment and get it back out again if an application wants to do that.

[mahmoud]

I think "need" is a strong word here. I'd like to see such a need demonstrated before putting the defect at the same level as the feature. For instance, can you get werkzeug or anything else to even have a rule to match a slash in a path segment?

I'd term it an advanced use case at the very least. Maybe open a broader discussion to making the _encode_* and _decode_* functions public in a sort of hazmat way. Now that I understand those semantics, I think they're pretty stable (and way better than the anemic footgusn in the stdlib).

[glyph]

One could quibble about path segments but certainly query parameters need this treatment; they are often used by applications as a map for storing quasi-arbitrary data.

[mahmoud]

Sure, that's how a lot of PayPal's structured logging worked. Based on my experience, I'd've preferred a dedicated object for that and/or encode/decode functions with clear uses and explanations.

Another approach that occurs to me, but I'm only mentioning for brainstorming purposes, is to have a single proxy object wrapper utility that marks a value as unescaped, and have each field check for that and apply the respective encoding (query encoding for query string parts, path encoding for path parts, etc.).

u = URL()
u.replace(path=escape('a', '///', 'b'))

I don't know why, but it feels kind of Django-y. :P