Stricter pointer/ref checks & public segment encoding/decoding methods

[brettz9]

Two related issues:

1. Regarding a proper JSON Reference, the JSON Reference spec states:

 If a URI contains a fragment identifier, then the fragment should be
   resolved per the fragment resolution mechansim of the referrant
   document.  If the representation of the referrant document is JSON,
   then the fragment identifier SHOULD be interpreted as a
   [JSON-Pointer].

The JSON Pointer spec, in turn, says about JSON Pointers:

   Since the characters '~' (%x7E) and '/' (%x2F) have a special meaning
   in JSON Pointer, they need to be encoded as '~0' and '~1'
   respectively, when appearing in a reference token.

I would think therefore, that when executing isPtr upon encountering any tildes not followed by 0 or 1, false should be given. Similarly when executing isRef, especially if we detect that the target document is indeed JSON. This will also have the consequence of potentially throwing for this reason in pathFromPtr.

The spec seems to make no mention of any requirement to do this, however:

Evaluation of each reference token begins by decoding any escaped
   character sequence; this is performed by first transforming any
   occurrence of the sequence '~1' to '/', then transforming any
   occurrence of the sequence '~0' to '~'.

...but I think that when one asks whether something is a pointer or reference, it is good to know whether it has had proper escaping.

2 .I also think that as the JsonRefs tool, you should make the segment encoding and decoding functions public.

[whitlockjc]

1. Good point. We can do this.
2. Agreed.

[brettz9]

Are you open to reopening this for the sake of the first of my two requests? If so, it would be nice to be able to avoid throwing an error in the case where tildes are not followed by 0 or 1 since it is not technically illegal, and I also have a specific use-case for this (using tildes to surround variables within JSON references which may be substituted at some point, e.g., locale).

[whitlockjc]

I thought I handled them. Also, I'm not throwing an error for tildes followed by something other than 0 or 1, just returning false.

[brettz9]

Sorry about that then--haven't looked too carefully yet.

[brettz9]

As far as the tildes and error though, what I mean is that since isPtr will return false for such entries, the pathFromPtr (and in turn findRefs) functions will fail (not to mention not give the user the option to find them).

[whitlockjc]

Good point. #getRefDetails is there to help diagnose busted references, as in ones that don't show up in #findRefs and related. We could make it an option to support invalid references when finding/resolving references so at least the metadata would be there to help identify busted references.

All of this being said, if you have an invalid reference, for any reason, finding/resolving references will not process them so it's not really a big change. I think if a user expects to find/resolve a reference and it's not happening, advising them to use #getRefDetails would suffice.

Right now, these were the changes related to closing this issue. Please confirm that I've addressed everything:

• I exposed #encodePath and #decodePath (I thought about exposing #encodePtr and #decodePtr but the first would be impossible to support because you couldn't tell if / was a path segment delimiter or part of a path segment. Without the first, the second didn't seem as useful but I'm open to suggestions.)
• I updated #isPtr to validate the JSON Pointer tokens
• I updated #encodePath and #decodePath to encode/decode in the proper order

Let me know if there is more and we'll reopen this.

[brettz9]

encodePtr and decodePtr were even more in the spirit of what I was looking for, but I hear you about the ambiguity.

It could be handled as JS does with encodeURI/encodeURIComponent, where using encodePtr/encodePtrComponent would cause the latter to escape / and the former to not escape, though one must know which to use. (The decode equivalents would seem to always need to convert ~1 to /, and thus one would only need decodePtrComponent unless decodePtr accepted a string and returned a path--which would be weird). Maybe it is just best to let the user convert to and from paths and supply to these methods you provided. So I guess I'm ok for now without adding any new methods.

As far as getRefDetails, since one must know the location in advance for getRefDetails, I don't think it will meet my needs. One use case I have, for example, is to find all of these poorly formed references and do variable substitutions (http://example.com/~var1~/and/~var2~) to make them well-formed and then have them resolved. I specifically use tildes to do this since they are (at least in the present spec) safe from having other meanings and can potentially alert me when I have not yet substituted the variables.

[brettz9]

Further to the discussion of finding invalid pointers, I think isPtr itself (and relevant calling functions) ought to accept a second argument as to whether the treatment should be strict or not (since the spec seems to technically allow poorly formed pointers).

[whitlockjc]

So you want me to remove the exposed decodePath and encodePath? I'm not against it, just wanting to be clear.

As for needing to know to use #getRefDetails, I understand. The biggest issue I've got with some of these APIs is based on my needs, I am pretty good. If you need something else, letting me know is how we'll get it sorted. One thing we could do is update #findRefs to allow for a configuration to return metadata for JSON Reference-like objects that do not pass validation and to include the reasons for them being invalid. #getRefDetails would do some of this but like you say, knowing you need to use it is painful. (This might not be a simple thing to implement since I'll have to break out a #isPtr and #isRef to throw errors and allow trapping them for #findRefs, #resolveRefs and friends. Shouldn't be too hard.)

In the end, I just want #isPtr and #isRef to just return booleans and to not throw errors. As long as I can keep it like that but enhance #findRefs, since all other APIs use it, we should be good to go. Any idea on a good configuration option? options.strictMode comes to mind, which would default to true of course.

[whitlockjc]

If I understand this correctly, these will be the next steps:

• Remove decodePath and encodePath as they aren't really useful as-is and the pointer versions aren't really possible
• Update #findRefs to allow you to specify some "strict mode" to allow you to find potential JSON References that are invalid

[brettz9]

As far as decodePath and encodePath, on further reflection, I actually like them as is. I was just talking about the idea of encodePtr, etc., which I no longer think is necessary due to the reasons you mention.

[whitlockjc]

Okay. So keep the new methods as-is. :) What about the configuration option to make #findRefs find invalid references?