Address several IDNA issues

[annevk]

As reported at
#53 (comment) this is
causing issues in non-browser implementations.

---

Preview | Diff

[domenic]

So what should we do in Node.js/whatwg-url? Should we take this as license to violate the spec, so that our users can use our libraries on YouTube?

I'd kind of prefer the spec just match browsers here (and in most places); we've never had success getting implementers to comment on URL parsing issues before, and I doubt it's going to start here.

[annevk]

Sigh. I guess you're right. It's just such dumb Latin-alphabet-centric behavior that I'm having a hard time accepting this is what we want long term. And I'd rather fix issues once than multiple times.

[annevk]

Final attempt: https://groups.google.com/d/msg/mozilla.dev.platform/Hd8ATeJcV1A/xEeGH--tAQAJ

[rmisev]

Following tests show, what not all browsers use fast path for ASCII only input:

Input	URL Standard	Chrome	Edge	Firefox
ab--c	(failure)	ab--c	ab--c	ab--c
xn--a	(failure)	xn--a	(failure)	xn--a
xn--nxa.xn--nxa	xn--nxa.xn--nxa	xn--nxa.xn--nxa	xn--nxa.xn--nxa β.β	β.β
xn--nxa.β	xn--nxa.xn--nxa	xn--nxa.xn--nxa	xn--nxa.xn--nxa β.β	β.β
ab--c.xn--nxa	(failure)	ab--c.xn--nxa	ab--c.xn--nxa ab--c.β	ab--c.β
ab--c.β	(failure)	(failure)	ab--c.xn--nxa ab--c.β	ab--c.β
xn--a.xn--nxa	(failure)	xn--a.xn--nxa	(failure)	xn--a.xn--nxa
xn--a.β	(failure)	(failure)	(failure)	xn--xn--a-pm43a.β

As you see Edge checks the xn-- labels for ASCII only input, while Chrome doesn't. The Chrome's behavior doesn't look consistent for me. I think Edge's is better.

Firefox doesn't fail on invalid labels at all and the last test result shows browser's bug, which must be fixed anyway.

Note: I tested with URL object, prefixing the input with http://. If Edge's result has two lines, then first is host returned in href, second - hostname.

[annevk]

Thanks for those tests. Chrome's behavior looks consistent? If all of the input is ASCII, don't run ToASCII. Label splitting happens as part of ToASCII so is not a consideration until then.

[rmisev]

If follow Chrome's behavior, then it's unclear xn--a.xn--nxa is valid or not. It is accepted when I enter it in the address bar. But then Chrome converts it to xn--a.β. So valid becomes invalid... 😕

[annevk]

I see, the problem is that they use ToUnicode (which never fails) after which it no longer roundtrips. That does indeed seem bad.

[annevk]

@rmisev sorry for leaving this. So what is the processing model we want? ASCII lowercase input; split input on "."; if a label starts with "xn--" or contains non-ASCII, run input through ToASCII? That's getting rather complicated. Always invoking ToASCII and setting the ignore hyphen flag might end up being better, no?

[rmisev]

I think simpler is better, so always ToASCII.

[annevk]

I updated the PR to point to the proposed update of UTS46 and set CheckHyphens to false.

[TimothyGu]

Is there anything preventing this PR from being applied? Never mind; this.

[domenic]

We're working on validating the tests via jsdom/whatwg-url in jsdom/whatwg-url#90, but kind of stuck since we don't implement CheckJoiners and CheckBidi. We'll probably go ahead and merge anyway, but would Node.js be able to create an implementation that passes the tests instead? That'd be even better.

[TimothyGu]

would Node.js be able to create an implementation that passes the tests instead?

Yes I think so. In fact, cases 3 and 4 here (the two corresponding to CheckBidi) are already handled correctly in Node.js 8.0.0. CheckJoiners was not included in the release, since at the time the PR was merged there was not yet a resolution on CheckJoiners.

[domenic]

Great. Would it make sense to put off merging this PR until there is a corresponding verification from Node.js that the tests and spec are implementable? That's been our general preference so far, using either whatwg-url or Node.js.

[TimothyGu]

until there is a corresponding verification from Node.js that the tests and spec are implementable?

And indeed there is: nodejs/node#13362 I'll look into implementing the changes in tr46.js and whatwg-url as well.

Marginally off-topic, but just wanted to point out that there seems to be a circular dependency between the URL Standard, Unicode IDNA spec, whatwg-url, Node.js, and WPT; specifically:

• This PR depends on the proposed draft of Unicode IDNA
• This PR depends on independent verification of implementability (Node.js or whatwg-url)
• Node.js and whatwg-url PRs depend on this PR and WPT updates before they can be applied
• WPT update depends on this PR

[domenic]

OK, looks like Node.js has a passing implementation, so let's merge this! BTW the dependency is not cyclic, as we don't need a merged PR, just an existing PR :).