Header value definition needs work

[jeenalee]

The spec defines header value as "a byte sequence that matches the field-content token production" [0, 1].

Field-content is defined as field-content = field-vchar [ 1*( SP / HTAB ) field-vchar ] [2]. This seems to say that field-content is at most a field-vchar followed by some whitespace and another field-vchar. It seems wrong that it would be at most 2 vchars.

Should the Fetch spec define header value as "a byte sequence that matches the field-value token production"? Field-value token production is defined as field-value = *( field-content / obs-fold ) [2].

Thank you.

[0] https://fetch.spec.whatwg.org/#concept-header-name
[1] https://github.com/whatwg/fetch/blob/master/Overview.html#L440
[2] https://tools.ietf.org/html/rfc7230#section-3.2

[jeenalee]

We found an errata today, which defines field-content = field-vchar [ 1*( SP / HTAB / field-vchar ) field-vchar ] [0].

Did you mean that the header value should match the field-content production in the errata?

[0] https://www.rfc-editor.org/errata_search.php?rfc=7230 (Errata ID: 4189)

[malisas]

How is this for field-content?

field-content = [field-vchar [*(field-vchar/SP/HTAB/obs-fold) field-vchar]]

The notes from the errata linked above says:

-what the authors propably wanted to say:
a string of octets is a field-value if, and only if:
-it is *( field-vchar / SP / HTAB / obs-fold )
-if it is not empty, it starts and ends with field-vchar

With this re-write, you can:

1. have an empty field-content string, or
2. have a single field-vchar, or
3. have two field-vchars sandwiching some optional content

I guess we then get rid of field-value altogether.

[annevk]

Thank you for reporting this. I suspect it's a duplicate of either #213 or #115. Clearly something is wrong here. I also didn't know errata was filed for this particular production so thanks for pointing that out too.

@hiroshige-g have you looked into this since February? Should we just go back to what we had before HTTP was revised? Or at least something that means the same. This new production has just led to flurry of issues.

[mnot]

Note that the errata you mention is "Held for Document Update", and the notes say:

This has been edited from the original report after discussion, but even this is not right. There's more here than can be reasonably fixed in an errata report, and the proper fix needs to be done in a revision of the document -- hence, "Held for Document Update". Note that this is a valid report, and that a fix is needed. The one above is the best approach for now, and a better fix will be developed in 7230bis.

@reschke, any comment?

[reschke]

@mnot --no, we haven't figured out the best fix yet.

[annevk]

@reschke is there an open issue I can track? This has been tripping up multiple implementers now, I feel like I should add some pointers in Fetch even if it's non-final.

[reschke]

I opened httpwg/http-core#19

[annevk]

I started creating tests to sort this out since it's a rather longstanding issue. My first attempt was to pass all bytes through setRequestHeader() and new Headers().

Firefox fails for 0x00 in the former and for 0x00, 0x0A, and 0x0D in the latter. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1330297 on that since it seems silly.

Chrome consistently fails for 0x00, 0x0A, and 0x0D.

Safari TP consistently fails for 0x00-0x1F and 0x7F.

I guess next is testing what actually arrives at the server.

[annevk]

Browsers that did that not throw on 0x09, 0x0A, 0x0D, and 0x20 do appear to strip them before sending them to the server (verified through the console in Firefox, not verified in Chrome since the console is crap for that). It seems the testing server is eating 0x0B and 0x0C. Filed w3c/wptserve#111.

Chrome also doesn't appear to allow "," to roundtrip, but other browsers do manage that. Not sure what is going on there.

[annevk]

I think what I recommend is that we strip any leading and trailing 0x09, 0x0A, 0x0D, and 0x20. So a single 0x0A would not make us throw. Then, if value still contains 0x00, 0x0A, or 0x0D, throw. Then pass it through to the server unchanged.

So a header value would be a byte sequence that does not have any leading or trailing HTTP whitespace bytes and does not contain 0x00, 0x0A, or 0x0D.

[annevk]

@jeenalee @malisas @reschke @tyoshino @hiroshige-g feedback on the above header value proposal is most welcome. web-platform-tests/wpt#4525 has browser tests.

PR for the Fetch Standard to follow.

[annevk]

Also paging @youennf who implemented the relevant bits for WebKit (and made their code stricter than other browsers).

[youennf]

So the plan is to still refer to the RFC, and until the RFC is updated, add the guidelines you described above. Seems fine to me.

[annevk]

Firefox was already filed.
https://bugs.webkit.org/show_bug.cgi?id=167112
https://developer.microsoft.com/en-us/microsoft-edge/platform/issues/10552455/
https://bugs.chromium.org/p/chromium/issues/detail?id=681769

[youennf]

Looking at it a bit more closely, I think browsers currently forbid non ASCII UTF-8 content.
Since the new rule is based on bytes, my take is that it would allow such content.
I like this idea but this might depart from what browsers do today.

[youennf]

See XMLHttpRequest/setrequestheader-bogus-value.htm as an example.
WebKit (and probably Chromium) is currently checking on a character-basis.

[annevk]

@youennf I'm not sure I follow. IDL ByteString means that input > U+00FF throws. But input < U+100 becomes bytes. So you could get a UTF-8 byte sequence, if you play by the rules. E.g., if your input to setRequestHeader() is U+00E2 U+0082 U+00AC nobody would throw I believe and the byte sequence would be the UTF-8 sequence for €.

[youennf]

Ah right, WebKit related IDLs need to be fixed.

[jamshid]

I'm sorry but can someone just say Yes or No whether empty header values are legal HTTP?
If not, this RFC 7230 change causes a lot of grief for zero benefit. Maybe it's just a tightening of rules, but they were rarely if ever enforced.

[mnot]

Are they allowed by HTTP? Yes. Are they allowed / defined to have meaning for an individual header field? It depends on the field's definition. That said, it's not good practice to do it.

[Mouvedia]

It depends on the field's definition.

e.g. HTTP2-Settings
It's unclear if it's allowed though.