Add a final newline normalization for form payloads

[andreubotella]

When entries are added to a form's entry list through the "append an entry" algorithm, their newlines are normalized, but entries can be added to an entry list through other means. This change adds a final newline normalization before serializing the form payload, since "append an entry" cannot be changed because its results are observable through the FormData object or through the formdata event.

This change additionally changes the input passed to the application/x-www-form-urlencoded and text/plain serializers to be a list of name-value pairs, where the values are strings rather than File objects. This simplifies the serializer algorithms.

Closes #6247. Closes whatwg/url#562.

• At least two implementers are interested (and none opposed):
  • WebKit implements this (only observable when passing a FormData object as the body of a Request or Response, since it doesn't support FACEs or the formdata event).
  • Firefox
  • Chromium
• Tests are written and can be reviewed and commented upon at:
  • Add test cases for newline handling in form payloads web-platform-tests/wpt#26740
• Implementation bugs are filed:
  • Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=1167095
  • Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1686765

(See WHATWG Working Mode: Changes for more details.)

---

/form-control-infrastructure.html ( diff )
/index.html ( diff )

[domenic]

This looks solid editorially, but I haven't been following a lot of the discussion. Tests, which help generate a list of ways browsers deviate from the tests, seem like a good next step.

I'll also note that we would probably benefit from factoring out a "normalize newlines to CRLF" operation, either in HTML (if we consider that sort of thing kinda legacy) or Infra (if we anticipate it being used more widely). I'm pretty sure that beyond just this section, there are other parts of HTML that would use it. But that could be done as a followup.

[andreubotella]

This looks solid editorially, but I haven't been following a lot of the discussion. Tests, which help generate a list of ways browsers deviate from the tests, seem like a good next step.

I've now linked to web-platform-tests/wpt#26740, which used to contain the tests for whatwg/url#562 before it grew in scope. Now I've added a lot of tests that should cover this change.

[annevk]

So I was thinking that we should check for the submission type when constructing an entry list and flatten things there inline. I guess that is not possible because of the formdata event? It would be a lot nicer if we can find an algorithm that does not have as much duplication.

cc @whatwg/forms

[andreubotella]

So I was thinking that we should check for the submission type when constructing an entry list and flatten things there inline. I guess that is not possible because of the formdata event? It would be a lot nicer if we can find an algorithm that does not have as much duplication.

You could do the normalization right after step 7 in constructing the entry list so it wouldn't affect the formdata event, but it would still affect the FormData constructor:

form.addEventListener("formdata", evt => {
    evt.formData.append("a", "b\nc");
});
const formData = new FormData(form);
assert_equals(formData.get("a"), "b\nc");  // in all browsers (except for Webkit, that doesn't support the formdata event)

You could do the normalization at some point after constructing the entry list in the form submission algorithm, but I chose to do it in the different behaviors in step 23 because enctype isn't the only piece of data that determines the serialization type. But we could test for enctype == "multipart/form-data" and an http(s) URL.

[andreubotella]

I just realized that Request and Response performing newline normalization on a FormData body (as Safari does and as tested by /FileAPI/files/send-file-formdata-*) means that the normalization must happen in the multipart/form-data encoding algorithm, not in the form submission algorithm.

[annevk]

If only Safari does it that's not necessarily what we want to do though, right? It makes some sense to do newline normalization for user input (i.e., through <form>), but I'm less convinced we should do it for programmatic input. The more we can just leave that as-is the better I think. Having said that, I don't feel too strongly. Would be great to get some input from implementers.

[andreubotella]

So, for implementer feedback, this is the current state of things across browsers, and the way this PR changes it. See #6247 for an explanation of the terms "String/File-sourced entries" and "FormData-sourced entries", but in short, the former are user inputs (and form-associated custom elements whose submission value isn't a list of entries), and the latter are programmatically generated.

• application/x-www-form-urlencoded and text/plain serializations:
  • String/File-sourced entries:
    • Current spec: names and values which derive from strings are normalized
    • This PR, Chrome, Firefox, Safari: all names and values are normalized
  • FormData-sourced entries:
    • Current spec: unchanged
    • This PR, Chrome (on application/x-www-form-urlencoded), Firefox: all names and values are normalized
    • Chrome on text/plain: unchanged except for values which derive from an original filename, which are normalized
    • Safari: N/A
• multipart/form-data serialization:
  • String/File-sourced entries:
    • Current spec, this PR, Chrome, Safari: names and string values are normalized, filenames are unchanged
    • Firefox: string values are normalized (names and filenames N/A)*
  • FormData-sourced entries:
    • Current spec, Chrome: unchanged
    • Firefox: string values are normalized (names and filenames N/A)*
    • This PR, Safari: names and string values are normalized, filenames are unchanged

*. Firefox turns newlines in multipart/form-data names and filenames into spaces. Per #6282 that's now non-conforming.

[annevk]

@mfreed7 @whatwg/forms @cdumez @tkent-google thoughts on the summary above? I still like the idea of trying not to touch FormData-sourced entries at all so developers have full control over the output there.

[andreubotella]

I still like the idea of trying not to touch FormData-sourced entries at all so developers have full control over the output there.

I just realized that this might not be simpler spec-wise for the cases of urlencoded and text/plain.

The distinction between String/File-sourced entries and FormData-sourced entries in the spec is that the former are added to the entry list through the "append an entry" algorithm, which normalizes the entry's name and string value (not filename). By the spec and Chrome (not Firefox or Safari), this normalization is visible when creating a FormData object from a form, or when observing the formdata event, and this PR doesn't change that. What it does is adding a new normalization on top before serializing the payloads, which is fine because newline normalization is idempotent.

But I opened whatwg/url#562 because (for String/File-sourced entries), the spec as it is now requires that filenames don't end up normalized in urlencoded and text/plain payloads, which goes against all implementations. We can't patch "append an entry" to either normalize the filename or turn the File value into a string value with the normalized filename, because that's observable from the formdata event*. The only way to spec this without changing the formdata behavior is to do the conversion after the "constructing the entry list" algorithm, by which point we've lost track of which entries are String/File-sourced. We could keep track of that with an additional boolean per entry, but that seems to add more complexity than the alternative.

I suspect implementing this in Chromium would have the same issue about keeping track of which entries were normalized.

*. Although we might also want to change what's observable from the formdata event, since only Chrome implements it by the spec.

[mfreed7]

Thanks, as in #6247, for the great summary of the state of things.

I'm in favor of the proposal as written, which is effectively to normalize everything except for filenames on multipart/form-data. Even filenames seem like they should be normalized, given that newlines of any kind in filenames are almost surely a mistake. But for compat reasons, this likely needs to stay as-is.

@annevk, about your suggestion to leave FormData sourced un-normalized, do you have any intuition about the use cases? My (uninformed by data) guess is that mismatched line endings probably end up causing more bugs and confusion than the corresponding number of "real" use cases that require preservation of line endings. It sounds like you disagree - if so, is there any data/evidence to point to there?

[annevk]

(Apologies for the earlier reply, I had not properly paged this back in.)

I am mainly bothered by the inelegance of normalizing twice. Also, if you use FormData with fetch() today there is no normalization (this changes that) so it seemed somewhat reasonable that if you use FormData in this context there is none either. I could see always normalizing when serializing too, but then ideally form submission itself would not do it (as it would happen later). In the end though, if we want to normalize during form submission and serialization, that is fine with me as well.

[andreubotella]

I am mainly bothered by the inelegance of normalizing twice. Also, if you use FormData with fetch() today there is no normalization (this changes that) so it seemed somewhat reasonable that if you use FormData in this context there is none either. I could see always normalizing when serializing too, but then ideally form submission itself would not do it (as it would happen later). In the end though, if we want to normalize during form submission and serialization, that is fine with me as well.

You can observe the results of constructing the entry list with new FormData(form) – at least according to the spec and Chromium. Gecko and Webkit defer the normalization to the serialization, so if the double normalization is an issue, it might be worth considering whether normalizing during the form submission is needed. See also web-platform-tests/wpt#27922.

In any case, this PR can be merged now until we decide whether to change the behavior of new FormData(form) – the second normalization doesn't depend on the first one having been applied.

Edit: Marking Firefox as interested in this PR, since I'll be implementing it.

[andreubotella]

The WPT PR (web-platform-tests/wpt#26740) is now approved, and I asked @annevk to merge it, believing this PR to have some level of consensus. I now realize that's not as clear as I thought. Let me know if I should revert it in the meantime.

In any case, I think this change is ready for another look.

[annevk]

I pushed some nits that I hope are acceptable. There are further things that could be cleaned up, but they affect a large part of the existing text as well and are probably best tackled separately (e.g., referencing the members of an entry explicitly).

Let's figure out how to land this with #6624 and then land this next week now there's implementer buy-in.

[andreubotella]

I've removed the various notes about double normalization so it's easier to land this PR and #6624 together. See #6624 (review)

[andreubotella]

I hadn't realized that the "convert to a list of name-value pairs" algorithm as I'd written it leaves filenames unnormalized in application/x-www-form-urlencoded and text/plain, which wasn't my intention, goes against the WPT tests, and doesn't match Safari. This commit should fix it.