http: add uniqueHeaders option to request and createServer.

[ShogunPanda]

This PR adds a new option to both http.request and http.createServer.

The option is named separateHeadersValues (default value false) and it changes existing behavior as follows:

• For IncomingMessage, all HTTP headers are parsed and all values tracked without performing existing squashing logic.
  All values in message.headers will either be a string if the header has been received once and an array of strings if it
  has been received multiple times.
• For ClientRequest and ServerResponse, adding an header via .setHeader will append the new value to the list of existing ones. Using an array will behave as the .setHeader was called multiple times. The header will be sent multiple times with the various values.

At the moment this is only on HTTP(S). Once the approach is acknowledged, I'll extend this PR to HTTP2 as well.
Fixes #3591.

[nodejs-github-bot]

Review requested:

• @nodejs/http
• @nodejs/net

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/41736/

[mcollina]

lgtm

[mcollina]

This needs a performance check.

[apapirovski]

I think the incoming logic technically breaks the spec expectations. See https://datatracker.ietf.org/doc/html/rfc7230#section-3.2.2

I would recommend looking at how HTTP2 handles incoming headers since it even has a whitelist of known single-value headers according to the specification.

It's also worth noting that X-Robots-Tag breaks the spec expectations since it can't be combined into a single comma-delimited header, which is technically a requirement.

All of that could probably be tolerated if that's what the user wants, but it's probably worth noting in the documentation re: spec compatibility and maybe even linking to known single-value headers. That said, it does feel like using this type of approach comes with a lot of extra overhead for any user while only benefitting a few select headers...

Would love to have @jasnell chime in here.

[ShogunPanda]

@apapirovski I agree with you, in the sending phase it would break compliance if used with most well-known fields.

For custom fields, instead, it's impossible to say since their definition is not part of the standard. So the multisend might or might not be legit according to the use case.

That's also why in the tests I purposely tests against X- fields only.

As the option is disabled by the default, the existing compliant behaviour is kept and the user must explicitly opt-in (which means understanding the risks).
But, as you correctly pointed out, in this case we need to emphasise the consequences in the docs.

I'll integrate the docs shortly.
Thanks for the very good point!

After @apapirovski comment, I retract my approval

[mcollina]

Can we figure out a slightly different API / structure to allow the users to customize the behavior? So that we could support X-Robots-Tag?

Or Maybe we could handle X-Robots-Tag as a "single header field" and just support it as a special case within Node.js? It seems it's one of the major exceptions to the rule.

Wdyt @apapirovski?

[apapirovski]

Or Maybe we could handle X-Robots-Tag as a "single header field" and just support it as a special case within Node.js? It seems it's one of the major exceptions to the rule.

Yea, I was sort of imagining that the alternative could be some sort of an optional property containing a list of array-producing headers or something to that effect?

Not actually sure on the best API but seems like the downside of the approach proposed in this PR is it then forces the end-user to do their own post-processing on all headers, not just the non-standard ones like X-Robots-Tag.

[ShogunPanda]

I personally don't like adding additional specific header exceptions to the code.
Given that in the future new headers might be standardized (either de jure or de facto), then we might lose control on this.
I'm leaning towards a completely rethinking of this problem.

What about introducing a new set of optional callbacks in the API, both on IncomingMessage and OutgoingMessage which control how headers are processed?
Something like (names are provisional):

• onSetHeader: Used in OutgoingMessage when setHeader and similar are called, used to eventually modify the current logic which now completely replaces the value.
• onSerializeHeader: Used in OutgoingMessage when the data is being sent out, so the user can eventually bypass the squashing logic
• onReceiveHeader: Used in IncomingMessage when a new header is received in the request, this might be used to check if we want to join with an eventual previous value or we want to replace it.

@mcollina @apapirovski WDYT?

[mcollina]

How does the WHATWG Headers object manage this case?

I'm ok in adding a list because it does not seem to be something that requires a lot of changes over time, so maybe we are ok.

[ShogunPanda]

I've checked the WHATWG spec.
When parsing (so for IncomingMessage), they follow the same logic we currently use the same logic as WHATWG, with the noticeable exception of Set-Cookie. So we're good here.

The problem arise in OutgoingMessage. WHATWG spec technically supports appending of an header to a value, while we only support setting (which means replacing).

Now, given the original problem, we probably don't have to modify IncomingMessage. I would add (like you suggested) a list parameter to http.request and createServer, named multipleHeaders or similar, which will let the user explicitly select which headers (eventually including well-known one) they want to send multiple times with separate values.

For developer experience, I would also add the new methods in the API:

1. Add a OutgoingMessage.appendHeader method to make the use of the parameter above easier.
2. Add a IncomingMessage.headersValues which duplicates IncomingMessage.headers with the exception that values are ALWAYS lists (even for an header received once) and never joined.

By following the new methods approach we know that existing functionality or user expectation will not break and we can provide missing functionality.

What do you think?

[mcollina]

I'm good with that proposal - bear in mind that the data structure we use for Headers could hopefully be improved/made faster.

[ShogunPanda]

Hello all!
I've reworked the PR as proposed.
At the end of the day, rereading issue #3591 better, I realize the problem was mostly in IncomingMessage as the server already supports multiple headers.
I think users were using node on both on client and server and this led them to think that sending X-Robots-Tag multiple times was not supported, while this is not the case.

Anyway, the final version of this PR makes the following changes which should solve the issue forever:

1. Added a http.OutgoingMessage.appendHeader to append values to an header.
   Basically from now instead of doing this:

   message.setHeader('name', ['value1', 'value2', 'value3'])
   

   You can do:

   message.setHeader('name', 'value1')
   message.appendHeader('name', ['value2', 'value3'])
   

   Note that if there is now existing value, appendHeader falls back into setHeader so in theory one could just use the former in all use cases.

2. Added a new option uniqueHeaders in http.request and http.createServer. This is a (optional) list of headers which should only be sent once when sending the request of the response.
   If the header is present in the list and its value is an array, then sent header value will be value.join(', ').
   For instance, following the previous example, if uniqueHeaders contains name, then the sent header will be name: value1, value2, value3

3. Added two new properties in http.IncomingMessage, called headersDistinct and trailersDistinct. They behave like headers and trailers, but the array values are always an array of string. Each item is the value of a request header, so if an header is received multiple times, then the array's size is greater than one.

   For instance, given this headers:

   Connection: close
   X-Custom: value1
   X-Custom: value2
   

   Then message.headersDistinct will be:

   {
      connection: ['close'],
      'x-custom': ['value1', 'value2']
   }
   

Let me know if this works for you. If that's the case, I'll finalize this PR by checking if we can improve data structures as @mcollina suggested.

[aduh95]

This needs a rebase.

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/44091/

[mcollina]

have you tried to reduce the number of retries?

[ShogunPanda]

That was not really the issue IMHO. In some cases the benchmark CI stopped before even reaching 10ish tests

[aduh95]

@mcollina @ronag I tried to run benchmarks on the CI but the job is never able to finish. Are we confident to land this?

We should try to revive nodejs/build#2656, not sure it would solve the issue, but it would definitely help debugging.

[ShogunPanda]

Landed in abc175e