http2: near full http1 compatibility, add tests #15702

apapirovski · 2017-09-30T18:53:04Z

As a preface, this is a lot of changes but I had no way to separate them out into small chunks due to the order in which they happened. I started with working on support for HEAD requests and other minor http2 tweaks but as I got the express & on-finished test suites to work, I had to change a ton of that code and as a result it's hard for me to separate it out since some of these changes have broader effect on the compatibility layer.

With this PR we pass 100% of tests in express & on-finished that are applicable to http2. (There are tests in on-finished that don't apply re: handling upgrade and there are 4 tests in express that use a 3rd party module that we can't do anything about because it uses http1 prototype directly.)

Full or near full list of changes follows:

Add complete prop for Http2ServerRequest
Add method setter since some user-land modules use this
Add socket and connection on Http2ServerResponse
Add a proxy socket that is in reality referring to the Http2Stream for all the most commonly used aspects of the socket in http1
Major rework of writeHead, write, end, addHeader and removeHeader. A lot of user-land modules, including express, rely on the fact that in http1 all write methods call writeHead if headers haven't been written yet.
Add support for http1 style handling of HEAD requests (expects end call and triggers callback, even if writeHead already happened)
Dump data in Readable when kFinish runs if the user hasn't accessed it yet, this is what h1 does
No longer unset kStream and instead unset kResponse & kRequest
no longer emit an error event for the CANCEL rst code (8) — this is done mostly for h1 compatibility but also because a lot of clients seem to use this to signify client-requested abort (similar to how you could just trigger shutdown with code 0). The code is still emitted on streamClosed if anyone wants to know. Since the error emits on stream aren't actionable in http2, it seems like this has very minor implications and it allows us to support the user code already out there much easier. I could potentially just do some manipulation in my proxy socket instead but it really hampers its performance and as a result the performance of express & on-finished.
Separate commit that adds address related info on the session instead of needing to access the socket.

Performance has actually improved by about 5% with all of the changes in here, even when accessing the socket — despite the use of a Proxy.

Please let me know if I can provide more info for any of these changes. It came a long way since I started this work.

cc @jasnell, @mcollina & @dougwilson

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
commit message follows commit guidelines

Affected core subsystem(s)

http2, test

mcollina

This is a lot to digest, I'll do my best to review asap, but it might happen in a while. Can you also update the docs accordingly? I think we should document what you are doing with the Proxy.

👍 on the Express compatibility.

apapirovski · 2017-10-01T13:00:44Z

Thanks @mcollina! Pushed a tiny change that got rid of the unnecessary createProxySocket function (used to do more in it, now I can just call new Proxy directly) and also added setTimeout handler for the proxy (which sets a session timeout) since that's the one other common use case. Update: Also correctly binding socket functions now.

Will work on docs and add a commit either later today or sometime the coming week.

ronag · 2017-10-01T14:18:50Z

lib/internal/http2/compat.js

-
-  get code() {
-    return this[kState].closedCode;
+  get complete() {


What's this? I can't see this is a part of the official h1 api?

It's not documented but libraries use it.

ronag · 2017-10-01T14:27:44Z

lib/internal/http2/compat.js

@@ -387,9 +477,16 @@ class Http2ServerResponse extends Stream {
    if (typeof name !== 'string')
      throw new errors.TypeError('ERR_INVALID_ARG_TYPE', 'name', 'string');

+    if (this[kStream].headersSent)
+      throw new errors.Error('ERR_HTTP2_HEADERS_SENT');
+


We check that name is valid but not value?

assertValidHeader(name, value); takes care of value. We're going to properly validate name & value further downstream at some point (outside of compat), not yet implemented.

ronag · 2017-10-01T14:33:16Z

lib/internal/http2/compat.js

@@ -71,19 +84,24 @@ function statusMessageWarn() {
 }

 function onStreamData(chunk) {
-  if (!this[kRequest].push(chunk))
+  const request = this[kRequest];
+  if (request !== undefined && !request.push(chunk))


Should request === undefined even be possible here?

Technically? Yes. Likely? No.

jasnell · 2017-10-02T15:04:43Z

lib/internal/http2/core.js

@@ -823,6 +822,44 @@ class Http2Session extends EventEmitter {
    return settings;
  }

+  // Proxies for socket address-related getters so users don't
+  // ever need to reach through to the socket
+  get remoteAddress() {


These need to be added to the docs in /doc/api/http2.md. Should be done in this PR but can be done in a separate one if necessary

jasnell

Code changes LGTM with green CI.
Would prefer docs to be updated in this PR tho.

Really great to see this!

apapirovski · 2017-10-02T19:30:41Z

@jasnell I've been thinking over the socket warning a bit more and I'm wondering if we really should have it or not... There are some legitimate reasons to access the socket, such as with the address-related stuff but also for TLS, for stuff like getPeerCertificate(), getSession(), etc.

Realistically, it's not like we want to maintain the whole range of socket-related getters, setters and methods on the session. And it seems somewhat arbitrary as to what we actually expose on the session and what we don't.

I'm personally almost leaning towards removing the 2nd commit (http2: add socket address getters on session) and adjusting the warning to only appear if users try to use stuff like resume, pause, read, write or end on the socket proxy. Thoughts?

jasnell · 2017-10-02T19:44:41Z

Yeah, I know getting to this information is important, it's just a really bad idea to expose the socket easily with http2 connections. We've seen things like that come back and bite us in the past. Let me stew on this a bit more to see if I can think of a reasonable alternative... if there's not, then I'm good with dropping the warning for now.

apapirovski · 2017-10-02T19:47:08Z

Sounds good. I think my preferred option is to have the warning just for certain methods as mentioned above (the ones that are actually destructive) but I realize that would only apply to compatibility mode and anyone using the proper http2 API wouldn't hear a peep.

mcollina · 2017-10-03T00:20:37Z

I'm 👍 on having the warning on some of the stream-related APIs. In fact, I would not make it a warning, but I'd rather throw. If you are calling one of those methods you definitely have a bug.

Extensive re-work of http1 compatibility layer based on tests in express, on-finished and finalhandler. Fix handling of HEAD method to match http1. Adjust write, end, etc. to call writeHead as in http1 and as expected by user-land modules. Add socket proxy that instead uses the Http2Stream for the vast majority of socket interactions. Add and change tests to closer represent http1 behaviour. Refs: nodejs#15633 Refs: https://github.com/expressjs/express/tree/master/test Refs: https://github.com/jshttp/on-finished/blob/master/test/test.js Refs: https://github.com/pillarjs/finalhandler/blob/master/test/test.js

apapirovski · 2017-10-03T01:57:09Z

Ok, I've got an updated version ready to go just in case. It throws ERR_HTTP2_NO_SOCKET_MANIPULATION with the following message

HTTP/2 sockets should not be directly read from, written to, paused and/or resumed.

if the user tries to read, write, pause or resume the request.socket or response.socket. I added end to the list of things that are called on stream since hapi seems to use it.

It also removes the socket warning and the socket address related getters. Will commit or adjust further once there's consensus.

Working on the docs at the moment so should have those ready sometime tomorrow.

apapirovski · 2017-10-03T14:48:56Z

Ok, I've force pushed a commit with updates as per above and a new commit with the doc changes (plus some formatting fixes as we had the wrong heading levels for a lot of the compatibility stuff).

jasnell · 2017-10-03T21:39:39Z

Still LGTM with green CI :-)

benjamingr · 2017-10-05T23:29:28Z

CI: https://ci.nodejs.org/job/node-test-pull-request/10416

mcollina · 2017-10-06T21:05:43Z

Landed as 2da7d9b.

Extensive re-work of http1 compatibility layer based on tests in express, on-finished and finalhandler. Fix handling of HEAD method to match http1. Adjust write, end, etc. to call writeHead as in http1 and as expected by user-land modules. Add socket proxy that instead uses the Http2Stream for the vast majority of socket interactions. Add and change tests to closer represent http1 behaviour. Refs: #15633 Refs: https://github.com/expressjs/express/tree/master/test Refs: https://github.com/jshttp/on-finished/blob/master/test/test.js Refs: https://github.com/pillarjs/finalhandler/blob/master/test/test.js PR-URL: #15702 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

Extensive re-work of http1 compatibility layer based on tests in express, on-finished and finalhandler. Fix handling of HEAD method to match http1. Adjust write, end, etc. to call writeHead as in http1 and as expected by user-land modules. Add socket proxy that instead uses the Http2Stream for the vast majority of socket interactions. Add and change tests to closer represent http1 behaviour. Refs: nodejs/node#15633 Refs: https://github.com/expressjs/express/tree/master/test Refs: https://github.com/jshttp/on-finished/blob/master/test/test.js Refs: https://github.com/pillarjs/finalhandler/blob/master/test/test.js PR-URL: nodejs/node#15702 Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Matteo Collina <matteo.collina@gmail.com>

nodejs-github-bot added the http2 Issues or PRs related to the http2 subsystem. label Sep 30, 2017

mcollina reviewed Oct 1, 2017

View reviewed changes

apapirovski force-pushed the patch-http2-compat-http1 branch from 26de916 to 892a2a5 Compare October 1, 2017 12:59

apapirovski force-pushed the patch-http2-compat-http1 branch from 892a2a5 to b2fbd3e Compare October 1, 2017 13:45

ronag reviewed Oct 1, 2017

View reviewed changes

apapirovski force-pushed the patch-http2-compat-http1 branch from b2fbd3e to aa6b0d0 Compare October 2, 2017 12:00

jasnell reviewed Oct 2, 2017

View reviewed changes

jasnell approved these changes Oct 2, 2017

View reviewed changes

apapirovski force-pushed the patch-http2-compat-http1 branch from aa6b0d0 to eaec69a Compare October 3, 2017 14:45

doc: http2 socket documentation, fix formatting

9eff92c

apapirovski force-pushed the patch-http2-compat-http1 branch from eaec69a to 9eff92c Compare October 3, 2017 14:48

benjamingr approved these changes Oct 5, 2017

View reviewed changes

mcollina closed this Oct 6, 2017

MylesBorins mentioned this pull request Oct 7, 2017

v8.7.0 proposal #15762

Merged

jackhorton mentioned this pull request Oct 11, 2017

test: parallel/test-http2-server-rst-stream is flaky nodejs/node-chakracore#415

Open

apapirovski deleted the patch-http2-compat-http1 branch October 14, 2017 16:04

MylesBorins added the dont-land-on-v6.x label Oct 17, 2017

mcollina mentioned this pull request May 16, 2018

http2: fix end without read #20621

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

http2: near full http1 compatibility, add tests #15702

http2: near full http1 compatibility, add tests #15702

apapirovski commented Sep 30, 2017 •

edited

Loading

mcollina left a comment

apapirovski commented Oct 1, 2017 •

edited

Loading

ronag Oct 1, 2017

apapirovski Oct 1, 2017

ronag Oct 1, 2017

apapirovski Oct 1, 2017 •

edited

Loading

ronag Oct 1, 2017

apapirovski Oct 1, 2017

jasnell Oct 2, 2017

jasnell left a comment

apapirovski commented Oct 2, 2017 •

edited

Loading

jasnell commented Oct 2, 2017

apapirovski commented Oct 2, 2017 •

edited

Loading

mcollina commented Oct 3, 2017

apapirovski commented Oct 3, 2017

apapirovski commented Oct 3, 2017

jasnell commented Oct 3, 2017

benjamingr commented Oct 5, 2017

mcollina commented Oct 6, 2017

http2: near full http1 compatibility, add tests #15702

http2: near full http1 compatibility, add tests #15702

Conversation

apapirovski commented Sep 30, 2017 • edited Loading

Checklist

Affected core subsystem(s)

mcollina left a comment

Choose a reason for hiding this comment

apapirovski commented Oct 1, 2017 • edited Loading

ronag Oct 1, 2017

Choose a reason for hiding this comment

apapirovski Oct 1, 2017

Choose a reason for hiding this comment

ronag Oct 1, 2017

Choose a reason for hiding this comment

apapirovski Oct 1, 2017 • edited Loading

Choose a reason for hiding this comment

ronag Oct 1, 2017

Choose a reason for hiding this comment

apapirovski Oct 1, 2017

Choose a reason for hiding this comment

jasnell Oct 2, 2017

Choose a reason for hiding this comment

jasnell left a comment

Choose a reason for hiding this comment

apapirovski commented Oct 2, 2017 • edited Loading

jasnell commented Oct 2, 2017

apapirovski commented Oct 2, 2017 • edited Loading

mcollina commented Oct 3, 2017

apapirovski commented Oct 3, 2017

apapirovski commented Oct 3, 2017

jasnell commented Oct 3, 2017

benjamingr commented Oct 5, 2017

mcollina commented Oct 6, 2017

apapirovski commented Sep 30, 2017 •

edited

Loading

apapirovski commented Oct 1, 2017 •

edited

Loading

apapirovski Oct 1, 2017 •

edited

Loading

apapirovski commented Oct 2, 2017 •

edited

Loading

apapirovski commented Oct 2, 2017 •

edited

Loading