HTTP2 server memory explodes and terminated

[vilicvane]

Version

20.13.0 and later, including latest 22.x

Platform

Linux hostname 6.5.0-28-generic #29~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr 4 14:39:20 UTC 2 x86_64 x86_64 x86_64 GNU/Linux

Subsystem

No response

What steps will reproduce the bug?

I may need some help to narrow down the problem as it happens sporadically.

Currently I only know that it happens after:

1. A session created.
2. The server pushStream to the client (and callback called).
3. The client sends a request to create another stream and the server hangs.

I have tested with versions 20.10.0, 20.11.0, 20.12.0 and 20.12.1 and there's no problem. But since 20.13.0 the issue appears.

How often does it reproduce? Is there a required condition?

No response

What is the expected behavior? Why is that the expected behavior?

No response

What do you see instead?

I tried to use --inspect and pause on hanging (also the memory usage increases rapidly to several gigabytes and then terminated by OS I think) but no luck.

And tried --prof there's part of that looks suspicious:

   ticks parent  name
  27305   70.6%  /usr/bin/node
  24662   90.3%    JS: ^submitRstStream node:internal/http2/core:450:25
  24662  100.0%      JS: ^finishCloseStream node:internal/http2/core:1963:27
  24662  100.0%        JS: ^closeStream node:internal/http2/core:1930:21
  24662  100.0%          JS: ^_destroy node:internal/http2/core:2325:11
  24662  100.0%            JS: ^_destroy node:internal/streams/destroy:90:18

I am not a security guy but at first I thought it's something related to HTTP2 rapid reset attack and spent a whole day debugging towards that. At the end of the day I decided to downgrade Node.js and found out the older version just works.

Additional information

Also the HTTP2 server is on internet but generally only used by myself. Though I do saw a Russian IP connecting it periodically and that's also why I thought it was an attack.

But it also happens with only my own clients connected.

[benjamingr]

@nodejs/http2

[mcollina]

Can you provide steps to reproduce? We often need a reproducible example, e.g. some code that allows someone else to recreate your problem by just copying and pasting it. If it involves more than a couple of different file, create a new repository on GitHub and add a link to that.

[mcollina]

These were the commits in v20.13.0 that showed the problem

3bcd68337e] - http2: fix excessive CPU usage when using allowHTTP1=true (Eugene) #52713
[e01015996a] - http2: fix h2-over-h2 connection proxying (Tim Perry) #52368
[9f88736860] - http2: use internal addAbortListener (Chemi Atlow) #52081

[vilicvane]

@mcollina I will do a further narrowing down and try to get a minimal repro later, was expecting some hint so I can reach it easier (as I've already spent a whole day on this issue so have to deprioritize it for now).

[moish83]

@vilicvane, we started using http2 for calling one of our third party vendors, we are calling ~150-200K request per hour. Since we moved those calls to http2 via got and http2-wrapper, we started seeing pod crashes/restarts every once in a while, and it seems to have the same pattern as described in this issue, a sudden spike in cpu and memory followed by a restart.
We do see GOAWAY frames received from the server, not sure they are related though.
NodeJS version 22.12.0