http2 server receive speed is 20x slower than send speed

[DavyLandman]

Version

v20.10.0

Platform

Linux temp-vps1 6.1.0-13-arm64 #1 SMP Debian 6.1.55-1 (2023-09-29) aarch64 GNU/Linux

Subsystem

http2

What steps will reproduce the bug?

Define a small http2 server that can send a stream, or receive a stream.

const fs = require("fs");
const stream = require("stream");
const http2 = require("http2");
const crypto = require("crypto");

const server = http2.createSecureServer({
  key: fs.readFileSync("./localhost-privkey.pem"),
  cert: fs.readFileSync("./localhost-cert.pem"),
  allowHTTP1: true, // http2 upgrade
  settings: { initialWindowSize: 8 * 1024 * 1024} 
});

server.on('error', (err) => console.log(err));

const buffer = crypto.randomBytes(256*1024*1024);

server.on("connect", (session) => {
  session.setLocalWindowSize(8*1024*1024);
});

server.on("stream", (st, headers) => {
  st.respond({
    'content-type': 'application/octet-stream',
    ":status": 200,
  });
  if (headers[':path'] === "/download") {
    stream.Readable.from(buffer).pipe(st);
  }
  else {
    st.on('data', d => st.resume());
    st.on('error', console.error);
    st.on('end', () => st.close());
  }
})

server.listen(8443, () => {
  console.log("HTTPS (HTTP/2) listening on 8443 port");
});

generate ssl certs:

openssl req -x509 -newkey rsa:2048 -nodes -sha256 \
   -subj '//CN=localhost' -keyout localhost-privkey.pem -out localhost-cert.pem 

go to a different machine (preferably on the same local network) and run:

curl --http2 -k https://other-machine:8443/download --output /dev/null
dd if=/dev/urandom of=test.bin bs=100M count=4
curl --http2 -k -T test.bin https://chunky.swat.other-machine:8443/upload

observer the difference in speed between up and downstream.

$ curl --http2 -k https://other-machine:8443/download --output /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  256M    0  256M    0     0   356M      0 --:--:-- --:--:-- --:--:--  356M

$ curl --http2 -k -T test.bin https://other-machine:8443/upload
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  400M    0     0  100  400M      0  11.7M  0:00:33  0:00:33 --:--:-- 11.7M

How often does it reproduce? Is there a required condition?

I've manged to reproduce this with multiple vps on different cloud providers. switching around server and client side.

Loopback never reproduces it, and sometimes if you have 2 machines on the same switch, it also doesn't reproduce.

I suspect it's some kind of window related issue, I've already applied the manual fixes suggested in #38426. Without these the download speed is also this slow.

What is the expected behavior? Why is that the expected behavior?

The performance should be comparable to raw tcp speed (with some lee-way for http2 overhead).

See netcat performance between the 2 machines:

on server:

nc -l -p 5050 > /dev/null

on second machine:

 pv -a test.bin| nc other-machine 5050
[ 492MiB/s]

using nodejs as a http2 client has same performance issue

const fs = require("fs");
const http2 = require("http2");
const process = require("process");
const crypto = require("crypto");
const stream = require("stream");

const buffer = crypto.randomBytes(256*1024*1024);

let round = 0;
function upload() {
  const client = http2.connect(process.env["TARGET_HOST"] ?? "https://localhost:8443", {
    ca: fs.readFileSync("./localhost-cert.pem"),
    checkServerIdentity: () => undefined,
    settings: { initialWindowSize: 8 * 1024 * 1024}
  });

  client.once('remoteSettings', settings => {
    client.setLocalWindowSize(settings.initialWindowSize ?? (8 * 1024 * 1024));
  });

  const recv = client.request({":path": "/upload"}, {endStream: false});

  recv.on("response", () => {
    const start = process.hrtime();
    stream.Readable.from(buffer)
      .pipe(recv)
      .on("end", () => {
        const [timeSpendS, timeSpendNS] = process.hrtime(start);
        const mbSize = buffer.length / (1024 * 1024);
        console.log(
          `Uploading to h2 speed (${Math.trunc(mbSize)} MB): ${
            mbSize / (timeSpendS + timeSpendNS * 1e-9)
          } MB/s`
        );
        client.close();
        round++;
        if (round < 3) {
          // to give node a fair chance of getting close to curl
          // we go throught the upload 3 times (sequentually).
          // such that node has had time to JIT etc.
          upload();
        }
      });
  });
}


upload()

invoke with: TARGET_HOST="https://other-machine:8443" node n2-upload.js

What do you see instead?

Slow upload while the tcp connnection speed would allow for fast upload.

Additional information

I'm closing a previous issue (#50972) I reported, that one got too complex and big. This is a much smaller case and easier to debug, I hope. If there is something I can do to help out, please let me know.

I think this is not a dupe of #38426 as even when you manually setup a big enough window, the receive side is still slow. I suspect there is a call to nghttp2 missing that is not exposed on the nodejs side?

[preveen-stack]

else {
    st.on('data', d => st.resume());
    st.on('error', console.error);
    st.on('end', () => st.close());
  }

Wondering, if piping to a filestream would make any difference!

[DavyLandman]

Finally, I realized the documentation was wrong. The connect even never fired on the server side, so the setLocalWindowSize was never invoked. Changing this to session event got the performance for upload the same.

So it should have been:

    server.on("session", (session) => {
        session.setLocalWindowSize(windowSize);
    });