How do I get the result of octokit.issues.listComments as a stream?

[AleksandrHovhannisyan]

Hello! For a while now, I've been using the GitHub Issues API as a makeshift comment system for my personal website. I open one issue per article that I've written, and users can leave comments that my page then queries for. My server-side code uses the Octokit SDK to make authenticated requests to the API:

https://github.com/AleksandrHovhannisyan/aleksandrhovhannisyan.com/blob/34ebe5a22114937b21916737125cf65fb2d67554/api/comments.js#L43-L54

const response = await commentApi.paginate(
  commentApi.issues.listComments,
  {
    owner: site.issues.owner,
    repo: site.issues.repo,
    issue_number: issueNumber,
    per_page: 100,
  },
  (response) => response.data
);

Unfortunately, this tends to be a bit slow when I need to fetch lots of comments. I'm new to HTTP streaming as a concept but I read that it could help me with this problem, so that instead of buffering all the responses and sending them in one go, my server could stream the results to the client one by one as they arrive. At least that's my tenuous understanding.

I did a bit of research and found this thread on the octokit/request.js repo: [FEAT]: Allow streaming of the data. In that thread, it is suggested that support was added to stream data from Octokit endpoints. One of the code samples referenced in that issue is this code for streaming assets:

import { Octokit } from "@octokit/rest";
import { createWriteStream } from "node:fs";
import { join } from "node:path";
import { pipeline } from "node:stream/promises";

async function fetchAsset() {
  const octokit = new Octokit({ auth: process.env.GITHUB_TOKEN });
  const asset = await octokit.rest.repos.getReleaseAsset({
    owner: "foxglove",
    repo: "app",
    asset_id: 12345,
    headers: {
      accept: "application/octet-stream",
    },
    request: {
      parseSuccessResponseBody: false, // required to access response as stream
    },
  });

  const assetStream = asset.data as unknown as NodeJS.ReadableStream;
  const outputFile = createWriteStream(join("outputdir", "file.zip"));
  await pipeline(assetStream, outputFile);
}

So I updated my code to include the headers and request objects:

const response = await commentApi.paginate(
  commentApi.issues.listComments,
  {
    owner: site.issues.owner,
    repo: site.issues.repo,
    issue_number: issueNumber,
    per_page: 100,
    headers: {
      accept: "application/octet-stream",
    },
    request: {
      parseSuccessResponseBody: false,
    },
  },
  (response) => response.data
);

Unfortunately, when I do that, I get the following error upon making a request:

RequestError [HttpError]: Unsupported 'Accept' header: 'application/octet-stream'. Must accept 'application/json'. - https://docs.github.com/v3/media

Does this mean that streaming is not supported for the listComments API?

Environment:

• @octokit/auth-token: 5.1.1
• @octokit/core: 6.1.2
• @octokit/rest: 21.0.1

[wolfy1339]

You are sending the wrong header. Remove the custom accept header and your request should work

[AleksandrHovhannisyan]

Hey, thanks for your patience and help. The strange thing is that when I remove the headers object and only pass in the request params:

const response = await commentApi.paginate(
  commentApi.issues.listComments,
  {
    owner: site.issues.owner,
    repo: site.issues.repo,
    issue_number: issueNumber,
    per_page: 100, // this is the max number of results per page that the API supports
    request: {
      parseSuccessResponseBody: false,
    },
  },
  (response) => response.data
);

The response I get back from the API looks identical to what I get without it. And I can iterate over it as a normal array, just like before. Is that expected? Maybe I'm misunderstanding something about streams. (Again, lots of knowledge gaps here that I'm trying to fill.)