Offset (multiple API Calls per parameter) causes fetching problems

[gregorsart]

Our connected API only allows 20 results per fetch. But The API has a total of 200 dog breeds. Because with offset=0 you end up with the dogs beginning with "A", & "B" roughly, and with offset=180 you end up with the end of the alphabet, we decided to fetch 9 times so we get the full range.

All this happens in the backend code (api route)
We started with a for loop to get all fetches for all offsets.

    let dogPromises = [];
    for (let offset = 0; offset < 200; offset += 20) {
      dogPromises.push(
        `https://api.api-ninjas.com/v1/dogs?barking=${barking}&offset=${offset}`,
        {
          headers: {
            "X-Api-Key": process.env.DOG_API_KEY,
          },
        }
      );
    }

then we try to get a promises resolved and an array with an data object:

    const dogResponses = await Promise.all(dogPromises);
    const dataBarking = await dogResponses[0].json();

    response.status(200).json([...dataBarking]);
    return;
  }

  response.status(405).json({ message: "Method not allowed" });

We do not get errors (good thing) but we also do not get a filled array back, instead we get an empty array.

Help appreciated

[NikRoe]

Hi Gregor,

please send us a PR with the current state of your app :)

[gregorsart]

Dog App // Branch API Connect

It is a bit messy (commented out code everywhere), sry for that!

[NikRoe]

awesome, thanks! Will take a look

[NikRoe]

You were almost there!

When pushing new entries to the dogPromises Array, you need to adapt your code to this:

for (let offset = 0; offset < 200; offset += 20) {
      dogPromises.push(
        fetch(
          `https://api.api-ninjas.com/v1/dogs?barking=${barking}&offset=${offset}`,
          {
            headers: {
              "X-Api-Key": process.env.DOG_API_KEY,
            },
          }
        )
      );
    }

By doing this, you will get an array of your fetches, where each fetch returns a promise.

Then you can work with Promise.all and will get an Array of all responses.

const dogResponses = await Promise.all(dogPromises);

For each response object in the dogResponses array, the .json() method is called. The .json() method returns a promise that resolves with the result of parsing the body text as JSON.

Since calling .json() on each response gives another array of promises, we use Promise.all again to wait for all of these promises to resolve. The result is dataBarkingArray, an array of the parsed JSON data from each response.

const dataBarkingArray = await Promise.all(
      dogResponses.map((response) => response.json())
    );

[moonwave99]

Exactly @NikRoe !

If you want to refactor it a bit and make it a bit more functional, this may be interesting:

async function getResponse({ barking, offset }) {
  const params = new URLSearchParams({ barking, offset });
  const response = await fetch(
    `https://api.api-ninjas.com/v1/dogs?${params}`,
    {
      headers: {
        'X-Api-Key': process.env.DOG_API_KEY,
      },
    }
  );
  return await response.json();
}

async function getAllResponses(barking) {
  const totalRecords = 200;
  const recordsPerPage = 20;
  // generates an array like [0, 20, 40, ..., 180]
  const offsets = Array.from(
    { length: Math.floor(totalRecords / recordsPerPage) },
    (_, index) => index * recordsPerPage
  );  
  return Promise.all(
    offsets.map((offset) => getResponse({ barking, offset }))
  );
}

If you want to merge all results in one, you can do:

const pages = Promise.all(
  offsets.map((offset) => getResponse({ barking, offset }))
);

return pages.flat();

See Array.prototype.flat.

[gregorsart]

Thank you both for the detailed answer. We already use it in our code and it works very well.