Simplified requests for paged (paginated) content.
Please consider following this project's author, Jon Schlinkert, and consider starring the project to show your ❤️ and support.
Install with npm:
$ npm install --save paged-requestSee the release notes for information about changes made in v2.0.
This library recursively calls needle's .get method as long as the user-provided next() function returns a string (the next url to get). See an example.
Example
const request = require('paged-request');
request(url, options, next)
.then(acc => console.log(acc.pages.length))
.catch(console.error);url{string} - (required) the initial url to getoptions{object} - (optional) options object to pass to needlenext{function} - (required) function that returns the next url to get, a promise or undefined.
url{string} - the original (base) user-provided urlresp{object} - needle response objectacc{object} - accumulator object with the following properties:options{object} - user-provided options objectpages{array} - array of responsesurls{array} - array of requested urls
The next function should return a string (the next url to get), promise or undefined.
The following example shows how to loop over pages of CSS posts on smashingmagazine.com (an arbitrary example, but they have great content!).
const request = require('paged-request');
async function next(url, resp, acc) {
// do stuff to check response first if necessary
const regex = /href="\/.*?\/(\d+)\/"/;
const num = (regex.exec(resp.data) || [])[1];
if (num && /^[0-9]+$/.test(num) && +num <= n) {
// use the "original" url to avoid having to reparse
// and recreate the url each time
return `${acc.orig}/page/${num}/`;
}
}
request('https://www.smashingmagazine.com/category/css', {}, next)
.then(acc => console.log(acc.pages.length))
.catch(console.error);- renamed
.hrefsto.urlsin response object - now using axios instead of needle. Please see the axios documentation for API information.
Contributing
Pull requests and stars are always welcome. For bugs and feature requests, please create an issue.
Please read the contributing guide for advice on opening issues, pull requests, and coding standards.
Running Tests
Running and reviewing unit tests is a great way to get familiarized with a library and its API. You can install dependencies and run tests with the following command:
$ npm install && npm testBuilding docs
(This project's readme.md is generated by verb, please don't edit the readme directly. Any changes to the readme must be made in the .verb.md readme template.)
To generate the readme, run the following command:
$ npm install -g verbose/verb#dev verb-generate-readme && verbYou might also be interested in these projects:
- gists: Methods for working with the GitHub Gist API. Node.js/JavaScript | homepage
- github-base: Low-level methods for working with the GitHub API in node.js/JavaScript. | homepage
- repos: Tiny wrapper around github-base for getting publicly available information for a repository, or all of… more | homepage
| Commits | Contributor |
|---|---|
| 12 | jonschlinkert |
| 9 | doowb |
| 2 | whitneyit |
Jon Schlinkert
Copyright © 2021, Jon Schlinkert. Released under the MIT License.
This file was generated by verb-generate-readme, v0.8.0, on January 20, 2021.