A generalized scraper library using JSON based instructions. It focuses on readability and reusability with a tiny api footprint.
See the github issues page for an overview of ongoing work.
npm install scrape-pagesLets download the ten most recent images from NASA's image of the day archive.
// Define some instructions for the scraper:
const instructions = `
(
FETCH 'https://apod.nasa.gov/apod/archivepix.html' LABEL='index'
PARSE 'body > b > a' ATTR='href' MAX=10
# values are passed in a stream, so whatever previous value(s) were parsed above, are each passed below
FETCH 'https://apod.nasa.gov/apod/{{ value }}'
PARSE 'img[src^="image"]' ATTR='src'
FETCH 'https://apod.nasa.gov/apod/{{ value }}' READ=false WRITE=true LABEL='image'
)`
const folder = './downloads'
const options = {}
/**
* Outside of defining configuration objects, the api is very simple. You have the ability to:
* - start (and restart) the scraper
* - listen for events from the scraper
* - stop the scraper, or any labeled command
* - query the scraped data
*/
const scraper = new ScraperProgram(instructions, folder, options)
scraper.on('image:saved', ({ id }) => console.log('saved image', id))
await scraper.start().toPromise()
const result = scraper.query(['image'])
// result is [[{ filename: 'img1.jpg' }, { filename: 'img2.jpg' }, ...]]For more real world examples, visit the examples directory
A playground exists at https://scrape-pages.js.org to help visualize scraper flows. It is also a useful way to
share a config object with others.
The ScraperProgram class is meant to create a compiled, reusable, web scraper. It is backed by a sqlite database and observables under the hood. Each time the scraper is started, it wipes previous values from the database, except for the cache. If the folder provided is empty, it will run completely fresh.
Read the API docs here. Below, important concepts are outlined:
The scraper class extends the nodejs's EventEmitter, so you have all the same methods available.
| event | callback arguments | description |
|---|---|---|
'initialized' |
after start(), initialized means scraper has begun scraping |
|
'done' |
when the scraper has completed | |
'error' |
Error | if the scraper encounters an error |
'<scraper>:progress' |
download id | emits progress of download until completed |
'<scraper>:queued' |
download id | when a download is queued |
'<scraper>:saved' |
download id | when a download is completed |
The query function grabs saved values from the database. It can be called so long as the 'initialized' event has fired. Note that scraper.query() is a convenience wrapper around scraper.query.prepare()() use the latter to achieve faster queries, as the former will rebuild sqlite statements each time it is called. View its API docs
| argument | type | type file | description |
|---|---|---|---|
| config | configinit |
src/settings/config/types.ts | pages that are being downloaded & parsed |
| options | optionsinit |
src/settings/options/types.ts | knobs to tweak download behavior |
| params | paramsinit |
src/settings/params/types.ts | inputs values and output file locations |
the pattern to download data from a website is largely similar. it can be summed up like so:
- get a page from a url
- scrape the page for more urls
- get a page
- get some text or media from page
- get a page
- scrape the page for more urls
what varies is how much nested url grabbing is required and in which steps data is saved. this project is an attempt to generalize that process into a single static config file.
describing a site crawler with a single config enforces structure, and familiarity that is less common with other scraping libraries. not only does this make yours surface api much more condensed, and immediately recognizable, it also opens the door to sharing and collaboration, since passing json objects around the web is safer than executable code. hopefully, this means that users can agree on common configs for different sites, and in time, begin to contribute common scraping patterns.
generally, if you could scrape the page without executing javascript in a headless browser, this package should be able to scrape what you wish. however, it is important to note that if you are doing high volume production level scraping, it is always better to write your own scraper code.