Skip to content

Crawlers should support custom steps #17

@ekryski

Description

@ekryski

A crawler might look something like this.

var Crawler = require('roach').job();

Crawler.extend({
  // Your extra custom functions for parsing, etc.
  custom: function(params, data, next){
    // do some custom stuff with the data
    next(data);
  }
});

Crawler.visit(this.url)
       .custom()
       .find('a.link')
       .each()
       .click()
       .done();

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions