Skip to content

Latest commit

 

History

History

examples

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
Cargo.toml
Cargo.toml
 
 
README.md
README.md
 
 
advanced_configuration.rs
advanced_configuration.rs
 
 
auto_encoding.rs
auto_encoding.rs
 
 
blacklist.rs
blacklist.rs
 
 
budget.rs
budget.rs
 
 
cache.rs
cache.rs
 
 
cache_chrome_hybrid.rs
cache_chrome_hybrid.rs
 
 
callback.rs
callback.rs
 
 
chrome.rs
chrome.rs
 
 
chrome_screenshot.rs
chrome_screenshot.rs
 
 
chrome_screenshot_with_config.rs
chrome_screenshot_with_config.rs
 
 
chrome_viewport.rs
chrome_viewport.rs
 
 
chrome_web_automation.rs
chrome_web_automation.rs
 
 
configuration.rs
configuration.rs
 
 
cron.rs
cron.rs
 
 
css_scrape.rs
css_scrape.rs
 
 
debug.rs
debug.rs
 
 
depth.rs
depth.rs
 
 
download.rs
download.rs
 
 
download_to_react.rs
download_to_react.rs
 
 
encoding.rs
encoding.rs
 
 
example.rs
example.rs
 
 
loop.rs
loop.rs
 
 
openai.rs
openai.rs
 
 
openai_cache.rs
openai_cache.rs
 
 
openai_extra.rs
openai_extra.rs
 
 
openai_multi.rs
openai_multi.rs
 
 
queue.rs
queue.rs
 
 
rss.rs
rss.rs
 
 
scrape.rs
scrape.rs
 
 
serde.rs
serde.rs
 
 
sitemap.rs
sitemap.rs
 
 
smart.rs
smart.rs
 
 
subscribe.rs
subscribe.rs
 
 
subscribe_download.rs
subscribe_download.rs
 
 
transform_markdown.rs
transform_markdown.rs
 
 
url_glob.rs
url_glob.rs
 
 
url_glob_subdomains.rs
url_glob_subdomains.rs
 
 
whitelist.rs
whitelist.rs
 
 

README.md

Examples

First git clone https://github.com/spider-rs/spider.git and cd spider. Use the release flag for the best performance --release when running the examples below.

Basic

Simple concurrent crawl Simple.

  • cargo run --example example

Subscribe to realtime changes Subscribe.

  • cargo run --example subscribe

Live handle index mutation example Callback.

  • cargo run --example callback

Enable log output Debug.

  • cargo run --example debug

Scrape the webpage with and gather html Scrape.

  • cargo run --example scrape

Scrape and download the html file to fs Download HTML. *Note: Enable feature flag [full_resources] to gather all files like css, jss, and etc.

  • cargo run --example download

Scrape and download html to react components and store to fs Download to React Component.

  • cargo run --example download_to_react

Crawl the page and output the links via Serde.

  • cargo run --example serde --features serde

Crawl links with a budget of amount of pages allowed Budget.

  • cargo run --example budget

Crawl links at a given cron time Cron.

  • cargo run --example cron

Crawl links with chrome headed rendering Chrome.

  • cargo run --example chrome --features chrome_headed

Crawl links with view port configuration Chrome Viewport.

  • cargo run --example chrome_viewport --features chrome

Take a screenshot of a page during crawl Chrome Screenshot.

  • cargo run --example chrome_screenshot --features="spider/sync spider/chrome spider/chrome_store_page"

Crawl links with smart mode detection. Runs HTTP by default until Chrome Rendering is needed. Smart.

  • cargo run --example smart --features smart

Use different encodings for the page. Encoding.

  • cargo run --example encoding --features encoding

Use advanced configuration re-use. Advanced Configuration.

  • cargo run --example cache_chrome_hybrid --features="spider/sync spider/chrome spider/cache_chrome_hybrid"

Use chrome hybrid caching. Chrome Cache Hybrid.

  • cargo run --example advanced_configuration

Use URL globbing for a domain. URL Globbing.

  • cargo run --example glob --features glob

Use URL globbing for a domain and subdomains. URL Globbing Subdomains.

  • cargo run --example url_glob_subdomains --features glob

Downloading files in a subscription. Subscribe Download.

  • cargo run --example subscribe_download

Add links to gather mid crawl. Queue.

  • cargo run --example queue

Use OpenAI to get custom Javascript to run in a browser. OpenAI. Make sure to set OPENAI_API_KEY=$MY_KEY as an env variable or pass it in before the script.

  • cargo run --example openai

or

  • OPENAI_API_KEY=replace_me_with_key cargo run --example openai

or setting multiple actions to drive the browser

  • OPENAI_API_KEY=replace_me_with_key cargo run --example openai_multi

or to get custom data from the GPT with JS scripts if needed.

  • OPENAI_API_KEY=replace_me_with_key cargo run --example openai_extra