v2.6.15

• fix parsing links for top level redirected domains
• add website.with_preserve_host_header
• default tls reqwest_native_tls_native_roots

Full Changelog: v2.5.2...v2.6.15

HTML Transformations

Whats Changed

We Open Sourced our transformation utils for Spider cloud that provides high performance output to markdown, text, and other formats.

You can install spider_transformations on it's own or use the feature flag transformations when installing spider_utils.

use spider::tokio;
use spider::website::Website;
use spider_utils::spider_transformations::transformation::content::{
    transform_content, ReturnFormat, TransformConfig,
};
use tokio::io::AsyncWriteExt;

#[tokio::main]
async fn main() {
    let mut website: Website = Website::new("https://rsseau.fr");
    let mut rx2: tokio::sync::broadcast::Receiver<spider::page::Page> =
        website.subscribe(0).unwrap();
    let mut stdout = tokio::io::stdout();

    let mut conf = TransformConfig::default();
    conf.return_format = ReturnFormat::Markdown;

    let join_handle = tokio::spawn(async move {
        while let Ok(res) = rx2.recv().await {
            let markup = transform_content(&res, &conf, &None, &None);

            let _ = stdout
                .write_all(format!("- {}\n {}\n", res.get_url(), markup).as_bytes())
                .await;
        }
        stdout
    });

    let start = std::time::Instant::now();
    website.crawl().await;
    website.unsubscribe();
    let duration = start.elapsed();
    let mut stdout = join_handle.await.unwrap();

    let _ = stdout
        .write_all(
            format!(
                "Time elapsed in website.crawl() is: {:?} for total pages: {:?}",
                duration,
                website.get_links().len()
            )
            .as_bytes(),
        )
        .await;
}

Full Changelog: v2.5.2...v2.6.2

v2.5.3

Whats Changed

1. Add visited links string interning performance increase on look up and memory reduction on store. #204

Full Changelog: v2.4.1...v2.5.3

v2.4.1

Whats Changed

The screenshot performance has drastically increased by taking advantage of chromes params to handle full_screen without re-adjusting the layout and the optimize_for_speed param. This works well the concurrent interception handling to avoid stalling on re-layout. If you use the crawler to take screenshots it is recommended to upgrade.

• perf(chrome): add major screenshot performance custom command
• chore(utils): add trie match all base path
• chore(examples): add css scraping example

Full Changelog: v2.3.5...v2.4.1

v2.3.5

Whats Changed

Major performance improvement on chrome enabling concurrent request interception for resource heavy pages.

• add response headers when chrome is used
• add hybrid cache response and headers chrome
• fix chrome sub pages setup
• perf(chrome): add concurrent request interception

Full Changelog: v2.2.18...v2.3.5

v2.2.18

Whats Changed

We can now auto detect locales without losing out on performance. We default enabled the encoding flag for this change!

• get_html now properly encodes the HTML instead of UTF8 default encoding
• bump chromiumoxide@0.7.0
• fix chrome hang on ws connections handler
• fix fetch stream infinite loop on error
• fix chrome frame setting url ( this temp prevents hybrid caching from having the req/res for the page )

let mut website: Website = Website::new("https://tenki.jp");
// all of the content output has the proper encoding automatically

Full Changelog: v2.1.9...v2.2.18

v2.1.9

Whats New

This release brings bug fixes with chrome opening pages causing hangs. The builder method website.with_return_page_links can be used to attach the links found on the web page to the page object.

• chore(chrome): fix instances being left open from ignorable handler errors
• chore(scrape): add sitemap and smart [#206]
• feat(page): add return page links configuration
• chore(config): fix budget reset on crawl end

Thanks @DimitriTimoz

Full Changelog: v2.0.6...v2.1.9

v2.0.6

What's Changed

• add http response cookies map
• fix chrome fs feature flag build
• Update README.md by @James4Ever0 in #203

New Contributors

• @James4Ever0 made their first contribution in #203

Full Changelog: v2.0.3...v2.0.6

v2.0.3

Whats Changed

1. Scrape and Crawl now perform functionality identically as scrape re-uses crawl underneath.
2. Scrape API cleanup
3. Add get_chrome_page chrome page ref

Full Changelog: v1.99.30...v2.0.3

v1.99.30

Whats Changed

• feat Web automation steps by target url or path.
• add internal ViewPort for chrome handling.
• add partial eq configuration

    let mut automation_scripts = HashMap::new();

    automation_scripts.insert(
        "/en/blog".into(),
        Vec::from([
            WebAutomation::Evaluate(r#"document.body.style.background = "blue";"#.into()),
            WebAutomation::ScrollY(2000),
            WebAutomation::Click("article a".into()),
            WebAutomation::Wait(5000),
            WebAutomation::Screenshot {
                output: "example.png".into(),
                full_page: true,
                omit_background: true,
            },
        ]),
    );

    let mut website: Website = Website::new("https://rsseau.fr/en/blog")
        .with_chrome_intercept(true, true)
        .with_wait_for_idle_network(Some(WaitForIdleNetwork::new(Some(Duration::from_secs(30)))))
        .with_caching(cfg!(feature = "cache"))
        .with_limit(1)
        .with_automation_scripts(Some(automation_scripts))
        .build()
        .unwrap();

web-automation-chrome.mov

Full Changelog: v1.99.21...v1.99.30