How can I get past the google consent dialog?

[Rezan7CC]

Hello!

I'm trying to crawl a google link, however (at least in the EU) you get a cookie consent dialog as response.
I was thinkering a bit with trying to provide a consent cookie but it didn't really work out for me, getting the consent dialog as a response still.

  // Generate date in YYYYMMDD format
    let today = chrono::Local::now().format("%Y%m%d").to_string();

    // Equivalent binary data with interpolated date
    let raw_socs = format!(
        "\x08\x01\x12\x1C\x08\x01\x12\x12gws_{}-0_RC3\x1A\x02de \x01\x1A\x06\\b\\x80º¦±\x06",
        today
    );

    // Base64 encode
    let socs_cookie = base64::encode(raw_socs);
    
    let google_consent_cookie_string = format!("SOCS={}", socs_cookie);
    let mut website = spider::website::Website::new(website_name)
        .with_cookies(&google_consent_cookie_string)
        .with_depth(3)
        .with_user_agent(Some("SpiderBot")).build().unwrap();

    website.scrape().await;

Am I missing something obvious for bypassing that dialog?

Could this be worth creating an example for perhaps?

[j-mendez]

Hi you can use the feature flag "real_browser" that should help bypass this and theres a configuration field website.configuration.dismiss_dialogs.