Dismiss cookie permission, GDPR, and subscription popups before archiving

[pirate]

Might be able to graft some code from: https://www.i-dont-care-about-cookies.eu for hiding modals and overlays.

Also:
https://greasyfork.org/de/scripts/34815-anti-adblock-killer-reek-

[pirate]

This is blocked by:

• Switch all dependencies to pure python and release ArchiveBox pip package #177 (we need pyppeteer to run user scripts to dismiss popups)
• Add ability to run JS scripts during archiving with Playwright/Puppeteer #51 support for user scripted puppeteer flows

[pirate]

Useful scripts for evading bot detection in the future: https://github.com/niespodd/browser-fingerprinting#technical-insights-into-bypassing-bot-detection

[datoslabs]

Hi,

I am curious if there's any interest in the community to incorporate AI/LLM to help solve this issue.

Recently I was working on a simple web scraper / retriever for a LLM project and came across the need to dismiss not only cookie consent but also newsletter subscription pop-ups. After some research on web UI element detection computer vision models and failed POCs, I decided to give LLM a try. I tested in Google AI studio using Gemini 2.0 Flash model; to my surprise, it appears Gemini 2.0 not only is able to detect if a pop-up (or cookie consent banner) exists on the screen, it is also able to correctly identify the XPath to close the pop-up!

The test I choose was artnet.com, it consisted of a newsletter subscription pop-up with a light gray "X" button to dismiss. Due to the light color, most web UI element CV detection models either ignored the button (to close the pop-up) completely or gave it a very low score of 30% confidence level; however, Gemini 2.0 had no problem identifying the button based on the same screen shot. Moreover when I provided it with the corresponding HTML, it was able to correctly identify the XPath which we can use in Playwright/Puppeteer/Selenium to send the click event. Following are the screenshots from one of my tests in Google AI Studio:

In my limited tests, it took 50-70 seconds to "think" (I ran it a couple times across 2 different days and both tests took >>50 seconds to process). I am sure more testing and optimization can be done to improve on this if there's enough interests to incorporate this feature into the project. Also for reference, https://github.com/browser-use/browser-use have incorporated similar technique to close pop-up dialogs; though, it appears that they highlighted each UI controls by adding colored labels and borders before taking a screen shot as an input for the LLM instead relying on LLM to identify the XPath to a specific control.