Web Scraper

Scrape the webpage, convert it into Markdown, and enhance AI search applications.

Running

To run this project, you need to create a config/dev.yaml configuration file. You can copy a template from config/temp.yaml.

Then, use the following command to run the project directly on your local machine, requiring Chrome to be installed:

make dev

Open the following URL in the browser:

http://127.0.0.1:4090?u=https://github.com/zzzgydi/webscraper

use HTTP GET mode:

http://127.0.0.1:4090?u=https://github.com/zzzgydi/webscraper&headless=false

use HTTP GET mode and disable readability:

http://127.0.0.1:4090?u=https://github.com/zzzgydi/webscraper&headless=false&readability=false

OR you can use it with curl:

POST /v1/scrape

Scrapes the webpage and returns the result in Markdown format.

Request Body

Parameter	Type	Description
url_list	array	List of URLs to scrape
headless	boolean	(Optional) Whether to run in headless mode
readability	boolean	(Optional) Whether to enhance readability of HTML

Example Request:

curl -X POST -H "Content-Type: application/json" \
     -d '{"url_list":["https://google.com"], "headless": false, "readability": false}' \
     http://127.0.0.1:4090/v1/scrape

This will start the server and output logs to the output/log directory.

Deployment

Create a config/prod.yaml file and set Chrome's remote_url to ws://chromedp:9222 if you run the following command:

docker compose up

Contributions

Any form of contribution is welcome. If you have any questions or suggestions, please create an issue.

Acknowledgments

• github.com/chromedp/chromedp
• github.com/PuerkitoBio/goquery
• github.com/JohannesKaufmann/html-to-markdown
• and so on...

License

This project is released under the MIT license. For details, please see the LICENSE file.