Crawling-bot

You can download a CSV file easily using this Crawling-bot. One of the primary goals of this project is to integrate Azure OpenAI to generate meaningful data based on the user's current needs.

How to Use

1. Install required packages:

pip install -r requirements.txt

The requirements.txt file contains the dependencies needed for this project.

describe function

• crawling.py: Crawls content lists and retrieves data as a CSV file.
• detail-brute-force-crawling.py: Crawls content details and comments, then saves the data as a CSV file.
• requirements.txt: Lists project dependencies.
• crawling_result: Stores the resulting CSV files.