Add more options to Crawlee CLI for crawler creation #414
Assignees
Labels
enhancement
New feature or request.
t-tooling
Issues with this label are in the ownership of the tooling team.
Comments
janbuchar
added
enhancement
vdusek
changed the title
Mantisus
pushed a commit
to Mantisus/crawlee-python
that referenced
this issue
Dec 10, 2024
This adds a unified `crawlee/project_template` template. The original `playwright` and `beautifulsoup` templates are kept for compatibility with older versions of the CLI. The user is now prompted for package manager type (pip, poetry), crawler type, start URL and whether or not Apify integration should be set up. - closes apify#317 - closes apify#414 (http client selection is not implemented) - closes apify#511 - closes apify#495 ### TODO - [x] http client selection - [x] disable poetry option if it isn't installed - [x] rectify the pip-based setup 1. **manual dependency installation** - no automatic installation, just dump requirements.txt and tell the user to handle it any way they want 2. **pip+venv** - dump requirements.txt, make a virtualenv (.venv) using the current python interpreter, install requirements and tell user to activate it - ~should be disabled if `venv` module is not present~ it's stdlib - [x] test the whole thing on Windows (mainly the various package manager configurations) - [x] fix how cookiecutter.json is read (it is not present when installing via pip)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
At the moment, the Crawlee CLI provides two options for creating a crawler in the CLI:
It would be great if we could add more alternatives, such as selecting the request module (like curl cffi,httpx) and parser modules (like parsel). This would allow the project to be bootstrapped with the chosen dependencies.
Additionally, it would be super helpful to include the ability to specify the domain name in the CLI we want to start with, similar to how it’s done in Scrapy.
The text was updated successfully, but these errors were encountered: