Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Improved project bootstrapping #538

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Conversation

janbuchar
Copy link
Collaborator

@janbuchar janbuchar commented Sep 20, 2024

This adds a unified crawler template. The original playwright and beautifulsoup templates are kept for compatibility with older versions of the CLI.

The user is now prompted for package manager type (pip, poetry), crawler type, start URL and whether or not Apify integration should be set up.

@janbuchar janbuchar added the t-tooling Issues with this label are in the ownership of the tooling team. label Sep 20, 2024
@github-actions github-actions bot added this to the 98th sprint - Tooling team milestone Sep 20, 2024
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

@github-actions github-actions bot added the tested Temporary label used only programatically for some analytics. label Sep 20, 2024
Copy link

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Pull Request Tookit has failed!

Pull request is neither linked to an issue or epic nor labeled as adhoc!

@janbuchar
Copy link
Collaborator Author

okay, this could use some testing now... plus there's a bunch of stuff to consider

  • the pip thing is probably not very ergonomic and I'm not sure how to approach it well... should we create a .venv for users? should we have a requirements.txt and use it as a lockfile?
  • does this work on windows?
  • isn't the creation dialog too long now? maybe some of the options don't need the prompt and just CLI flags are enough
  • should we include http client selection?

@janbuchar janbuchar marked this pull request as ready for review September 20, 2024 12:33
Copy link
Collaborator

@vdusek vdusek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, just a few comments

@@ -0,0 +1,12 @@
{
"project_name": "crawlee-python-beautifulsoup-project",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it correct?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I'll probably just change it to my-crawler or something.

@@ -0,0 +1,37 @@
# {{cookiecutter.project_name}}

Project skeleton generated by Crawlee (Beautifulsoup template).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it always BS?

src/crawlee/_cli.py Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
t-tooling Issues with this label are in the ownership of the tooling team. tested Temporary label used only programatically for some analytics.
Projects
None yet
2 participants