Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/crawlee/_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -226,10 +226,10 @@ def create(
f'activate the virtual environment in ".venv" ("source .venv/bin/activate") '
f'and run your project using "python -m {package_name}".'
)
elif package_manager == 'poetry':
elif package_manager in {'poetry', 'uv'}:
typer.echo(
f'To run it, navigate to the directory: "cd {project_name}", '
f'and run it using "poetry run python -m {package_name}".'
f'and run it using "{package_manager} run python -m {package_name}".'
)

typer.echo(f'See the "{project_name}/README.md" for more information.')
Expand Down
2 changes: 1 addition & 1 deletion src/crawlee/project_template/cookiecutter.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"crawler_type": ["beautifulsoup", "parsel", "playwright", "playwright-camoufox"],
"__crawler_type": "{{ cookiecutter.crawler_type|lower|replace('-', '_') }}",
"http_client": ["httpx", "curl-impersonate"],
"package_manager": ["poetry", "pip", "manual"],
"package_manager": ["poetry", "pip", "uv", "manual"],
"enable_apify_integration": false,
"start_url": "https://crawlee.dev",
"_jinja2_env_vars": {
Expand Down
12 changes: 10 additions & 2 deletions src/crawlee/project_template/hooks/post_gen_project.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,21 @@

Path('_pyproject.toml').rename('pyproject.toml')

# % if cookiecutter.package_manager == 'poetry'
# % if cookiecutter.package_manager in ['poetry', 'uv']
Path('requirements.txt').unlink()

# % if cookiecutter.package_manager == 'poetry'
subprocess.check_call(['poetry', 'install'])
# % elif cookiecutter.package_manager == 'uv'
subprocess.check_call(['uv', 'sync'])
# % endif

# % if cookiecutter.crawler_type == 'playwright'
subprocess.check_call(['poetry', 'run', 'playwright', 'install'])
manager = "{{ cookiecutter.package_manager }}"
subprocess.check_call([manager, 'run', 'playwright', 'install'])
# % endif


# % elif cookiecutter.package_manager == 'pip'
import venv # noqa: E402

Expand Down
21 changes: 15 additions & 6 deletions src/crawlee/project_template/hooks/pre_gen_project.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,21 @@
# % if cookiecutter.package_manager == 'poetry'
# % if cookiecutter.package_manager in ['poetry', 'uv']
import subprocess
import re

manager = "{{cookiecutter.package_manager}}"
manager_text = manager.title()
# % if cookiecutter.package_manager == 'poetry'
version_regex = r'Poetry \(version 2\..*\)'
r_version = '2.x'
# % elif cookiecutter.package_manager == 'uv'
version_regex = r'uv (0\..*)'
r_version = '0.x'
# % endif

try:
version = subprocess.check_output(['poetry', '--version']).decode().strip()
version = subprocess.check_output([manager, '--version']).decode().strip()
except OSError as exc:
raise RuntimeError('You chose to use the Poetry package manager, but it does not seem to be installed') from exc

if not re.match(r'Poetry \(version 2\..*\)', version):
raise RuntimeError(f'Poetry 2.x is required, but "{version}" is installed')
raise RuntimeError(f'You chose to use the {manager_text} package manager, but it does not seem to be installed') from exc
if not re.match(version_regex, version):
raise RuntimeError(f'{manager_text} {r_version} is required, but "{version}" is installed')
# % endif
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,28 @@ RUN echo "Python version:" \
pip install -r /dev/stdin --no-dependencies \
&& echo "All installed Python packages:" \
&& pip freeze
# % elif cookiecutter.package_manager == 'uv'
RUN pip install -U pip setuptools \
&& pip install 'uv<1'

ENV UV_PROJECT_ENVIRONMENT="/usr/local"

COPY pyproject.toml uv.lock ./

RUN echo "Python version:" \
&& python --version \
&& echo "Installing dependencies:" \
# Check if playwright is already installed
&& PLAYWRIGHT_INSTALLED=$(pip freeze | grep -q playwright && echo "true" || echo "false") \
&& if [ "$PLAYWRIGHT_INSTALLED" = "true" ]; then \
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I follow, so I will think aloud:

This is here because we might already have some version of playwright installed in the base image in location "/usr/local". When we do, we exclude it from uv sync -> which btw syncs to where the python installation lives (so no isolated environment, but isolation is the image - so who cares).

If there is specific playwright version requirement(probably just theoretical use case) - we will ignore it and take whatever version already exists in the base image.

Is my understanding correct?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's right, playwright may already be installed in the standard FROM apify/actor-python-playwright image.

This is similar to the logic used in poetry, but keeping the features that uv provides

echo "Playwright already installed, excluding from uv sync" \
&& uv sync --frozen --no-install-project --no-editable -q --no-dev --inexact --no-install-package playwright; \
else \
echo "Playwright not found, installing all dependencies" \
&& uv sync --frozen --no-install-project --no-editable -q --no-dev --inexact; \
fi \
&& echo "All installed Python packages:" \
&& pip freeze
# % elif cookiecutter.package_manager == 'pip'
RUN pip install -U pip setuptools

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,37 @@ python -m pip install .

When the dependencies are installed, you may launch the crawler with:

```sh
python -m {{cookiecutter.__package_name}}
```

{% elif cookiecutter.package_manager == 'uv' -%}
To get started, ensure you have [UV](https://docs.astral.sh/uv/), a package and dependency management system, installed on your machine. We recommend installing it with the following command:

```sh
pipx install uv
```

Next, install the project dependencies:

```sh
uv sync
```

Finally, launch the crawler with:

```sh
uv run python -m {{cookiecutter.__package_name}}
```
{% elif cookiecutter.package_manager == 'pip' -%}
To install dependencies, your can run the following command:

```sh
python -m pip install .
```

When the dependencies are installed, you may launch the crawler with:

```sh
python -m {{cookiecutter.__package_name}}
```
Expand Down
2 changes: 1 addition & 1 deletion uv.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading