Skip to content

Latest commit

 

History

History
165 lines (103 loc) · 4.13 KB

CONTRIBUTING.md

File metadata and controls

165 lines (103 loc) · 4.13 KB
Raw

Development

Here you'll find a contributing guide to get started with development.

Environment

For local development, it is required to have Python 3.9 (or a later version) installed.

We use Poetry for project management. Install it and set up your IDE accordingly.

Dependencies

To install this package and its development dependencies, run:

make install-dev

Code checking

To execute all code checking tools together, run:

make check-code

Linting

We utilize ruff for linting, which analyzes code for potential issues and enforces consistent style. Refer to pyproject.toml for configuration details.

To run linting:

make lint

Formatting

Our automated code formatting also leverages ruff, ensuring uniform style and addressing fixable linting issues. Configuration specifics are outlined in pyproject.toml.

To run formatting:

make format

Type checking

Type checking is handled by mypy, verifying code against type annotations. Configuration settings can be found in pyproject.toml.

To run type checking:

make type-check

Unit tests

We employ pytest as our testing framework, equipped with various plugins. Check pyproject.toml for configuration details and installed plugins.

We use pytest as a testing framework with many plugins. Check pyproject.toml for configuration details and installed plugins.

To run unit tests:

make unit-tests

To run unit tests with HTML coverage report:

make unit-tests-cov

Documentation

We adhere to the Google docstring format for documenting our codebase. Every user-facing class or method is documented. Documentation standards are enforced using Ruff.

Our API documentation is generated from these docstrings using pydoc-markdown with additional post-processing. Markdown files in the docs/ folder complement the autogenerated content. Final documentation is rendered using Docusaurus and published to GitHub Pages.

To run the documentation locally, you need to have Node.js version 20 or higher installed. Once you have the correct version of Node.js, follow these steps:

Navigate to the website/ directory:

cd website/

Enable Corepack, which installs Yarn automatically:

corepack enable

Build the API reference:

./build_api_reference.sh

Install the necessary dependencies:

yarn

Start the project in development mode with Hot Module Replacement (HMR):

yarn start

Or using make:

make run-docs

Release process

Publishing new versions to PyPI is automated through GitHub Actions.

  • Beta releases: On each commit to the master branch, a new beta release is automatically published. The version number is determined based on the latest release and conventional commits. The beta version suffix is incremented by 1 from the last beta release on PyPI.
  • Stable releases: A stable version release may be created by triggering the run_release GitHub Actions workflow. The version number is determined based on the latest release and conventional commits (auto release type), or it may be overriden using the custom release type.

Publishing to PyPI manually

  1. Do not do this unless absolutely necessary. In all conceivable scenarios, you should use the run_release workflow instead.

  2. Make sure you know what you're doing.

  3. Update the version number:

  • Modify the version field under tool.poetry in pyproject.toml.
[tool.poetry]
name = "crawlee"
version = "x.z.y"
  1. Generate the distribution archives for the package:
poetry build
  1. Set up the PyPI API token for authentication:
poetry config pypi-token.pypi YOUR_API_TOKEN
  1. Upload the package to PyPI:
poetry publish