Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use uv for dev and CI environments #1625

Merged
Merged
Show file tree
Hide file tree
Changes from 15 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 10 additions & 8 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,29 +42,31 @@ jobs:
if [ ! -f "/etc/lsb-release" ] ; then
echo "DISTRIB_RELEASE=18.04" > /etc/lsb-release
fi
- name: Set up Python
- name: Set Up Python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
cache: pip
cache-dependency-path: |
pyproject.toml
- name: Install Dependencies and lightly
run: pip install -e '.[all]'
- name: Set Up Environment
run: |
make install-uv reset-venv
source .venv/bin/activate
make install-fixed-extras
- name: Clean Previous Coverage Data
run: rm -f .coverage
- name: Run Pytest (Python 3.7)
if: matrix.python == '3.7'
run: |
source .venv/bin/activate
export LIGHTLY_SERVER_LOCATION="localhost:-1"
python -m pytest -s -v --runslow --ignore=./lightly/openapi_generated/
- name: Run Pytest with Coverage (Python 3.10)
if: matrix.python == '3.10'
run: |
source .venv/bin/activate
export LIGHTLY_SERVER_LOCATION="localhost:-1"
pip install pytest-cov==5.0.0
uv pip install pytest-cov==5.0.0
python -m pytest -s -v --runslow --cov=./lightly --cov-report=xml --ignore=./lightly/openapi_generated/
- name: Upload coverage to Codecov
- name: Upload Coverage to Codecov
if: matrix.python == '3.10'
uses: codecov/codecov-action@v4
with:
Expand Down
7 changes: 5 additions & 2 deletions .github/workflows/test_api_deps_only.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,18 @@ jobs:
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Set up Python
- name: Set Up Python
uses: actions/setup-python@v5
with:
python-version: "3.10"
- name: Install API Dependencies
- name: Set Up Environment
run: |
make install-uv reset-venv
source .venv/bin/activate
make install-api-only
- name: Test API Client
run: |
source .venv/bin/activate
# Test if ApiWorkflowClient can be created.
export LIGHTLY_SERVER_LOCATION="localhost:-1"
python -c "from lightly.api import ApiWorkflowClient; ApiWorkflowClient(token='abc')"
14 changes: 8 additions & 6 deletions .github/workflows/test_code_format.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,18 +19,20 @@ jobs:
if [ ! -f "/etc/lsb-release" ] ; then
echo "DISTRIB_RELEASE=18.04" > /etc/lsb-release
fi
- name: Set up Python
- name: Set Up Python
uses: actions/setup-python@v5
with:
python-version: "3.7"
cache: pip
cache-dependency-path: |
pyproject.toml
- name: Install Dependencies and lightly
run: pip install -e '.[all]'
- name: Set Up Environment
run: |
make install-uv reset-venv
source .venv/bin/activate
make install-fixed-extras
- name: Run Format Check
run: |
source .venv/bin/activate
make format-check
- name: Run Type Check
run: |
source .venv/bin/activate
make type-check
18 changes: 9 additions & 9 deletions .github/workflows/test_minimal_deps.yml
Original file line number Diff line number Diff line change
Expand Up @@ -33,23 +33,23 @@ jobs:
strategy:
matrix:
python: ["3.7"]

dependencies: ["minimal", "minimal-extras"]
steps:
- uses: actions/checkout@v4
- name: Set up Python
- name: Set Up Python
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
cache: pip
cache-dependency-path: |
pyproject.toml
- name: Install Minimal Dependencies
- name: Set Up Environment
run: |
# Install libavformat-dev and libavdevice-dev for PyAV as versions before 9.0
# do not provide wheels including FFMPEG.
sudo apt-get install libavformat-dev libavdevice-dev
pip install -e ."[minimal]"
- name: Install Package Without Dependencies
run: pip install --no-deps .
make install-uv reset-venv
source .venv/bin/activate
make install-${{ matrix.dependencies }}
- name: Run Tests
run: |
source .venv/bin/activate
export LIGHTLY_SERVER_LOCATION="localhost:-1"
python -m pytest -s -v --runslow
18 changes: 10 additions & 8 deletions .github/workflows/test_setup.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,26 +37,28 @@ jobs:
if [ ! -f "/etc/lsb-release" ] ; then
echo "DISTRIB_RELEASE=18.04" > /etc/lsb-release
fi
- name: Set up Python 3.7
- name: Set Up Python
uses: actions/setup-python@v5
with:
python-version: 3.7
cache: pip
cache-dependency-path: |
pyproject.toml
- name: Install Dependencies and lightly
run: pip install .
- name: basic tests of CLI
- name: Set Up Environment
run: |
make install-uv reset-venv
source .venv/bin/activate
make install-fixed
- name: Test CLI
run: |
source .venv/bin/activate
LIGHTLY_SERVER_LOCATION="localhost:-1"
lightly-crop --help
lightly-train --help
lightly-embed --help
lightly-magic --help
lightly-download --help
lightly-version
- name: test of CLI on a real dataset
- name: Test CLI on Dataset
run: |
source .venv/bin/activate
LIGHTLY_SERVER_LOCATION="localhost:-1"
git clone https://github.com/alexeygrigorev/clothing-dataset-small clothing_dataset_small
INPUT_DIR_1="clothing_dataset_small/test/dress"
Expand Down
12 changes: 8 additions & 4 deletions .github/workflows/tests_unmocked.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,20 @@ jobs:
steps:
- name: Checkout Code
uses: actions/checkout@v4
- name: Set up Python 3.7
- name: Set Up Python
uses: actions/setup-python@v5
with:
python-version: 3.7
cache: pip
cache-dependency-path: |
pyproject.toml
- name: Install Dependencies and lightly
run: pip install .
- name: run unmocked tests
- name: Set Up Environment
run: |
make install-uv reset-venv
source .venv/bin/activate
make install-fixed
- name: Run Unmocked Tests
run: |
source .venv/bin/activate
export LIGHTLY_SERVER_LOCATION=${{ secrets.LIGHTLY_SERVER_LOCATION }}
bash tests/UNMOCKED_end2end_tests/run_all_unmocked_tests.sh ${{ secrets.DUMMY_USER_TOKEN_STAGING }}
7 changes: 7 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,13 @@ Follow these steps to start contributing:
$ pip install -e ".[dev]"
```

If you are using [uv](https://github.com/astral-sh/uv) instead of pip, you can use
the following command:

```bash
$ make install-dev
```

5. **(Optional)** Install pre-commit hooks:

```bash
Expand Down
104 changes: 96 additions & 8 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -78,15 +78,7 @@ dist: clean
python -m build
ls -l dist

## install the package to active site
install: clean
pip install .

# Should be same command as in the docs:
# https://docs.lightly.ai/docs/install-lightly#install-the-lightly-python-client
install-api-only: clean
pip install -r requirements/base.txt
pip install . --no-deps

# uninstall package from active site
uninstall: clean
Expand All @@ -100,3 +92,99 @@ tox:
find:
@read -p "Enter Term: " term; \
grep -rnw ./ -e "$$term"


### Virtual Environment

.PHONY: install-uv
install-uv:
curl -LsSf https://astral.sh/uv/0.2.25/install.sh | sh


.PHONY: reset-venv
reset-venv:
deactivate || true
rm -rf .venv
uv venv .venv


### Dependencies

# When running these commands locally, it is recommended to first reset the environment
# with: `make reset-venv && source .venv/bin/activate`
# Otherwise old dependencies might linger around.

# Set EDITABLE to -e to install the package in editable mode outside of CI. This is
# useful for local development.
ifdef CI
EDITABLE=
else
EDITABLE=-e
endif

# Date until which dependencies installed with --exclude-newer must have been released.
# Dependencies released after this date are ignored.
EXCLUDE_NEWER_DATE="2024-08-07"

# Install package for local development.
.PHONY: install-dev
install-dev:
uv pip install ${EDITABLE} . --all-extras --requirement pyproject.toml


# Install package with API dependencies only.
# Should be same command as in the docs with some extra flags for CI:
# https://docs.lightly.ai/docs/install-lightly#install-the-lightly-python-client
.PHONY: install-api-only
install-api-only:
uv pip install --exclude-newer ${EXCLUDE_NEWER_DATE} --requirement requirements/base.txt
uv pip install --exclude-newer ${EXCLUDE_NEWER_DATE} . --no-deps

# Install package with minimal dependencies.
#
# This command is split into multiple steps:
# 1. Install the dev dependencies to be able to run tests. We don't want to use
# the minimal versions for these dependencies.
# 2. Then we reinstall the package with minimal dependencies.
# 3. Finally we install setuptools<50. This is necessary for compatibility with old
# PyTorch Lightning versions that do not include the correct setuptools dependencies.
#
# Explanation of flags:
# --exclude-newer: We don't want to install dependencies released after that date to
# keep CI stable.
# --resolution=lowest-direct: Only install minimal versions for direct dependencies.
# Transitive dependencies will use the latest compatible version.
# Using --resolution=lowest would also download the latest versions for transitive
# dependencies which is not a realistic scenario and results in some extremely old
# dependencies being installed.
# --reinstall: Reinstall dependencies to make sure they satisfy the constraints.
.PHONY: install-minimal
install-minimal:
uv pip install --exclude-newer ${EXCLUDE_NEWER_DATE} ${EDITABLE} ".[dev]"
uv pip install --resolution=lowest-direct --exclude-newer ${EXCLUDE_NEWER_DATE} --reinstall ${EDITABLE} ".[minimal]"
uv pip install --exclude-newer ${EXCLUDE_NEWER_DATE} --reinstall "setuptools<50"

# Install package with minimal dependencies including extras.
# See install-minimal for explanation of flags.
# We do not use --all-extras because it includes the dev dependencies for which we don't
# want to install the minimal versions.
.PHONY: install-minimal-extras
install-minimal-extras:
uv pip install --exclude-newer ${EXCLUDE_NEWER_DATE} ${EDITABLE} ".[dev]"
uv pip install --resolution=lowest-direct --exclude-newer ${EXCLUDE_NEWER_DATE} --reinstall ${EDITABLE} ".[matplotlib,minimal,timm,video]" --requirement pyproject.toml
uv pip install --exclude-newer ${EXCLUDE_NEWER_DATE} --reinstall "setuptools<50"

# Install package with fixed dependencies.
.PHONY: install-fixed
install-fixed:
uv pip install --exclude-newer ${EXCLUDE_NEWER_DATE} --reinstall ${EDITABLE} . --requirement pyproject.toml

# Install package with fixed dependencies and extras.
.PHONY: install-fixed-extras
install-fixed-extras:
uv pip install --exclude-newer ${EXCLUDE_NEWER_DATE} --reinstall ${EDITABLE} . --all-extras --requirement pyproject.toml

# Install package with the latest dependencies.
.PHONY: install-latest
install-latest:
uv pip install --upgrade --reinstall ${EDITABLE} . --all-extras --requirement pyproject.toml
8 changes: 6 additions & 2 deletions lightly/api/api_workflow_selection.py
Original file line number Diff line number Diff line change
@@ -1,9 +1,10 @@
from __future__ import annotations

import time
import warnings
from typing import Dict, List, Optional, Union
from typing import TYPE_CHECKING, Dict, List, Optional, Union

import numpy as np
from numpy.typing import NDArray

from lightly.active_learning.config.selection_config import SelectionConfig
from lightly.openapi_generated.swagger_client.models import (
Expand All @@ -16,6 +17,9 @@
TagData,
)

if TYPE_CHECKING:
from numpy.typing import NDArray

Check warning on line 21 in lightly/api/api_workflow_selection.py

View check run for this annotation

Codecov / codecov/patch

lightly/api/api_workflow_selection.py#L21

Added line #L21 was not covered by tests


def _parse_active_learning_scores(scores: Union[np.ndarray, List]):
"""Makes list/np.array of active learning scores serializable."""
Expand Down
2 changes: 1 addition & 1 deletion lightly/embedding/_base.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@
import omegaconf
from omegaconf import DictConfig
from pytorch_lightning import LightningModule, Trainer
from pytorch_lightning.callbacks.callback import Callback
from pytorch_lightning.callbacks import Callback
from torch import Tensor
from torch.nn import Module
from torch.optim import Optimizer
Expand Down
8 changes: 6 additions & 2 deletions lightly/embedding/embedding.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,13 @@
# Copyright (c) 2020. Lightly AG and its affiliates.
# All Rights Reserved

from __future__ import annotations

import time
from typing import List, Optional, Tuple
from typing import TYPE_CHECKING, List, Optional, Tuple

import numpy as np
import torch
from numpy.typing import NDArray
from torch.nn import Module
from torch.optim import Optimizer
from torch.optim.lr_scheduler import _LRScheduler
Expand All @@ -20,6 +21,9 @@
from lightly.utils.benchmarking import BenchmarkModule
from lightly.utils.reordering import sort_items_by_keys

if TYPE_CHECKING:
from numpy.typing import NDArray

Check warning on line 25 in lightly/embedding/embedding.py

View check run for this annotation

Codecov / codecov/patch

lightly/embedding/embedding.py#L25

Added line #L25 was not covered by tests


class SelfSupervisedEmbedding(BaseEmbedding):
"""Implementation of self-supervised embedding models.
Expand Down
Loading