feat: Replace `pip-compile` with `uv` #4460

pyranota · 2024-09-30T13:47:28Z

Makes more idiomatic and is much faster on cold starts.
Useful if starting fresh instance or/and adding new dependency to python

Important

Replace pip-compile with uv for Python dependency management, update Docker and Nix configurations, and refactor TypeScript and Python annotations.

Dependency Management:
- Replace pip-compile with uv in python_executor.rs and ansible_executor.rs for Python dependency resolution.
- Add get_annotation_python() in worker.rs to handle no_uv annotation for Python scripts.
- Update uv_pip_compile() to handle no_uv flag and fallback to pip-compile if necessary.
Docker and Environment:
- Install uv in Dockerfile, DockerfileSlim, and DockerfileSlimEe.
- Add uv to shell.nix for Nix environment setup.
Code Refactoring:
- Rename get_annotation() to get_annotation_ts() in worker.rs and update references in bun_executor.rs, worker_lockfiles.rs, and scripts.rs.
- Update handle_python_deps() and handle_ansible_python_deps() to use uv_pip_compile().
Miscellaneous:
- Update RUST_LOG to "debug" in shell.nix for development purposes.

^{This description was created by}^{for 621e5a2. It will automatically update as commits are pushed.}

- Replace pip-compile with uv packages - Pin rust version - Add var to trigger windmill print more info in stdout

Some of the flags are included by default in UV and can be safely removed: - --resolver=backtracking - --no-emit-index-url Also uv does not support `--pip-args` and suggests to directly pass args to uv.

cloudflare-workers-and-pages · 2024-09-30T13:50:33Z

Deploying windmill with Cloudflare Pages

Latest commit:	`8e48066`
Status:	⚡️ Build in progress...

View logs

pyranota · 2024-09-30T13:51:11Z

Currently testing change. Small note:
uv handles extra index differently compared to pip-compile, package will be checked against extra index first and if not found fallback to main index, quite the opposite for pip-compile where main index will be checked first and then extra.

Edit: Resolved by using --index-strategy unsafe-best-match. Read more - https://docs.astral.sh/uv/pip/compatibility/#packages-that-exist-on-multiple-indexes

…ation_ts`

Put `# no_uv` on top the file for specific python script Or set `USE_PIP_COMPILE` variable to `true`

Add #no_uv to the end of requirements (requirements.in) That way if something breaks for customer, then they put #no_uv and new lockfile will be resolved

ellipsis-dev

👍 Looks good to me! Reviewed everything up to 80fbdd2 in 34 seconds

More details

Looked at 588 lines of code in 11 files
Skipped 0 files when reviewing.
Skipped posting 3 drafted comments based on config settings.

1. docker/DockerfileSlim:18

Draft comment:
pip-tools is still being installed, which is unnecessary if uv is replacing pip-compile. Consider removing this line.
Reason this comment was not posted:
Comment was on unchanged code.

2. docker/DockerfileSlimEe:18

Draft comment:
pip-tools is still being installed, which is unnecessary if uv is replacing pip-compile. Consider removing this line.
Reason this comment was not posted:
Marked as duplicate.

3. shell.nix:31

Draft comment:
pip-tools is still being included, which is unnecessary if uv is replacing pip-compile. Consider removing this line.
Reason this comment was not posted:
Marked as duplicate.

Workflow ID: wflow_kUt9IB3lRJVitQsx

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

…thon-uv

error[E0658]: attributes on expressions are experimental --> windmill-worker/src/python_executor.rs:144:5 | 144 | #[cfg(feature = "enterprise")] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | = note: see issue #15701 <rust-lang/rust#15701> for more information

ellipsis-dev

👍 Looks good to me! Incremental review on 175528e in 22 seconds

More details

Looked at 15 lines of code in 1 files
Skipped 0 files when reviewing.
Skipped posting 1 drafted comments based on config settings.

1. backend/windmill-worker/src/python_executor.rs:146

Draft comment:
Using unwrap here can cause a panic if the expected capture group is not present. Consider handling this case more gracefully to avoid potential runtime errors.
Reason this comment was not posted:
Comment was on unchanged code.

Workflow ID: wflow_Ac1Y9G8sFlzsPKkk

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

Will force recalculation of lockfile And block uv from using cached values

ellipsis-dev

👍 Looks good to me! Incremental review on 4f06bfe in 31 seconds

More details

Looked at 15 lines of code in 1 files
Skipped 0 files when reviewing.
Skipped posting 1 drafted comments based on config settings.

1. backend/windmill-worker/src/python_executor.rs:261

Draft comment:
The added arguments --python-preference only-system and --no-python-downloads ensure that uv does not manage Python installations, which aligns with the PR intent.
Reason this comment was not posted:
Confidence changes required: 0%
The added arguments to the uv command in the uv_pip_compile function are intended to prevent uv from managing Python installations. This is consistent with the PR description and comments.

Workflow ID: wflow_aaeBOQd0dDUKyrRH

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

ellipsis-dev

👍 Looks good to me! Incremental review on abee3d7 in 17 seconds

More details

Looked at 14 lines of code in 1 files
Skipped 0 files when reviewing.
Skipped posting 1 drafted comments based on config settings.

1. .github/DockerfileBackendTests:47

Draft comment:
Consider verifying the integrity of the uv installation script before executing it to prevent potential security risks.
Reason this comment was not posted:
Comment did not seem useful.

Workflow ID: wflow_1pQj35qzDhf2xDeZ

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

pyranota · 2024-10-02T18:51:24Z

@rubenfiszel Can't pass checks because environment of tests is managed by repository and I cant alter it to include uv

Instead postfix hash for requirements.in with `-no_uv`

ellipsis-dev

👍 Looks good to me! Incremental review on 04aac60 in 43 seconds

More details

Looked at 41 lines of code in 1 files
Skipped 0 files when reviewing.
Skipped posting 0 drafted comments based on config settings.

Workflow ID: wflow_NlqBFn1F4ei1slY7

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

ellipsis-dev

👍 Looks good to me! Incremental review on 621e5a2 in 19 seconds

More details

Looked at 12 lines of code in 1 files
Skipped 0 files when reviewing.
Skipped posting 1 drafted comments based on config settings.

1. backend/windmill-worker/src/python_executor.rs:152

Draft comment:
Redundant log message. Consider removing this line as the fallback to pip-compile is already logged on line 180.
Reason this comment was not posted:
Confidence changes required: 50%
The comment on line 152 is redundant since the same information is already logged on line 180. It's better to keep the logging consistent and avoid redundancy.

Workflow ID: wflow_3aCdEJ1ooo8gn7GV

You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

pyranota added 4 commits September 30, 2024 12:01

Update shell.nix

b652c37

- Replace pip-compile with uv packages - Pin rust version - Add var to trigger windmill print more info in stdout

Replace pip-compile with uv (dirty + untested)

cc1b891

Fix arguments passed to uv

a6cfbdb

Some of the flags are included by default in UV and can be safely removed: - --resolver=backtracking - --no-emit-index-url Also uv does not support `--pip-args` and suggests to directly pass args to uv.

Remove extra dbg!

25e1ea4

pyranota and others added 9 commits September 30, 2024 15:53

Replace 'pip-compile' with 'uv' in Dockerfile

4429338

Add fallback option to pip-compile (Disabled)

8d5ac55

Add uv to docker/DockerfileSlim*

844db2e

Add get_annotation_python and rename get_annotation to `get_annot…

b3d1328

…ation_ts`

Add option to fallback to pip-compile

e440fb7

Put `# no_uv` on top the file for specific python script Or set `USE_PIP_COMPILE` variable to `true`

Put back pip-tools into shell.nix

e1aceac

Make sure lockfile resolves again if #no_uv used

6bfa8dd

Add #no_uv to the end of requirements (requirements.in) That way if something breaks for customer, then they put #no_uv and new lockfile will be resolved

Put pip install pip-tools in original spot

0cae22c

Merge branch 'main' into python-uv

80fbdd2

pyranota marked this pull request as ready for review October 2, 2024 10:44

pyranota requested a review from rubenfiszel as a code owner October 2, 2024 10:44

ellipsis-dev bot reviewed Oct 2, 2024

View reviewed changes

pyranota added 3 commits October 2, 2024 10:52

Fix compilation error

4f40e46

Merge branch 'python-uv' of github.com:windmill-labs/windmill into py…

ba11eed

…thon-uv

Fix EE compilation error

175528e

error[E0658]: attributes on expressions are experimental --> windmill-worker/src/python_executor.rs:144:5 | 144 | #[cfg(feature = "enterprise")] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | = note: see issue #15701 <rust-lang/rust#15701> for more information

ellipsis-dev bot reviewed Oct 2, 2024

View reviewed changes

pyranota added 3 commits October 2, 2024 15:21

Add no_cache annotation

3065288

Will force recalculation of lockfile And block uv from using cached values

Target uv cache to /tmp/windmill/cache

4e7e060

Prohibit uv from managing python

4f06bfe

ellipsis-dev bot reviewed Oct 2, 2024

View reviewed changes

Add uv to DockerfileBackendTests

abee3d7

ellipsis-dev bot reviewed Oct 2, 2024

View reviewed changes

pyranota mentioned this pull request Oct 2, 2024

feature: Python to use uv to resolve and cache dependencies for python runs #3392

Open

pyranota added 2 commits October 3, 2024 12:35

Pin uv version to 0.4.18 in Dockerfiles

49821d3

Dont put #no_uv in requirements.in

04aac60

Instead postfix hash for requirements.in with `-no_uv`

ellipsis-dev bot reviewed Oct 3, 2024

View reviewed changes

rubenfiszel and others added 2 commits October 3, 2024 15:23

Merge branch 'main' into python-uv

5424a4b

Push Warning to logs if fallbacked to pip-compile

621e5a2

ellipsis-dev bot reviewed Oct 3, 2024

View reviewed changes

Merge branch 'main' into python-uv

8e48066

rubenfiszel merged commit b54c9ee into main Oct 3, 2024
5 of 6 checks passed

rubenfiszel deleted the python-uv branch October 3, 2024 14:00

github-actions bot locked and limited conversation to collaborators Oct 3, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Replace `pip-compile` with `uv` #4460

feat: Replace `pip-compile` with `uv` #4460

pyranota commented Sep 30, 2024 •

edited by ellipsis-dev bot

Loading

cloudflare-workers-and-pages bot commented Sep 30, 2024 •

edited

Loading

pyranota commented Sep 30, 2024 •

edited

Loading

ellipsis-dev bot left a comment

ellipsis-dev bot left a comment

ellipsis-dev bot left a comment

ellipsis-dev bot left a comment

pyranota commented Oct 2, 2024 •

edited

Loading

ellipsis-dev bot left a comment

ellipsis-dev bot left a comment

feat: Replace pip-compile with uv #4460

feat: Replace pip-compile with uv #4460

Conversation

pyranota commented Sep 30, 2024 • edited by ellipsis-dev bot Loading

cloudflare-workers-and-pages bot commented Sep 30, 2024 • edited Loading

Deploying windmill with Cloudflare Pages

pyranota commented Sep 30, 2024 • edited Loading

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

pyranota commented Oct 2, 2024 • edited Loading

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

ellipsis-dev bot left a comment

Choose a reason for hiding this comment

feat: Replace `pip-compile` with `uv` #4460

feat: Replace `pip-compile` with `uv` #4460

pyranota commented Sep 30, 2024 •

edited by ellipsis-dev bot

Loading

cloudflare-workers-and-pages bot commented Sep 30, 2024 •

edited

Loading

pyranota commented Sep 30, 2024 •

edited

Loading

pyranota commented Oct 2, 2024 •

edited

Loading