Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Avoid deadlocks when multiple uv processes lock resources #6790

Merged
merged 1 commit into from
Aug 29, 2024
Merged

Avoid deadlocks when multiple uv processes lock resources #6790

merged 1 commit into from
Aug 29, 2024

Conversation

zanieb
Copy link
Member

@zanieb zanieb commented Aug 29, 2024 •
edited
Loading

This is achieved by updating the LockedFile::acquire API to be async — as in some cases we were attempting to acquire the lock synchronously, i.e., without yielding, which blocked the runtime.

Closes #6691 — I tested with the reproduction there and a local release build and no longer reproduce the deadlock with these changes.

Some additional context in the internal Discord thread

@zanieb
Avoid deadlocks when multiple uv processes lock resources
ba7a1f0 
This is achieved by updating the `LockedFile::acquire` API to be async — as in some cases we were attempting to acquire the lock synchronously, i.e., without yielding, which blocked the runtime
@zanieb zanieb added the bug Something isn't working label Aug 29, 2024
zanieb
zanieb commented Aug 29, 2024
View reviewed changes
crates/uv-fs/src/lib.rs
Comment on lines +345 to +355
/// The same as [`LockedFile::acquire`], but for synchronous contexts. Do not use from an async
/// context, as this can block the runtime while waiting for another process to release the
/// lock.
pub fn acquire_blocking(
path: impl AsRef<Path>,
resource: impl Display,
) -> Result<Self, std::io::Error> {
let file = fs_err::File::create(path.as_ref())?;
let resource = resource.to_string();
Self::lock_file_blocking(file, &resource)
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could remove this entirely too.

zanieb
zanieb commented Aug 29, 2024
View reviewed changes
crates/uv-fs/src/lib.rs
Comment on lines +363 to +364
let file = fs_err::File::create(path.as_ref())?;
let resource = resource.to_string();
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to do this for spawn_blocking. Are there better ways?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it's ok, I believe it needs to have a static lifetime (or be owned) for this...

@zanieb zanieb marked this pull request as ready for review August 29, 2024 04:23
@zanieb zanieb mentioned this pull request Aug 29, 2024
Draft
@zanieb zanieb merged commit e3d5d3d into main Aug 29, 2024
57 checks passed
@zanieb zanieb deleted the zb/lock-file-async branch August 29, 2024 16:16
@BrewTestBot BrewTestBot mentioned this pull request Aug 30, 2024
Merged
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Sep 4, 2024
@tmeijn
build(deps): update astral-sh/uv to v0.4.4
59d659b 
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.4.0` -> `0.4.4` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.4.4`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#044)

[Compare Source](astral-sh/uv@0.4.3...0.4.4)

##### Enhancements

-   Allow customizing the project environment path with `UV_PROJECT_ENVIRONMENT` ([#&#8203;6834](astral-sh/uv#6834))
-   Warn when `VIRTUAL_ENV` is set but will not be respected in project commands ([#&#8203;6864](astral-sh/uv#6864))
-   Add `--no-hashes` to `uv export` ([#&#8203;6954](astral-sh/uv#6954))
-   Make HTTP headers title case for backward compatibility ([#&#8203;6887](astral-sh/uv#6887))
-   Pin `.python-version` in `uv init` ([#&#8203;6869](astral-sh/uv#6869))
-   Support `file://` URLs for `UV_PYTHON_INSTALL_MIRROR` ([#&#8203;6950](astral-sh/uv#6950))
-   Introduce more docker tags for uv ([#&#8203;6053](astral-sh/uv#6053))

##### Bug fixes

-   Avoid canonicalizing the cache directory ([#&#8203;6949](astral-sh/uv#6949))
-   Show all PyPy versions in `uv python list --all-versions` ([#&#8203;6917](astral-sh/uv#6917))
-   Avoid incorrect `requires-python` marker simplifications ([#&#8203;6268](astral-sh/uv#6268))

##### Documentation

-   Add documentation for `UV_PROJECT_ENVIRONMENT` ([#&#8203;6987](astral-sh/uv#6987))
-   Add optional dependencies section to the lockfile document ([#&#8203;6982](astral-sh/uv#6982))
-   Document use of the `file://` scheme in Python installation mirrors ([#&#8203;6984](astral-sh/uv#6984))
-   Fix outdated references to the help menu documentation in the first steps page ([#&#8203;6980](astral-sh/uv#6980))
-   Show env option in CLI reference documentation ([#&#8203;6863](astral-sh/uv#6863))
-   Add bind mount example to `docker.md` ([#&#8203;6921](astral-sh/uv#6921))

### [`v0.4.3`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#043)

[Compare Source](astral-sh/uv@0.4.2...0.4.3)

##### Enhancements

-   Show build backend output when `--verbose` is provided ([#&#8203;6903](astral-sh/uv#6903))
-   Allow `uv sync --frozen --package` without copying member `pyproject.toml` ([#&#8203;6943](astral-sh/uv#6943))

##### Bug fixes

-   Avoid panic with missing temporary directory ([#&#8203;6929](astral-sh/uv#6929))
-   Avoid updating incorrect dependencies for sorted `uv add` ([#&#8203;6939](astral-sh/uv#6939))
-   Use lower-bound semantics for all Python compatibility comparisons ([#&#8203;6882](astral-sh/uv#6882))

### [`v0.4.2`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#042)

[Compare Source](astral-sh/uv@0.4.1...0.4.2)

##### Enhancements

-   Adding support for `.pyc`  files in `uv run` ([#&#8203;6886](astral-sh/uv#6886))
-   Treat missing `top_level.txt` as non-fatal ([#&#8203;6881](astral-sh/uv#6881))

##### Bug fixes

-   Fix `is_disjoint` check for supported environments ([#&#8203;6902](astral-sh/uv#6902))
-   Remove dangling archives in `uv cache clean ${package}` ([#&#8203;6915](astral-sh/uv#6915))
-   Error when discovered Python is incompatible with `--isolated` workspace ([#&#8203;6885](astral-sh/uv#6885))
-   Warn when discovered Python is incompatible with PEP 723 script ([#&#8203;6884](astral-sh/uv#6884))

### [`v0.4.1`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#041)

[Compare Source](astral-sh/uv@0.4.0...0.4.1)

##### Enhancements

-   Add `uv export --format requirements-txt` ([#&#8203;6778](astral-sh/uv#6778))
-   Allow `@` references in `uv tool install --from` ([#&#8203;6842](astral-sh/uv#6842))
-   Normalize version specifiers by sorting ([#&#8203;6333](astral-sh/uv#6333))
-   Respect the user's upper-bound in `requires-python` ([#&#8203;6824](astral-sh/uv#6824))
-   Use Windows registry to discover Python on Windows directly ([#&#8203;6761](astral-sh/uv#6761))
-   Hint at `--no-workspace` in `uv init` failures ([#&#8203;6815](astral-sh/uv#6815))
-   Update to last PyPy releases ([#&#8203;6784](astral-sh/uv#6784))

##### Bug fixes

-   Avoid deadlocks when multiple uv processes lock resources ([#&#8203;6790](astral-sh/uv#6790))
-   Expand tildes when matching against `PATH` ([#&#8203;6829](astral-sh/uv#6829))
-   Fix `uv init --no-project` alias ([#&#8203;6837](astral-sh/uv#6837))
-   Ignore pre-release segments when discovering via `requires-python` ([#&#8203;6813](astral-sh/uv#6813))
-   Support inline optional tables in `uv add` and `uv remove` ([#&#8203;6787](astral-sh/uv#6787))
-   Update default `hello.py` to pass `ruff format` ([#&#8203;6811](astral-sh/uv#6811))
-   Avoid stripping root for user path display ([#&#8203;6865](astral-sh/uv#6865))
-   Error when user-provided environments are disjoint with Python ([#&#8203;6841](astral-sh/uv#6841))
-   Retain alphabetical sorting for `pyproject.toml` in `uv add` operations ([#&#8203;6388](astral-sh/uv#6388))))

##### Documentation

-   Add a link to the multiple index docs in the alternative index guide ([#&#8203;6826](astral-sh/uv#6826))
-   Add docs for inline exclude newer in PEP 723 scripts ([#&#8203;6831](astral-sh/uv#6831))
-   Enumerate available Docker tags ([#&#8203;6768](astral-sh/uv#6768))
-   Omit `[pip]` section from configuration file docs ([#&#8203;6814](astral-sh/uv#6814))
-   Update `project.urls` in `pyproject.toml`  ([#&#8203;6844](astral-sh/uv#6844))
-   Add docs for AWS CodeArtifact usage ([#&#8203;6816](astral-sh/uv#6816))

##### Other changes

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@konstin konstin konstin approved these changes

@charliermarsh charliermarsh charliermarsh approved these changes

Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Deadlock in uv pip install with git links
3 participants
@zanieb @charliermarsh @konstin