Skip to content

Commit

Permalink
Add [python-repos].find_links as preferred alias for `[python-repos…
Browse files Browse the repository at this point in the history
…].repos` (#16582)

`find_links` matches the Python ecosystem with pip and Pex. This would have been a better name from the start. 

At the same time, we are trying to give stronger backward compatibility for users: #16547. It's relatively cheap for us to support both option names ~indefinitely.

Note that #15627 took away the ability to give multiple flag names for the same option, which we could have used here. I debated reviving that, but I think this is preferable because it makes clear which option is preferred. Users will get a deprecation message when using `[python-repos].repos`, although they can silence it with `--ignore-warnings`.

[ci skip-rust]
  • Loading branch information
Eric-Arellano authored Aug 19, 2022
1 parent 0e233d3 commit 79d35f7
Show file tree
Hide file tree
Showing 6 changed files with 66 additions and 21 deletions.
25 changes: 17 additions & 8 deletions docs/markdown/Python/python/python-third-party-dependencies.md
Original file line number Diff line number Diff line change
Expand Up @@ -479,28 +479,37 @@ Django @ file:///Users/pantsbuild/prebuilt_wheels/django-3.1.1-py3-none-any.whl
There are two mechanisms for setting up custom Python distribution repositories:
#### Simple repositories as defined by PEP 503
#### PEP-503 compatible indexes
If your custom repo is of this type, i.e., "private PyPI", aka "cheese shop", use the option `indexes` in the `[python-repos]` scope.
Use `[python-repos].indexes` to add [PEP 503-compatible](https://peps.python.org/pep-0503/)
indexes, like PyPI.
```toml pants.toml
[python-repos]
indexes.add = ["https://custom-cheeseshop.net/simple"]
```
To exclusively use your custom index—i.e. to not use PyPI—use `indexes = [..]` instead of `indexes.add = [..]`.
To exclusively use your custom index, i.e. to not use the default of PyPI, use `indexes = [..]`
instead of `indexes.add = [..]`.

#### A Pip findlinks repository
#### pip `--find-links`

If your custom repo is of this type, use the option `repos` in the `[python-repos]` scope.
Use the option `[python-repos].find_links` for flat lists of packages. Same as pip's
[`--find-links`](https://pip.pypa.io/en/stable/cli/pip_wheel/?highlight=find%20links#cmdoption-f)
option, you can either use:

* a URL to an HTML file with links to wheel and/or sdist files, or
* a `file://` absolute path to an HTML file with links, or to a local directory with wheel and/or
sdist files.

```toml
[python-repos]
repos = ["https://your/repo/here"]
find_links = [
"https://your/repo/here",
"file:///Users/pantsbuild/prebuilt_wheels",
]
```

Indexes are assumed to have a nested structure (like <http://pypi.org/simple>), whereas repos are flat lists of packages.

#### Authenticating to custom repos

To authenticate to custom repos, you may need to provide credentials
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ python_tests(name="target_types_test", sources=["target_types_test.py"])
python_tests(
name="rules_test",
sources=["rules_test.py"],
timeout=240,
timeout=300,
# We want to make sure the default lockfile works for both macOS and Linux.
tags=["platform_specific_behavior"],
)
4 changes: 3 additions & 1 deletion src/python/pants/backend/python/goals/lockfile.py
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,9 @@ def warn_python_repos(option: str) -> None:
)
)

if python_repos.repos:
if python_repos._find_links:
warn_python_repos("find_links")
if python_repos._repos:
warn_python_repos("repos")
if python_repos.indexes != (python_repos.pypi_index,):
warn_python_repos("indexes")
Expand Down
2 changes: 1 addition & 1 deletion src/python/pants/backend/python/lint/isort/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ python_tests(
name="tests",
overrides={
"rules_integration_test.py": {
"timeout": 180,
"timeout": 240,
"tags": ["platform_specific_behavior"],
}
},
Expand Down
43 changes: 38 additions & 5 deletions src/python/pants/backend/python/subsystems/repos.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,9 @@

from __future__ import annotations

from typing import cast

from pants.base.deprecated import resolve_conflicting_options
from pants.option.option_types import StrListOption
from pants.option.subsystem import Subsystem
from pants.util.strutil import softwrap
Expand All @@ -14,29 +17,59 @@ class PythonRepos(Subsystem):
"""
External Python code repositories, such as PyPI.
These options may be used to point to custom cheeseshops when resolving requirements.
These options may be used to point to custom package indexes when resolving requirements.
"""
)

pypi_index = "https://pypi.org/simple/"

repos = StrListOption(
_find_links = StrListOption(
help=softwrap(
"""
URLs and/or file paths corresponding to pip's `--find-links` option.
Per [pip's documentation](https://pip.pypa.io/en/stable/cli/pip_wheel/?highlight=find%20links#cmdoption-f),
URLs should be to HTML files with links to `.whl` and/or
sdist files. Local paths must be absolute, and can either be to an HTML file with
links or to a directory with `.whl` and/or sdist files, e.g.
`file:///Users/pantsbuild/prebuilt_wheels`.
"""
)
)
_repos = StrListOption(
help=softwrap(
"""
URLs of code repositories to look for requirements. In Pip and Pex, this option
corresponds to the `--find-links` option.
"""
),
advanced=True,
removal_version="3.0.0.dev0",
removal_hint="A deprecated alias for `[python-repos].find_links`.",
)
indexes = StrListOption(
default=[pypi_index],
help=softwrap(
"""
URLs of code repository indexes to look for requirements. If set to an empty
list, then Pex will use no indices (meaning it will not use PyPI). The values
should be compliant with PEP 503.
URLs of [PEP-503 compatible](https://peps.python.org/pep-0503/) code repository
indexes to look for requirements.
If set to an empty list, then Pex will use no indexes (meaning it will not use PyPI).
"""
),
advanced=True,
)

@property
def find_links(self) -> tuple[str, ...]:
return cast(
"tuple[str, ...]",
resolve_conflicting_options(
old_option="repos",
new_option="find_links",
old_scope=self.options_scope,
new_scope=self.options_scope,
old_container=self.options,
new_container=self.options,
),
)
11 changes: 6 additions & 5 deletions src/python/pants/backend/python/util_rules/pex_requirements.py
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,7 @@ async def determine_resolve_pex_config(
if request.resolve_name is None:
return ResolvePexConfig(
indexes=python_repos.indexes,
find_links=python_repos.repos,
find_links=python_repos.find_links,
manylinux=python_setup.manylinux,
constraints_file=None,
no_binary=FrozenOrderedSet(),
Expand Down Expand Up @@ -413,7 +413,7 @@ async def determine_resolve_pex_config(

return ResolvePexConfig(
indexes=python_repos.indexes,
find_links=python_repos.repos,
find_links=python_repos.find_links,
manylinux=python_setup.manylinux,
constraints_file=constraints_file,
no_binary=FrozenOrderedSet(no_binary),
Expand Down Expand Up @@ -517,21 +517,22 @@ def _common_failure_reasons(
yield softwrap(
"""
- The `indexes` arguments have changed from when the lockfile was generated.
(Indexes are set via the option `[python-repos].indexes`
(Indexes are set via the option `[python-repos].indexes`)
"""
)
if InvalidPythonLockfileReason.FIND_LINKS_MISMATCH in failure_reasons:
yield softwrap(
"""
- The `find_links` arguments have changed from when the lockfile was generated.
(Find links is set via the option `[python-repos].repos`
(Find links is set via the option `[python-repos].find_links` or the deprecated
`[python-repos].repos`)
"""
)
if InvalidPythonLockfileReason.MANYLINUX_MISMATCH in failure_reasons:
yield softwrap(
"""
- The `manylinux` argument has changed from when the lockfile was generated.
(manylinux is set via the option `[python].resolver_manylinux`
(manylinux is set via the option `[python].resolver_manylinux`)
"""
)

Expand Down

1 comment on commit 79d35f7

@kaos
Copy link
Member

@kaos kaos commented on 79d35f7 Nov 28, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a niche case here, when using PANTS_SHA which uses the deprecated option: https://github.com/pantsbuild/setup/blob/ab8581bb52d92fcb3fda1b1dfb20204ae8888a90/pants#L483

which results in the surprising warning (for the user as there's no mention of python-repos in their configuration, and unaware of the above use):

╰─❯ PANTS_SHA=d47aecab6764c6c4ca3f80f3bc98d0985af399e7 ./pants ...
10:13:51.00 [WARN] DEPRECATED: option 'repos' in scope 'python-repos' is scheduled to be removed in version 3.0.0.dev0.

A deprecated alias for `[python-repos].find_links`.

(the user in this case being me, and I wasn't surprised for long as I quickly found the source of it..)

Just noting this for the record, as it would need to be addressed before considering actually removing the deprecated option.

Please sign in to comment.