Python generate-lockfiles not completing

[pranav-kunapuli]

Hey everyone, I'm moving my organization's python backends to a monorepo architecture and felt that Pants was the best build tool for our use case. I'm working through reorganizing our repository and setting up Pants, but I'm stuck at the generate-lockfiles step. The command seems to be going on forever with no end in sight. I ran the process overnight (8+ hours) and still no completion. I've generated BUILD files using pants tailor ::, but I can't proceed until my lockfiles are created. Any advice on how to proceed?

For additional context, our repository currently uses poetry for dependency management and I have the following pants.toml file:

[GLOBAL]
pants_version = "2.15.0"

# Packages needed for pants to run various goals
backend_packages = [
  "pants.backend.python",
  "pants.backend.python.lint.bandit",
  "pants.backend.python.lint.black",
  "pants.backend.python.lint.docformatter",
  "pants.backend.python.lint.flake8",
  "pants.backend.python.lint.isort",
  "pants.backend.python.typecheck.mypy",
]

[source]
# Directory structure rooted around a shared modules directory and various services
root_patterns = [
  "/shared",
  "/services",
  "/tests",
]

[python]
# The default interpreter constraints for code in this repo. Individual targets can override
#  this with the `interpreter_constraints` field. See
#  https://www.pantsbuild.org/docs/python-interpreter-compatibility.

# Modify this if you don't have Python 3.9 on your machine.
# This can be a range, such as [">=3.8,<3.11"], but it's usually recommended to restrict
# to a single minor version.
interpreter_constraints = [">=3.10.*"]

# Enable the "resolves" mechanism, which turns on lockfiles for user code. See
# https://www.pantsbuild.org/docs/python-third-party-dependencies. This also adds the
# `generate-lockfiles` goal for Pants to generate the lockfile for you.
enable_resolves = true

# [python.resolves]
# python-default = "python-default.lock"

[python-bootstrap]
# We search for interpreters both on the $PATH and in the `$(pyenv root)/versions` folder.
# If you're using macOS, you may want to leave off the <PATH> entry to avoid using the
# problematic system Pythons. See
# https://www.pantsbuild.org/docs/python-interpreter-compatibility#changing-the-interpreter-search-path.

# search_path = ["<PATH>", "<PYENV>"]
search_path = ["<PYENV>"]

[kaos]

the likely culprit is the open-ended IC interpreter_constraints = [">=3.10.*"]. Try narrowing it down to what you actually need.

[pranav-kunapuli]

Thanks @kaos and @benjyw, giving your suggestions a shot now!

[benjyw]

Hi! One thing that stands out to me is the interpreter constraints. interpreter_constraints = [">=3.10.*"] is open-ended, which means Pants is trying to generate a lockfile compatible with 3.10, 3.11 and the 3.12 alphas. I recommend locking down to ==3.10.* and seeing if that helps.

[jsirois]

Actually, the fixed resolve runs just as fast with no --pip-version. It's just that a broken resolve takes ~forever with old Pip; so you can run into this again using Pants 2.15.0 and the only recourse to figure out what's wrong will be to do what I did and copy out the Pex command line and try with --pip-version added; so buyer beware.

[pranav-kunapuli]

Finally got the lockfiles generated! 🎉 The issues I had to resolve were effectively the SQLAlchemy and alembic version pinning so all of the constraints could play nicely together.

However, I actually ran into a dumber problem which was that we had an outdated requirements.txt file lingering in the root of the repository which was known to be unused, but pants was picking it up and treating it like the source-of-truth of dependency versions. Removing this outdated requirements.txt file was the final step needed to get the lock files to generate.

All in all, I think this was less to do with Poetry and more to do with some stupidity on my part and the slowness of old pip.

[jsirois]

"Pants was picking it up" - is this because of a tailor-generated target or a human authored one?

[pranav-kunapuli]

This was auto-generated with the command pants tailor ::

[jsirois]

Ah, damn. Yeah. The magic + cruft tanked you there. I'm not sure what we could do about that.