'UndefinedEnvironmentName: 'python_full_version' does not exist in evaluation environment' when building lambda with any 3rd party dev on arm64 macOS

[huonw]

Describe the bug

I'm trying to build a python_awslambda on an arm64 macOS device. As soon as I add any 3rd party dependency (with prebuilt wheels that are highly compatible: https://pypi.org/project/urllib3/1.26.9/#files) in a minimal project https://github.com/huonw/pants-awslambda-UndefinedEnvironmentName, I get an error about python_full_version:

$ tree .
tree .     
.
├── 3rdparty
│   └── python
│       └── default.lock
├── BUILD
├── pants
├── pants.toml
└── src
    └── lambda-package
        ├── BUILD
        └── lambda.py

4 directories, 6 files

# pants.toml                                                          
[GLOBAL]
pants_version = "2.10.1rc0"

backend_packages = [
  "pants.backend.python",
  "pants.backend.awslambda.python",
]

use_deprecated_python_macros = false

[python]
interpreter_constraints = ["CPython==3.9.*"]
enable_resolves = true

[anonymous-telemetry]
enabled = false

# src/lambda-package/BUILD
python_sources()

# wheel compatible with all Pythons/platforms: https://pypi.org/project/urllib3/1.26.9/#files
python_requirement(name="urllib3", requirements=["urllib3==1.26.9"])

python_awslambda(
    name="awslambda",
    handler="lambda.py:handler",
    runtime="python3.9",
)

# src/lambda-package/lambda.py
import urllib3

def handler():
    print(urllib3.__version__)

$ ./pants package src/lambda-package:awslambda
19:59:40.97 [WARN] AWS Lambdas built on macOS may fail to build. If your lambda uses any third-party dependencies without binary wheels (bdist) for Linux available, it will fail to build. If this happens, you will either need to update your dependencies to only use dependencies with pre-built wheels, or find a Linux environment to run ./pants package. (See https://realpython.com/python-wheels/ for more about wheels.)

(If the build does not raise an exception, it's safe to use macOS.)
19:59:42.03 [INFO] Completed: Building src.lambda-package/awslambda.zip with 1 requirement: urllib3==1.26.9
19:59:42.03 [ERROR] 1 Exception encountered:

  ProcessExecutionFailure: Process 'Building src.lambda-package/awslambda.zip with 1 requirement: urllib3==1.26.9' failed with exit code 1.
stdout:

stderr:
ERROR: Exception:
Traceback (most recent call last):
  File "/Users/huon/.cache/pants/named_caches/pex_root/venvs/s/e1a0e82b/venv/lib/python3.10/site-packages/pip/_internal/cli/base_command.py", line 223, in _main
    status = self.run(options, args)
  File "/Users/huon/.cache/pants/named_caches/pex_root/venvs/s/e1a0e82b/venv/lib/python3.10/site-packages/pip/_internal/cli/req_command.py", line 180, in wrapper
    return func(self, options, args)
  File "/Users/huon/.cache/pants/named_caches/pex_root/venvs/s/e1a0e82b/venv/lib/python3.10/site-packages/pip/_internal/commands/download.py", line 130, in run
    requirement_set = resolver.resolve(
  File "/Users/huon/.cache/pants/named_caches/pex_root/venvs/s/e1a0e82b/venv/lib/python3.10/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 90, in resolve
    if not req.match_markers():
  File "/Users/huon/.cache/pants/named_caches/pex_root/venvs/s/e1a0e82b/venv/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 290, in match_markers
    return any(
  File "/Users/huon/.cache/pants/named_caches/pex_root/venvs/s/e1a0e82b/venv/lib/python3.10/site-packages/pip/_internal/req/req_install.py", line 291, in <genexpr>
    self.markers.evaluate({'extra': extra})
  File "/Users/huon/.cache/pants/named_caches/pex_root/venvs/s/e1a0e82b/venv/lib/python3.10/site-packages/pip/_vendor/packaging/markers.py", line 336, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/Users/huon/.cache/pants/named_caches/pex_root/venvs/s/e1a0e82b/venv/lib/python3.10/site-packages/pip/_vendor/packaging/markers.py", line 247, in _evaluate_markers
    groups[-1].append(_evaluate_markers(marker, environment))
  File "/Users/huon/.cache/pants/named_caches/pex_root/venvs/s/e1a0e82b/venv/lib/python3.10/site-packages/pip/_vendor/packaging/markers.py", line 252, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/Users/huon/.cache/pants/named_caches/pex_root/venvs/s/e1a0e82b/venv/lib/python3.10/site-packages/pip/_vendor/packaging/markers.py", line 232, in _get_env
    raise UndefinedEnvironmentName(
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'python_full_version' does not exist in evaluation environment.
pid 76671 -> /Users/huon/.cache/pants/named_caches/pex_root/venvs/96480d4730684b9f35e3b227b5f7ff2f259301b2/1a8c9b6a06d34ea8e284376abc61eb459eadac8f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/huon/.cache/pants/named_caches/pex_root --log /private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/process-executionOtvmPW/.tmp/tmpu539vkm5/pip.log download --dest /private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/process-executionOtvmPW/.tmp/tmp8xv0mshb/cp39-cp39-linux_x86_64 --only-binary :all: --constraint __constraints.txt urllib3==1.26.9 --platform manylinux2014_x86_64 --platform linux_x86_64 --implementation cp --python-version 39 --abi cp39 --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 2 and STDERR:
Failed to resolve for platform linux_x86_64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'python_full_version' does not exist in evaluation environment.

NB. the build works if I comment out the import urllib3 from src/lambda-package/lambda.py

Pants version
The linked project reproduces this for me on all of (with just minor adjustments to the pants.toml):

• 2.10.0
• 2.10.1rc0
• 2.11.0rc5
• 2.11.0
• 2.12.0.dev3

This seems to be related to #14995, but the fix for that has apparently landed in all but 2.10.0.

OS
macOS

Additional info

• https://github.com/huonw/pants-awslambda-UndefinedEnvironmentName

[jsirois]

@huonw the issue here is the lockfile, which is generated by Poetry and exported to requirements.txt format: https://github.com/huonw/pants-awslambda-UndefinedEnvironmentName/blob/main/3rdparty/python/default.lock

In the process of exporting the Poetry lockfile, Poetry adds the python_full_version environment marker. In order to be able to evaluate that marker, you need the full (say 3.9.9) Python version. Your lambda target only has enough information to say 3.9 but not to determine the patch version of Python.

There are two ways to fix this:

1. Use the "pex" lockfile generator instead of the default Poetry lockfile generator.
2. Instead of specifying runtime="python3.9" for your lambda, specify complete_platforms which allows you to hand the resolver the complete set of information about the target Python interpreter, which includes, amongst other things, the Python full version.

To do the former, use Pants 2.11.0 and configure [python].lockfile_generator = "pex": https://www.pantsbuild.org/docs/python-third-party-dependencies#pex-vs-poetry-for-lockfile-generation
The lockfile will look roughly like this; N.B.: there is no python_full_version environment marker used, just python_version which is knowable from runtime="python3.9", it's "3.9":

{
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "44ece4d53fb1706f667c9bd1c648f5469a2ec925fcf3a776667042d645472c14",
              "url": "https://files.pythonhosted.org/packages/ec/03/062e6444ce4baf1eac17a6a0ebfe36bb1ad05e1df0e20b110de59c278498/urllib3-1.26.9-py2.py3-none-any.whl"
            },
            {
              "algorithm": "sha256",
              "hash": "aabaf16477806a5e1dd19aa41f8c2b7950dd3c746362d7e3223dbe6de6ac448e",
              "url": "https://files.pythonhosted.org/packages/1b/a5/4eab74853625505725cefdf168f48661b2cd04e7843ab836f3f63abf81da/urllib3-1.26.9.tar.gz"
            }
          ],
          "project_name": "urllib3",
          "requires_dists": [
            "PySocks!=1.5.7,<2.0,>=1.5.6; extra == \"socks\"",
            "brotli>=1.0.9; ((os_name != \"nt\" or python_version >= \"3\") and platform_python_implementation == \"CPython\") and extra == \"brotli\"",
            "brotlicffi>=0.8.0; ((os_name != \"nt\" or python_version >= \"3\") and platform_python_implementation != \"CPython\") and extra == \"brotli\"",
            "brotlipy>=0.6.0; (os_name == \"nt\" and python_version < \"3\") and extra == \"brotli\"",
            "certifi; extra == \"secure\"",
            "cryptography>=1.3.4; extra == \"secure\"",
            "idna>=2.0.0; extra == \"secure\"",
            "ipaddress; python_version == \"2.7\" and extra == \"secure\"",
            "pyOpenSSL>=0.14; extra == \"secure\""
          ],
          "requires_python": "!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,<4,>=2.7",
          "version": "1.26.9"
        }
      ],
      "platform_tag": [
        "cp310",
        "cp310",
        "manylinux_2_35_x86_64"
      ]
    }
  ],
  "path_mappings": {},
  "pex_version": "2.1.84",
  "prefer_older_binary": false,
  "requirements": [
    "urllib3==1.26.9"
  ],
  "requires_python": [],
  "resolver_version": "pip-2020-resolver",
  "style": "universal",
  "transitive": true,
  "use_pep517": null
}

An example of the latter is here: #15239 (comment). You should be able to follow that to the letter but you do not need two platform files as used there, just the linux lambda platform file is enough if you only intend to run the lambda out in AWS.

[huonw]

Ah I see. Thank you for the pointers @jsirois, they should unblock us from using Pants. 👍

The new pex lockfile generation looks nifty, but I think we'll be stuck using poetry to generate our lockfiles:

• We want to retain compatibility with dependabot (particularly focused on vulnerability scanning/notification, not 'keep everything at latest versions always').
• Additionally, it seems like we may be having to generate it manually, since it seems any call to generate-lockfiles will update all existing deps to their latest (constraint-satisfying) versions and we want to have more control over when deps update than that. This requirement seems to overlap with Implement a goal to automate a portion of (root) dependency upgrades #15275 and New goal for managing 3rd party dependencies #12880.

Given this, we'll be using complete_platforms. I imagine AWS updates their lambda Python environment semi-regularly, potentially invalidating the information in a generated complete_platforms file. I imagine usually this is fine, but could cause issues if a pex expecting a certain specific Python version is run on a newer one (or some other variation in the platform). Do you know if this causes problems in practice, or is it okay that the build may not match up exactly with the runtime platform?

[jsirois]

I think this likely won't cause problems in practice.

As to lock file updates, Pex does support a lock update -p just-update-this-rewuirement-but-not-the-rest --lock my.lock but Pants doesn't yet use it.

[huonw]

Thanks for the tip.

I had a chance to try the complete_platforms approach, and I don't seem to be able to get it to work:

Just Linux

I ran pex3 interpreter inspect --markers --tags within Lambda, to get: https://github.com/huonw/pants-awslambda-UndefinedEnvironmentName/blob/347bc22fd5b7328b4233eddad9df570b4eb84c03/src/lambda-package/complete_platform_lambda.json

I then adjusted the BUILD file to use this: https://github.com/huonw/pants-awslambda-UndefinedEnvironmentName/blob/e37005be9c29917546bb35bab882be05823cc894/src/lambda-package/BUILD (diff: huonw/pants-awslambda-UndefinedEnvironmentName@e37005b)

Running ./pants package src/lambda-package:awslambda still fails, with the same error:

...
pid 230 -> /Users/huon/.cache/pants/named_caches/pex_root/venvs/4a4903f4a53b18a8968a4b2bac7896c88ae1b1da/1a8c9b6a06d34ea8e284376abc61eb459eadac8f/pex --disable-pip-version-check --no-python-version-warning --exists-action a --isolated -q --cache-dir /Users/huon/.cache/pants/named_caches/pex_root --log /private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/process-executionFMne0q/.tmp/pex-pip-log_mj8_ufo/pip.log download --dest /private/var/folders/sv/vd266m4d4lvctgs2wpnhjs9w0000gn/T/process-executionFMne0q/.tmp/tmp72h7i6jd/cp39-cp39-linux_x86_64 --only-binary :all: --constraint __constraints.txt urllib3==1.26.9 --platform manylinux2014_x86_64 --platform linux_x86_64 --implementation cp --python-version 39 --abi cp39 --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 2 and STDERR:
Failed to resolve for platform linux_x86_64-cp-39-cp39. Resolve requires evaluation of unknown environment marker: 'python_full_version' does not exist in evaluation environment.

(I note that the pex invocation doesn't seem to be passing --complete-platforms?)

The archive is so I can double check that file is pulling in the correct file, and indeed it is ./pants package src/lambda-package:validate_complete_platform_lambda creates dist/src.lambda-package/validate_complete_platform_lambda.zip containing that file.

Mac and Linux

From #15239 (comment), it sounds like adding something for the build platform may help, but that adjustment didn't seem to (diff: huonw/pants-awslambda-UndefinedEnvironmentName@347bc22)

This gives the same error as before.

---

Any tips for what I'm doing wrong here? Thanks for tolerating my basic questions.

[jsirois]

You must remove the runtime parameter when adding complete_platforms. Did you try that?

[jsirois]

My apologies for the terse responses. At a keyboard now, so to clarify a few things:

Although both runtime and complete_platforms are allowed, that was probably a mistake for 95% of use-cases. You'll almost always want one or the other.

To expand on the:

I imagine AWS updates their lambda Python environment semi-regularly

Pex really isn't at play here. The complete platform data is the actual data used to perform a dependency resolve by PEP standards by Pip, Poetry, etc. Most of the data is not needed for most dependencies. On the environment marker side, a very large percentage of dependencies only use the python_version marker if any at all. As such, since the usage is sparse, the stability of your 3.9 resolve using a complete platform should be robust unless you happen to include a very titchy dependency in your transitive closure that uses python_full_version and the Python patch version changes in a way significant to it, etc.

And thanks very much for your example repo. That is, of course, gold and really helps root things out very quickly.

[huonw]

My apologies for the terse responses. At a keyboard now, so to clarify a few things:

No worries at all, thank you for taking the time to work through this.

You must remove the runtime parameter when adding complete_platforms. Did you try that?

Although both runtime and complete_platforms are allowed, that was probably a mistake for 95% of use-cases. You'll almost always want one or the other.

Ah, I didn't try that. Removing the runtime parameter does indeed work 👍 For future reference, the minimal working state seems to be: https://github.com/huonw/pants-awslambda-UndefinedEnvironmentName/blob/7707399255d62ae80103ef95ecaf845eeff652cf/src/lambda-package/BUILD

It seems like the docs don't seem mention any interaction between runtime and complete_platforms, and the code only validates that at least one is specified. Is there something that could be done to improve the situation for future? I'd love to contribute (e.g. add something to the docs), but it sounds like there's a small number of cases where one might indeed want to specify both so a code warning or error may not be appropriate?

Pex really isn't at play here. The complete platform data is the actual data used to perform a dependency resolve by PEP standards by Pip, Poetry, etc. Most of the data is not needed for most dependencies. On the environment marker side, a very large percentage of dependencies only use the python_version marker if any at all. As such, since the usage is sparse, the stability of your 3.9 resolve using a complete platform should be robust unless you happen to include a very titchy dependency in your transitive closure that uses python_full_version and the Python patch version changes in a way significant to it, etc.

Ah I see; thanks for the explanation. Looking through our lock file, all the uses of python_full_version seem to be either be python_full_version >= "3.7.0" or similar, with a small number that are python_full_version >= "3.6.2". So, sounds like it won't be a problem as you suspect 👍

And thanks very much for your example repo. That is, of course, gold and really helps root things out very quickly.

😄

[benjyw]

Can we close this?

[huonw]

I've got it working in our code (thanks again for the tech support), so feel free to close if you think that's satisfactory.

However, looking over https://www.pantsbuild.org/docs/awslambda-python, I suspect I'd personally still hit the same issues if I was starting again, with 2.13. Maybe:

1. the docs could be updated to call out the use of complete_platforms? Is it just for macOS?
2. the python_awslambda target could give an error/warning if both runtime and complete_platforms are specified?

I would offer PRs for both of these, but I'm not familiar enough with the constraints (or pants/pex) to validate if they make sense or phrase the general rules 😄

[huonw]

I think the docs were improved in #18001, and I think they're enough for past-me to have worked the problems in this issue, so I'll call this done.

I had an additional idea that might improve the situation more: #18195