Lambda SAM build (ARM64 archi) - Python libs fails

[gabriele-l]

I am trying to build a Lambda function using sam (1.36.0) within CodeBuild, the Lambda has arm64 architecture as well as the CodeBuild configuration (aws/codebuild/amazonlinux2-aarch64-standard:2.0).
Using Python 3.8.10
I have in my requirements.txt the dependencies below:

PyYAML
pandas
s3fs
xlsxwriter

CodeBuild throws error:

Build Failed
Error: PythonPipBuilder:ResolveDependencies - {wrapt==1.13.3(sdist)}

My build details:

version: 0.2
phases:
  install:
    runtime-versions:
      python: 3.8
    commands:
      - which python
      - python --version
      - pip install aws-sam-cli --upgrade
      - sam build --template $(pwd)/template.yaml
      - aws cloudformation package --template $(pwd)/.aws-sam/build/template.yaml --s3-bucket XXXXX --s3-prefix PREFIX_PATH_DUMMY --output-template-file product.template-eu-west-1.yaml
      - sam deploy --template-file ./product.template-eu-west-1.yaml --stack-name DUMMY --capabilities CAPABILITY_IAM

I also manually tested this on t4g Graviton instance, doing pip install requirements.txt, there it works. But when using CodeBuild it fails.

[ssenchenko]

So sorry you encountered a bug! Thanks for reporting the issue, we will be investigating it further.

[awood45]

To help with reproduction, are you able to individually install each of those packages on that environment? My immediate suspicion, especially since it's a non-container build (the right thing to do probably on a matching CodeBuild environment), is that one of your dependencies is having trouble building on ARM64.

[gabriele-l]

Hi @awood45, as described in my issue, all of these dependencies were successfully built on a freshly created Amazon linux 2 running on arm architecture (t4g). So they work fine on arm64 architecture.

When running in CodeBuild the dependency creating the issue is: s3fs

sam build --use-container --debug

s3fs
https://pypi.org/project/s3fs/

[Kotaimen]

Hi,

I ran into a very similar situtation: build arm64 python lambda function with dependency wrapt==1.13.3 fails.
(somewhat related to #2291)

Easiest way to reproduce this is to create the sample "hello-world" SAM app, then:

1. SAM 1.36.0
2. Add wrapt==1.13.3 to hello world function's requirements.txt,
3. Change lambda archiecture in template.yaml to arm64,
4. Run sam build --debug --use-container (yes I am using container build)

Output is trimmed, PipBuilder attempts to build wrapt wheel form sdist but the wheel is incompatible:

...
...
...
Running workflow 'PythonPipBuilder'
Running PythonPipBuilder:ResolveDependencies
calling pip download -r /tmp/samcli/source/requirements.txt --dest /tmp/samcli/scratch --exists-action i
Full dependency closure: {charset-normalizer==2.0.9(wheel), wrapt==1.13.3(sdist), idna==3.3(wheel), requests==2.26.0(wheel), certifi==2021.10.8(wheel), urllib3==1.26.7(wheel)}
initial compatible: {charset-normalizer==2.0.9(wheel), idna==3.3(wheel), requests==2.26.0(wheel), certifi==2021.10.8(wheel), urllib3==1.26.7(wheel)}
initial incompatible: {wrapt==1.13.3(sdist)}
Downloading missing wheels: {wrapt==1.13.3(sdist)}
calling pip download --only-binary=:all: --no-deps --platform manylinux2014_aarch64 --implementation cp --abi cp39 --dest /tmp/samcli/scratch wrapt==1.13.3
compatible wheels after second download pass: {charset-normalizer==2.0.9(wheel), idna==3.3(wheel), requests==2.26.0(wheel), certifi==2021.10.8(wheel), urllib3==1.26.7(wheel)}
Build missing wheels from sdists (C compiling True): {wrapt==1.13.3(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/wrapt-1.13.3.tar.gz
compatible after building wheels (no C compiling): {charset-normalizer==2.0.9(wheel), idna==3.3(wheel), requests==2.26.0(wheel), certifi==2021.10.8(wheel), urllib3==1.26.7(wheel)}
Build missing wheels from sdists (C compiling False): {wrapt==1.13.3(sdist)}
calling pip wheel --no-deps --wheel-dir /tmp/samcli/scratch /tmp/samcli/scratch/wrapt-1.13.3.tar.gz
compatible after building wheels (C compiling): {charset-normalizer==2.0.9(wheel), idna==3.3(wheel), requests==2.26.0(wheel), certifi==2021.10.8(wheel), urllib3==1.26.7(wheel)}
Final compatible: {idna==3.3(wheel), requests==2.26.0(wheel), charset-normalizer==2.0.9(wheel), certifi==2021.10.8(wheel), urllib3==1.26.7(wheel)}
Final incompatible: {wrapt==1.13.3(wheel)}
Final missing wheels: {wrapt==1.13.3(sdist)}
PythonPipBuilder:ResolveDependencies failed
Traceback (most recent call last):
  File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/actions.py", line 51, in execute
    package_builder.build_dependencies(
  File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 157, in build_dependencies
    self._dependency_builder.build_site_packages(requirements_path, artifacts_dir_path, scratch_dir_path)
  File "/usr/local/opt/sam-cli/lib/python3.9/site-packages/aws_lambda_builders/workflows/python_pip/packager.py", line 261, in build_site_packages
    raise MissingDependencyError(packages_without_wheels)
aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {wrapt==1.13.3(sdist)}
...
...
2021-12-26 20:30:33,747 | Build inside container returned response {"jsonrpc": "2.0", "id": 1, "error": {"code": 400, "message": "PythonPipBuilder:ResolveDependencies - {wrapt==1.13.3(sdist)}"}}

Build Failed
2021-12-26 20:30:34,474 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '67b4c998-fb4d-4526-abbe-b722ff8cd67b', 'installationId': '405b296d-235b-4111-99cd-df9827aebcaa', 'sessionId': 'e03338fe-6210-4ca6-978b-ab11c2960c06', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.9.9', 'samcliVersion': '1.36.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 217301, 'exitReason': 'BuildInsideContainerError', 'exitCode': 1}}]}
Error: PythonPipBuilder:ResolveDependencies - {wrapt==1.13.3(sdist)}

I believe this is partly due to wrapt has c extensions and doesn't provide a wheel for arm64 archiecture, and aws_lambda_builders is not handling the situtaion. I haven't dig deep into the python pip builder so currently i'm just reporting the issue here.

Cyril

[konz-rs]

same issue reported here: aws/aws-lambda-builders#302