Compatible wheel package determination ignores py2/py3 designation

[jasonmk]

Description:

CAVEAT: Today is the first time I've looked into how aws-sam-cli works so if I'm missing something glaringly obvious, I apologize. I'm also fairly new to python, so if there's an easy fix just with a requirements.txt flag or something, I'm all ears. However, I've tried a few different things to force it to not use the incompatible wheel and nothing has worked so far.

I'm trying to build a function on the Python 3.6 runtime that uses BeatifulSoup 4.7.1. The function fails because the version of beautifulsoup that gets installed is only compatible with Python2. There are separate versions on pypi for py2 and py3. It's possible that part of this issue may be poor naming on the part of the package maintainer, but all the information needed is included, but possibly not in the way the builder is looking for it.

On pypi.org, there are two wheel files for recent versions:

beautifulsoup4-4.7.1-py2-none-any.whl
beautifulsoup4-4.7.1-py3-none-any.whl

The issue is that on this line: https://github.com/awslabs/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/python_pip/packager.py#L375

Adding some additional debugging to the builder code reveals that it's downloading the py2 version first and then deciding that it's good enough. The code just exits out because it sees the abi of none without checking the implementation as well. Again, may be this is an incorrectly named package, but I don't know enough about the Python package naming system to know for sure. However, it seems like there could be an additional check before returning to validate that the implementation version matches.

Steps to reproduce the issue:

1. Create a Python3.6 lambda function and add beautifulsoup4 to the requirements.txt. Import it into your function.
2. Build your function with sam build
3. Run the function and see the syntax error - The error message is really bad, but if you trace it down you find that it comes from a line in the beautifulsoup4 __init__.py where there is a string in there designed to fail on python3 telling you that you have a version for python2 and need to rebuild it for python3.

Observed result:
Syntax error in module 'app': invalid syntax (__init__.py, line 52)

Expected result:
Function should actually run

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
aws-sam-cli 0.10.0

[jasonmk]

After further investigation, it looks like this might be able to be worked around by using a python3-only virtual env. I was hoping to avoid this in case we had a project that used both python2 and python3 functions, but in retrospect that would probably break completely anyways.

Apparently, even though I had python36 as my default python and was using the python36 runtime, aws-sam-cli was still hard-coded in my install to run under python27 and as a result was using pip2.7. Would it make more sense (or even be possible) to have the lambda-builder use the version of pip that matches the runtime version rather than the version matching sam-cli?

[jfuss]

@jasonmk Thanks for the detailed issue!

What OS are you running on?

We are adding some improvements in this area in #55. We (SAM CLI) pick the first python in the PATH, the PR will be adding support for picking the first executable in the PATH that is validated. From SAM CLI, you can also use the --use-container option that will run the build in a container. This will ensure the correct python version is used. It is slower but does give you another path forward. You can also muck with your PATH but I tend to just do the --use-container until we have #55 merged and released.

[jasonmk]

This is CentOS 7. I had tried it with a container but got a different error. However, that seems to be working now so it's possible I had the same cross-contamination issue there as well.

Definitely a lot slower....

I almost wonder if it wouldn't make more sense to treat python the same as node or ruby. Don't make any assumptions about anything (first in the path or default versions etc) and specifically look up the commands for the runtime in use. That would have the advantage of also letting you combine python2 and python3 functions in the same project without using containers. (I'm actually against using python2 at all for philosophical reasons, but I'm sure people out there do it.)

[jfuss]

#55 will get you that flexibility (authored by @thesriram). Python was the first one written and we are heading towards a more robust way of resolving the executables across all runtimes supported.

"(I'm actually against using python2 at all for philosophical reasons, but I'm sure people out there do it.)"
But you installed SAM CLI in Py2 :(

[jasonmk]

Not on purpose. I try to always keep python3 as the default, but apparently something got messed up. I'm not a python developer by far so I get confused a lot switching around between the system default and what I'm trying to use for my projects....

[jfuss]

I think this issue is a duplicate of #11 and #30. This both should be captured by PR #55. I am inclined to close in favor of the ones we have open.

On the installation, for linux we provide LinuxBrew as an installation mechanism. We are considering expanding this to Snap as well. Either would help in keeping you system sane and get you off Py2.

[jasonmk]

Never heard of linuxbrew. Snap wasn't CentOS compatible last I checked, but I know they're working on it.

Feel free to close if it's covered. I'll keep on eye on 55.

[sriram-mv]

@jasonmk Yes #55 should help you to run build with the appropriate python executable that matches your runtime.

[CoshUS]

Closed as #55 is merged.