Conforming with PEP 518 to use pyproject.toml instead of requirements.txt/setuptools

[tcco]

Describe your idea/feature/enhancement

Provide a clear description. Ex. I wish SAM CLI would [...]

I wish SAM CLI would utilize pyproject.toml for requirements. Subsequently, this would use additional package managers like poetry to in the sam build process.

Proposal

Add details of how to add this to the product.

Add functionality to read pyproject.toml when running sam build

Things to consider:

1. Will this require any updates to the SAM Spec

This would likely add another way to setup code architecture and application requirements in template.yaml files.

Additional Details

[sriram-mv]

This is pertinent to #1369 as well, we are trying to get reproducible builds within sam cli when creating our installers. What has your experience been like with poetry? Does it also allow for generation of pinned dependencies.

Reading about poetry, it seems to be pretty nice. @jfuss has also been looking into this recently. Might be another way to look at reproducible builds.

For adding functionality to read pyproject.toml for sam build, this could be another workflow in aws-sam-builders and sam build can choose that workflow in the future. Could you open an issue there as well?

[tcco]

@thesriram Thanks for the reply. Getting pinned requirements isn't bad. I have a python script that will read the poetry.lock files. Let me know if this is helpful. I've used this script to grep to a requirements.txt as a workaround for the time being.

 import tomlkit


 with open("poetry.lock") as f:
     lock = tomlkit.parse(f.read())
     for p in lock["package"]:
         if not p["category"] == "dev":
             print(f"{p['name']}=={p['version']}")

[ericofusco]

@tcco Poetry has a command to generate requirements.txt

poetry export -f requirements.txt --without-hashes > requirements.txt

[CaptainDriftwood]

Has any progress been made on this feature request?

[jfuss]

No progress on this front. This would be a change/addition to https://github.com/aws/aws-lambda-builders as that repo does all the heavy lifting.

Our pip integration is pretty custom: https://github.com/aws/aws-lambda-builders/blob/develop/aws_lambda_builders/workflows/python_pip/DESIGN.md to allow us to build for Lambda. What likely needs to happen is a very similar workflow to the pip needs to be created but reads/uses the pyproject.toml file instead to determine what dependencies to be downloaded.

[HCrane]

Bump to move this feature a little along as we also would like to use it soon (TM). Best wishes to our friends at aws!

[demolitionmode]

What likely needs to happen is a very similar workflow to the pip needs to be created but reads/uses the pyproject.toml file instead to determine what dependencies to be downloaded.

@jfuss, do you think it needs to be a separate workflow, or could the python_pip workflow be updated to allow use of the pyproject.toml file?
I did some really dirty testing to have SubprocessPip.main() call pip with args ["install", ".", "--target", ".aws-sam-foo"], and it seems to behave the same way as raw pip, in that it reads from [build-system] in pyproject.toml.

Do let me know if I've wandered off in the wrong direction.

[crane-denny]

Maybe this is another solution:

Support poetry on a level where it exports the dependencies without hashes into a temporary requirements.txt file. And then let your current code consume this file?

My manual workflow currently is

poetry export --without-hashes > project/requirements.txt
sam build

Of course don't export it into the code directory but maybe into .aws-sam?