Unable to build Python lambda with package dependency in a private Git repository

[berciq]

Description:

I am trying to use sam build to build a Python 3.7 lambda. My requirements.txt file contains the following dependency:

git+ssh://git@github.com/Berciq/commons.git@v0.0.4

So, as you can see, I am trying to include a package hosted in a private Git in my requirements.txt file as per the documentation. The package installs fine when using pip install -r requirements.txt but sam build throws the following error:

Build Failed
Error: PythonPipBuilder:ResolveDependencies - list index out of range

If I remove that dependency from my requirments.txt, sam build runs successfully.

Steps to reproduce the issue:

1. Use sam init to create any Python-based AWS SAM application (e.g. hello-world)
2. Add a private Git repo dependency to the requirements.txt file
3. Run sam build

Observed result:
The build will fail with a PythonPipBuilder:ResolveDependencies error.

Expected result:
The build should succeed, given that running pip install -r requirements.txt succeeds.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)
I'm running Ubuntu 20, AWS SAM CLI v.1.26.

But I was also able to reproduce the problem by installing and importing aws-lambda-builders as a module and running the PythonPipWorkflow, hence I'm raising the issue here.

This appears to be related in some way to Issue #229 but at the same time it's not exactly the same issue, as mine has to do specifically with dependencies hosted in a private repository.

[sriram-mv]

I tested this on a mac by supplying build of aws-sam-cli directly from github and that seemed to work.

I'm on python version 3.8.10, AWS SAM CLI version: 1.27.2

My requirements.txt looks like below

git+https://github.com/aws/aws-sam-cli

Could you try upgrading and also additional traceback information as well?

[prenx4x]

I tried and it was working for me for private repo and python 3.7.
my requirements.txt file looked like

requests
-e git+git@github.com:prenx4x/sampy38-jfs.git#egg=my-git-pack

Can you upgrade to latest sam-cli and try again and then provide additional info so we can try to repro it.

[berciq]

It looks as though PR #265 should address my issue. I will wait for to confirm once those changes carry over to the newest version of SAM CLI.

I am still getting the error with Python 3.7 and SAM CLI version 1.29.0.
Here is what my requirements.txt looks like:

requests
git+ssh://git@github.com-p/Berciq/sample-pkg.git@v0.0.4

And here is the output from sam build --debug:

2021-08-30 11:45:01,869 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-08-30 11:45:01,870 | Using config file: samconfig.toml, config environment: default
2021-08-30 11:45:01,870 | Expand command line arguments to:
2021-08-30 11:45:01,870 | --template_file=/home/bartek/sandbox/sample-sam-app/template.yaml --build_dir=.aws-sam/build --cache_dir=.aws-sam/cache 
2021-08-30 11:45:01,901 | 'build' command is called
2021-08-30 11:45:01,906 | No Parameters detected in the template
2021-08-30 11:45:01,925 | 2 stacks found in the template
2021-08-30 11:45:01,925 | No Parameters detected in the template
2021-08-30 11:45:01,942 | 2 resources found in the stack 
2021-08-30 11:45:01,942 | No Parameters detected in the template
2021-08-30 11:45:01,959 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2021-08-30 11:45:01,959 | --base-dir is not presented, adjusting uri hello_world/ relative to /home/bartek/sandbox/sample-sam-app/template.yaml
2021-08-30 11:45:01,959 | No Parameters detected in the template
2021-08-30 11:45:01,977 | Instantiating build definitions
2021-08-30 11:45:01,977 | No previous build graph found, generating new one
2021-08-30 11:45:01,977 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(python3.7, /home/bartek/sandbox/sample-sam-app/hello_world, Zip, , f5cc0c06-8e35-4ddf-90c9-e24b594ef076, {}, {}, []), Function: Function(name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='python3.7', memory=None, timeout=3, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/home/bartek/sandbox/sample-sam-app/hello_world', environment=None, rolearn=None, layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId': 'ServerlessRestApi'}}}, metadata=None, inlinecode=None, codesign_config_arn=None, stack_path=''))
2021-08-30 11:45:01,978 | Building codeuri: /home/bartek/sandbox/sample-sam-app/hello_world runtime: python3.7 metadata: {} functions: ['HelloWorldFunction']
2021-08-30 11:45:01,978 | Building to following folder /home/bartek/sandbox/sample-sam-app/.aws-sam/build/HelloWorldFunction
2021-08-30 11:45:01,979 | Loading workflow module 'aws_lambda_builders.workflows'
2021-08-30 11:45:01,981 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-08-30 11:45:01,982 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-08-30 11:45:01,982 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2021-08-30 11:45:01,983 | Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2021-08-30 11:45:01,984 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2021-08-30 11:45:01,984 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2021-08-30 11:45:01,985 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2021-08-30 11:45:01,986 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2021-08-30 11:45:01,987 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-08-30 11:45:01,987 | Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-08-30 11:45:01,999 | Running workflow 'PythonPipBuilder'
2021-08-30 11:45:01,999 | Running PythonPipBuilder:ResolveDependencies
2021-08-30 11:45:02,017 | calling pip download -r /home/bartek/sandbox/sample-sam-app/hello_world/requirements.txt --dest /tmp/tmpev9tvprz --exists-action i
2021-08-30 11:45:05,288 | PythonPipBuilder:ResolveDependencies raised unhandled exception
Traceback (most recent call last):
  File "aws_lambda_builders/workflow.py", line 269, in run
  File "aws_lambda_builders/workflows/python_pip/actions.py", line 42, in execute
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 145, in build_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 223, in build_site_packages
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 247, in _download_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 330, in _download_all_dependencies
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 330, in <setcomp>
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 468, in __init__
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 510, in _calculate_name_and_version
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 570, in get_package_name_and_version
  File "aws_lambda_builders/workflows/python_pip/packager.py", line 553, in _generate_egg_info
IndexError: list index out of range

Build Failed
2021-08-30 11:45:05,290 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': '7a7b2f7a-5fd4-4f80-a17e-ecf6f7869f0c', 'installationId': '8b2c648e-3b81-425e-8ddd-9194c667999e', 'sessionId': '42352b2a-fb70-49f6-804e-e63eb87a5956', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.7.10', 'samcliVersion': '1.29.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 3419, 'exitReason': 'WorkflowUnknownError', 'exitCode': 1}}]}
2021-08-30 11:45:06,218 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Error: PythonPipBuilder:ResolveDependencies - list index out of range

I should add that running the following works correctly:

pip3.7 install -r /home/bartek/sandbox/sample-sam-app/hello_world/requirements.txt -t /tmp/pip-test