samconfig.toml is ignored when running with a custom build directory

[eyalroth]

Description:

samconfig.toml is ignored when running with a custom build directory, probably due to its path being relative to the custom build directory instead of the root directory.

Steps to reproduce:

1. Create a new project using sam init with the Hello World Example:

$ sam init
1 - AWS Quick Start Templates
1 - Zip (artifact is a zip uploaded to S3)
8 - nodejs10.x
Project name [sam-app]: sam-test
1 - Hello World Example

2. Add a simple console.log(process.env.Foo); to hello-world/app.js.
3. Add an environment variable Foo which points to parameter Bar in the function in the template.yml:

Environment:
  Variables:
    Foo: !Ref Bar

4. Create the following samconfig.toml:

version=0.1
[default.local_invoke.parameters]
parameter_overrides="Bar=456"

5. Build with a custom build directory (sam build -b out/build).
6. Local invoke (sam local invoke -t out/build/template.yaml).

Observed result:

Environment variable read by the lambda is the default (Bar) instead of the set parameter (456).

This is not a specific problem with environment variables nor override parameters. This also happened with --log-file and I would assume with other command parameters as well. It's simply a bit more visible with this setup.

This does actually work in the following cases:

1. --config-file is set to the absolute path of the configuration file or a path relative to the custom build directory.
2. local invoke is executed without -t out/build/template.yaml (which differs from the instruction outputted by the build command).

sam build -b out/build --debug
2021-03-21 16:58:50,484 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-03-21 16:58:50,541 | 'build' command is called
2021-03-21 16:58:50,548 | No Parameters detected in the template
2021-03-21 16:58:50,574 | 2 stacks found in the template
2021-03-21 16:58:50,574 | No Parameters detected in the template
2021-03-21 16:58:50,595 | 2 resources found in the stack
2021-03-21 16:58:50,595 | No Parameters detected in the template
2021-03-21 16:58:50,630 | No Parameters detected in the template
2021-03-21 16:58:50,651 | Found Serverless function with name='HelloWorldFunction' and CodeUri='./hello-world/'
2021-03-21 16:58:50,651 | No Parameters detected in the template
2021-03-21 16:58:50,673 | Instantiating build definitions
2021-03-21 16:58:50,673 | No previous build graph found, generating new one
2021-03-21 16:58:50,673 | Unique function build definition found, adding as new (Function Build Definition: BuildDefinition(nodejs10.x, ./hello-world/, Zip, , 29230a6e-deb6-41ca-80b2-dcccba9d61a5, {}, {}, []), Function: Function(name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='nodejs10.x', memory=None, timeout=3, handler='app.lambdaHandler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='./hello-world/', environment={'Variables': {'Foo': 'Bar'}}, rolearn=None, layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId': 'ServerlessRestApi'}}}, metadata=None, inlinecode=None, codesign_config_arn=None, stack_path=''))
2021-03-21 16:58:50,674 | Building codeuri: ./hello-world/ runtime: nodejs10.x metadata: {} functions: ['HelloWorldFunction']
2021-03-21 16:58:50,674 | Building to following folder /Users/eyal/repos/sam/sam-test/out/build/HelloWorldFunction
2021-03-21 16:58:50,675 | Loading workflow module 'aws_lambda_builders.workflows'
2021-03-21 16:58:50,677 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-03-21 16:58:50,679 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-03-21 16:58:50,680 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2021-03-21 16:58:50,681 | Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2021-03-21 16:58:50,683 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2021-03-21 16:58:50,685 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2021-03-21 16:58:50,687 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2021-03-21 16:58:50,688 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2021-03-21 16:58:50,690 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-03-21 16:58:50,690 | Found workflow 'NodejsNpmBuilder' to support capabilities 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-03-21 16:58:50,691 | Running workflow 'NodejsNpmBuilder'
2021-03-21 16:58:50,691 | Running NodejsNpmBuilder:NpmPack
2021-03-21 16:58:50,691 | NODEJS packaging file:/Users/eyal/repos/sam/sam-test/hello-world to /var/folders/cv/b_4sr2pj0cx29m6f5grcc8q80000gn/T/tmpquub_78j
2021-03-21 16:58:50,691 | executing NPM: ['npm', 'pack', '-q', 'file:/Users/eyal/repos/sam/sam-test/hello-world']
2021-03-21 16:58:51,226 | NODEJS packed to hello_world-1.0.0.tgz
2021-03-21 16:58:51,226 | NODEJS extracting to /var/folders/cv/b_4sr2pj0cx29m6f5grcc8q80000gn/T/tmpquub_78j/unpacked
2021-03-21 16:58:51,228 | NodejsNpmBuilder:NpmPack succeeded
2021-03-21 16:58:51,228 | Running NodejsNpmBuilder:CopyNpmrc
2021-03-21 16:58:51,228 | NodejsNpmBuilder:CopyNpmrc succeeded
2021-03-21 16:58:51,228 | Running NodejsNpmBuilder:CopySource
2021-03-21 16:58:51,229 | NodejsNpmBuilder:CopySource succeeded
2021-03-21 16:58:51,229 | Running NodejsNpmBuilder:NpmInstall
2021-03-21 16:58:51,229 | NODEJS installing in: /Users/eyal/repos/sam/sam-test/out/build/HelloWorldFunction
2021-03-21 16:58:51,229 | executing NPM: ['npm', 'install', '-q', '--no-audit', '--no-save', '--production', '--unsafe-perm']
2021-03-21 16:58:53,869 | NodejsNpmBuilder:NpmInstall succeeded
2021-03-21 16:58:53,869 | Running NodejsNpmBuilder:CleanUpNpmrc
2021-03-21 16:58:53,870 | NodejsNpmBuilder:CleanUpNpmrc succeeded

Build Succeeded

Built Artifacts  : out/build
Built Template   : out/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke -t out/build/template.yaml
[*] Deploy: sam deploy --guided --template-file out/build/template.yaml

2021-03-21 16:58:53,881 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'c4874b78-3998-400a-be41-41320568e793', 'installationId': 'ce3c8c99-1775-4f86-8b41-0fcf0df0281a', 'sessionId': '66e27ba4-99f3-4b58-8e3a-0625a22c7820', 'executionEnvironment': 'CLI', 'pyversion': '3.8.8', 'samcliVersion': '1.20.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 3395, 'exitReason': 'success', 'exitCode': 0}}]}
2021-03-21 16:58:54,834 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

$ sam local invoke -t out/build/template.yaml --debug
2021-03-21 16:59:26,716 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-03-21 16:59:26,716 | local invoke command is called
2021-03-21 16:59:26,721 | No Parameters detected in the template
2021-03-21 16:59:26,753 | 2 stacks found in the template
2021-03-21 16:59:26,753 | No Parameters detected in the template
2021-03-21 16:59:26,776 | 2 resources found in the stack
2021-03-21 16:59:26,776 | No Parameters detected in the template
2021-03-21 16:59:26,798 | No Parameters detected in the template
2021-03-21 16:59:26,819 | Found Serverless function with name='HelloWorldFunction' and CodeUri='HelloWorldFunction'
2021-03-21 16:59:26,830 | Found one Lambda function with name 'HelloWorldFunction'
2021-03-21 16:59:26,830 | Invoking app.lambdaHandler (nodejs10.x)
2021-03-21 16:59:26,831 | Environment variables overrides data is standard format
2021-03-21 16:59:26,831 | Loading AWS credentials from session with profile 'None'
2021-03-21 16:59:28,851 | Resolving code path. Cwd=/Users/eyal/repos/sam/sam-test/out/build, CodeUri=/Users/eyal/repos/sam/sam-test/out/build/HelloWorldFunction
2021-03-21 16:59:28,851 | Resolved absolute path to code is /Users/eyal/repos/sam/sam-test/out/build/HelloWorldFunction
2021-03-21 16:59:28,852 | Code /Users/eyal/repos/sam/sam-test/out/build/HelloWorldFunction is not a zip/jar file
2021-03-21 16:59:28,877 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-nodejs10.x:rapid-1.20.0.

2021-03-21 16:59:28,877 | Mounting /Users/eyal/repos/sam/sam-test/out/build/HelloWorldFunction as /var/task:ro,delegated inside runtime container
2021-03-21 16:59:29,488 | Starting a timer for 3 seconds for function 'HelloWorldFunction'
2021-03-21T14:59:29.748Z	490fd8c7-528c-44f5-97bb-bfd1d6b584e6	INFO	Bar
END RequestId: 490fd8c7-528c-44f5-97bb-bfd1d6b584e6
REPORT RequestId: 490fd8c7-528c-44f5-97bb-bfd1d6b584e6	Init Duration: 0.12 ms	Duration: 242.32 ms	Billed Duration: 300 ms	Memory Size: 128 MB	Max Memory Used: 128 MB
2021-03-21 16:59:29,956 | Cleaning all decompressed code dirs
2021-03-21 16:59:29,956 | Sending Telemetry: {'metrics': [{'commandRun': {'requestId': 'ee06353b-019d-4073-8408-4ae95b6f5c6e', 'installationId': 'ce3c8c99-1775-4f86-8b41-0fcf0df0281a', 'sessionId': '95945cac-0688-4aa3-9350-fe2448461bb3', 'executionEnvironment': 'CLI', 'pyversion': '3.8.8', 'samcliVersion': '1.20.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam local invoke', 'duration': 3240, 'exitReason': 'success', 'exitCode': 0}}]}
2021-03-21 16:59:31,022 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
{"statusCode":200,"body":"{\"message\":\"hello world\"}"}2021-03-21 16:59:31,022 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-03-21 16:59:31,023 | Sending Telemetry: {'metrics': [{'runtimeMetric': {'requestId': 'cacfa22e-2346-4272-b621-c53a5d859e59', 'installationId': 'ce3c8c99-1775-4f86-8b41-0fcf0df0281a', 'sessionId': '95945cac-0688-4aa3-9350-fe2448461bb3', 'executionEnvironment': 'CLI', 'pyversion': '3.8.8', 'samcliVersion': '1.20.0', 'runtimes': ['nodejs10.x']}}]}
2021-03-21 16:59:33,337 | HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)

Expected result:

1. samconfig.toml in the root directory should be used even with a custom build directory.
2. --config-file path should be relative to the execution directory and not to the custom build directory.
3. Commands should at least warn that the configuration file does not exist if it's explicitly set with --config-file.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: MacOS 11.1.
2. sam --version: SAM CLI, version 1.20.0
3. AWS region: us-east-1

[ssenchenko]

This is a default behaviour as expected. SAM CLI always looks up samconfig.toml at the same directory as the template file. When template file is not specified, it is supposed to be in the current working dir. If it's the project's source root directory, config file is there too. However, when --template-file points to a custom build directory, there is no way to detect project source code root. Using --config-file is the only right approach to pick samconfig.toml in that case.
We were not explicit enough about that. The PR above is supposed to close this gap in the documentation and to make it clear for a user if and which configuration file is used.

[eyalroth]

@ssenchenko Thank you for addressing and fixing this :)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.