`sam sync` does not support custom bucket names

[luhn]

Description:

I don't use the default SAM bucket, I have my own. sam sync does not seem to support this.

Steps to reproduce:

Do sam init and create the zip Python 3.9 "Hello World" template.

Create the following samconfig.toml

version = 0.1

[default]
[default.deploy]
[default.deploy.parameters]
stack_name = "sam-test"
s3_bucket = "mybucket"
s3_prefix = "sam-test"
region = "us-west-2"
capabilities = "CAPABILITY_IAM"

Run sam build && sam deploy, which succeeds.

Observed result:

sam sync --stack-name sam-test gives the following output. You can see it's attempting to use the default managed SAM bucket.

2021-12-17 11:40:14,807 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-12-17 11:40:14,812 | Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
2021-12-17 11:40:14,812 | Sending Telemetry: {'metrics': [{'templateWarning': {'requestId': '5e92f8cb-75e3-4793-81f8-faee808f01a7', 'installationId': '1ef32602-7319-4d1a-bc65-fb2419c3fe35', 'sessionId': 'eeb5b278-0298-446b-9bcc-43424c2cd44d', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.12', 'samcliVersion': '1.36.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'warningName': 'CodeDeployWarning', 'warningCount': 0}}]}
2021-12-17 11:40:15,017 | Telemetry response: 200
2021-12-17 11:40:15,018 | Sending Telemetry: {'metrics': [{'templateWarning': {'requestId': 'd0f3bfd9-c6d7-40db-9c8b-337bf8efcd98', 'installationId': '1ef32602-7319-4d1a-bc65-fb2419c3fe35', 'sessionId': 'eeb5b278-0298-446b-9bcc-43424c2cd44d', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.12', 'samcliVersion': '1.36.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'warningName': 'CodeDeployConditionWarning', 'warningCount': 0}}]}
2021-12-17 11:40:15,283 | Telemetry response: 200
2021-12-17 11:40:15,284 | Using config file: samconfig.toml, config environment: default
2021-12-17 11:40:15,284 | Expand command line arguments to:
2021-12-17 11:40:15,284 | --template_file=/Users/luhn/Code/audit/test/template.yaml --stack_name=sam-test --dependency_layer --capabilities=('CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND') 

		Managed S3 bucket: aws-sam-cli-managed-default-samclisourcebucket-1aupim17uw7m6

		Default capabilities applied: ('CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND')
To override with customized capabilities, use --capabilities flag or set it in samconfig.toml
2021-12-17 11:40:16,112 | Using build directory as .aws-sam/auto-dependency-layer
2021-12-17 11:40:16,112 | Using build directory as .aws-sam/auto-dependency-layer

This feature is currently in beta. Visit the docs page to learn more about the AWS Beta terms https://aws.amazon.com/service-terms/.

The SAM CLI will use the AWS Lambda, Amazon API Gateway, and AWS StepFunctions APIs to upload your code without 
performing a CloudFormation deployment. This will cause drift in your CloudFormation stack. 
**The sync command should only be used against a development stack**.

Confirm that you are synchronizing a development stack and want to turn on beta features.

Enter Y to proceed with the command, or enter N to cancel:
 [y/N]: 2021-12-17 11:40:17,467 | �[33m
Experimental features are enabled for this session.
Visit the docs page to learn more about the AWS Beta terms https://aws.amazon.com/service-terms/.
�[0m
2021-12-17 11:40:17,477 | No Parameters detected in the template
2021-12-17 11:40:17,499 | 2 stacks found in the template
2021-12-17 11:40:17,499 | No Parameters detected in the template
2021-12-17 11:40:17,510 | 2 resources found in the stack 
2021-12-17 11:40:17,510 | No Parameters detected in the template
2021-12-17 11:40:17,519 | Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/'
2021-12-17 11:40:17,519 | --base-dir is not presented, adjusting uri hello_world/ relative to /Users/luhn/Code/audit/test/template.yaml
2021-12-17 11:40:17,519 | No Parameters detected in the template
2021-12-17 11:40:17,538 | Executing the build using build context.
2021-12-17 11:40:17,538 | Instantiating build definitions
2021-12-17 11:40:17,540 | Same function build definition found, adding function (Previous: BuildDefinition(python3.9, /Users/luhn/Code/audit/test/hello_world, Zip, , d23e058e-cbff-4bce-85b2-09954cf33d29, {}, {}, x86_64, []), Current: BuildDefinition(python3.9, /Users/luhn/Code/audit/test/hello_world, Zip, , 85a07967-200c-4a31-81df-7700103e6ad7, {}, {}, x86_64, []), Function: Function(name='HelloWorldFunction', functionname='HelloWorldFunction', runtime='python3.9', memory=None, timeout=3, handler='app.lambda_handler', imageuri=None, packagetype='Zip', imageconfig=None, codeuri='/Users/luhn/Code/audit/test/hello_world', environment=None, rolearn=None, layers=[], events={'HelloWorld': {'Type': 'Api', 'Properties': {'Path': '/hello', 'Method': 'get', 'RestApiId': 'ServerlessRestApi'}}}, metadata=None, inlinecode=None, codesign_config_arn=None, architectures=['x86_64'], stack_path=''))
2021-12-17 11:40:17,541 | Async execution started
2021-12-17 11:40:17,541 | Invoking function functools.partial(<bound method CachedOrIncrementalBuildStrategyWrapper.build_single_function_definition of <samcli.lib.build.build_strategy.CachedOrIncrementalBuildStrategyWrapper object at 0x1056eb3d0>>, <samcli.lib.build.build_graph.FunctionBuildDefinition object at 0x1053468e0>)
2021-12-17 11:40:17,541 | Running incremental build for runtime python3.9 for build definition d23e058e-cbff-4bce-85b2-09954cf33d29
2021-12-17 11:40:17,541 | Waiting for async results
2021-12-17 11:40:17,541 | Manifest is not changed for d23e058e-cbff-4bce-85b2-09954cf33d29, running incremental build
2021-12-17 11:40:17,541 | Building codeuri: /Users/luhn/Code/audit/test/hello_world runtime: python3.9 metadata: {} architecture: x86_64 functions: ['HelloWorldFunction']
2021-12-17 11:40:17,541 | Building to following folder /Users/luhn/Code/audit/test/.aws-sam/auto-dependency-layer/HelloWorldFunction
2021-12-17 11:40:17,542 | Loading workflow module 'aws_lambda_builders.workflows'
2021-12-17 11:40:17,546 | Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-12-17 11:40:17,548 | Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)'
2021-12-17 11:40:17,549 | Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)'
2021-12-17 11:40:17,551 | Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)'
2021-12-17 11:40:17,553 | Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)'
2021-12-17 11:40:17,555 | Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)'
2021-12-17 11:40:17,556 | Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)'
2021-12-17 11:40:17,558 | Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)'
2021-12-17 11:40:17,559 | Registering workflow 'CustomMakeBuilder' with capability 'Capability(language='provided', dependency_manager=None, application_framework=None)'
2021-12-17 11:40:17,559 | Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)'
2021-12-17 11:40:17,626 | Running workflow 'PythonPipBuilder'
2021-12-17 11:40:17,627 | Running PythonPipBuilder:CopySource
2021-12-17 11:40:17,629 | PythonPipBuilder:CopySource succeeded
2021-12-17 11:40:17,629 | Async execution completed
2021-12-17 11:40:17,630 | Auto creating dependency layer for each function resource into a nested stack
2021-12-17 11:40:17,630 | No Parameters detected in the template
2021-12-17 11:40:17,636 | 2 resources found in the stack sam-test
2021-12-17 11:40:17,636 | No Parameters detected in the template
2021-12-17 11:40:17,641 | Found Serverless function with name='HelloWorldFunction' and CodeUri='.aws-sam/auto-dependency-layer/HelloWorldFunction'
2021-12-17 11:40:17,641 | --base-dir is not presented, adjusting uri .aws-sam/auto-dependency-layer/HelloWorldFunction relative to /Users/luhn/Code/audit/test/template.yaml

Build Succeeded

Built Artifacts  : .aws-sam/auto-dependency-layer
Built Template   : .aws-sam/auto-dependency-layer/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke -t .aws-sam/auto-dependency-layer/template.yaml
[*] Test Function in the Cloud: sam sync --stack-name {stack-name} --watch
[*] Deploy: sam deploy --guided --template-file .aws-sam/auto-dependency-layer/template.yaml
        
2021-12-17 11:40:17,667 | Executing the packaging using package context.
2021-12-17 11:40:18,030 | Unable to export
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/samcli/lib/package/s3_uploader.py", line 114, in upload
    future.result()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/s3transfer/futures.py", line 106, in result
    return self._coordinator.result()
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/s3transfer/futures.py", line 265, in result
    raise self._exception
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/s3transfer/tasks.py", line 126, in __call__
    return self._execute_main(kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/s3transfer/tasks.py", line 150, in _execute_main
    return_value = self._main(**kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/s3transfer/upload.py", line 694, in _main
    client.put_object(Bucket=bucket, Key=key, Body=body, **extra_args)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/botocore/client.py", line 391, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/botocore/client.py", line 719, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/samcli/lib/package/packageable_resources.py", line 126, in export
    self.do_export(resource_id, resource_dict, parent_dir)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/samcli/lib/package/packageable_resources.py", line 148, in do_export
    uploaded_url = upload_local_artifacts(
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/samcli/lib/package/utils.py", line 171, in upload_local_artifacts
    return zip_and_upload(local_path, uploader, extension)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/samcli/lib/package/utils.py", line 189, in zip_and_upload
    return uploader.upload_with_dedup(zip_file, precomputed_md5=md5_hash, extension=extension)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/samcli/lib/package/s3_uploader.py", line 143, in upload_with_dedup
    return self.upload(file_name, remote_path)
  File "/opt/homebrew/Cellar/aws-sam-cli/1.36.0/libexec/lib/python3.8/site-packages/samcli/lib/package/s3_uploader.py", line 121, in upload
    raise NoSuchBucketError(bucket_name=self.bucket_name) from ex
samcli.commands.package.exceptions.NoSuchBucketError: 
S3 Bucket does not exist.
2021-12-17 11:40:18,033 | Sending Telemetry: {'metrics': [{'commandRunExperimental': {'requestId': '2898b15c-f378-4219-b192-da75e8d8e59d', 'installationId': '1ef32602-7319-4d1a-bc65-fb2419c3fe35', 'sessionId': 'eeb5b278-0298-446b-9bcc-43424c2cd44d', 'executionEnvironment': 'CLI', 'ci': False, 'pyversion': '3.8.12', 'samcliVersion': '1.36.0', 'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam sync', 'metricSpecificAttributes': {'experimentalAccelerate': True, 'experimentalAll': False}, 'duration': 3225, 'exitReason': 'ExportFailedError', 'exitCode': 1}}]}
2021-12-17 11:40:18,278 | Telemetry response: 200
Error: Unable to upload artifact HelloWorldFunction referenced by CodeUri parameter of HelloWorldFunction resource.

S3 Bucket does not exist.

Expected result:

I would expect a) sync to honor the settings in samconfig.toml or b) a CLI flag to set the S3 bucket name.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Mac OS Monterey
2. If using SAM CLI, sam --version: SAM CLI, version 1.36.0
3. AWS region: us-west-2

[hoffa]

Thanks for reporting this, will investigate. At glance it would indeed make sense for sync to work in a consistent manner to deploy.

[qingchm]

@luhn Sorry for the inconveniences! The configs for sync command is separate from those of deploy, so the [default.deploy.parameters] samconfig is not relevant to the sync command! To customize the sync parameters please provide the in [default.sync.parameters]! Currently sam sync command is only providing a limited scope of options for customization and unfortunately s3_bucket is not one of them. But for sure we will actively listen to the community for the most desired options to add in and we will try to push through some extra options in the near future! Hope this helps!

[dalmia]

Thanks for clarifying, @qingchm. How do you propose that we go ahead until you're working on adding support for this?

For example, I created a stack using sam-deploy with a custom S3 bucket (say my-bucket). Now, to sync my code changes, I need to use sam sync. But since I cannot use my-bucket anymore, how do you recommend we sync our changes?