Cache is invalid when using containers environment variables

[wnkz]

Description:

Cache is always invalid when building with containers environment variables (-e, --container-env-var)

Steps to reproduce:

• Create a any sample app with sam init
• Build multiple times with sam build -cu -e FOO=BAR

Observed result:

• "Cache is invalid" message
• Function's hash is changing

❯ sam build -cu -e FOO=BAR
Starting Build use cache
Starting Build inside a container
Cache is invalid, running build and copying resources to function build definition of b89956ad-7a89-401a-98b5-68dad76c9c2d
Building codeuri: /private/var/folders/bf/f_z6y6n959193x3csh_pcb0m0000gn/T/tmp.Zjp9bgzU/sam-app/hello_world runtime: python3.8 metadata: {} functions: ['HelloWorldFunction']

Fetching public.ecr.aws/sam/build-python3.8:latest Docker container image......
Mounting /private/var/folders/bf/f_z6y6n959193x3csh_pcb0m0000gn/T/tmp.Zjp9bgzU/sam-app/hello_world as /tmp/samcli/source:ro,delegated inside runtime container

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
❯ sam build -cu -e FOO=BAR
Starting Build use cache
Starting Build inside a container
Cache is invalid, running build and copying resources to function build definition of fb423804-b45f-431a-9eca-e5d58909cbe6
Building codeuri: /private/var/folders/bf/f_z6y6n959193x3csh_pcb0m0000gn/T/tmp.Zjp9bgzU/sam-app/hello_world runtime: python3.8 metadata: {} functions: ['HelloWorldFunction']

Fetching public.ecr.aws/sam/build-python3.8:latest Docker container image......
Mounting /private/var/folders/bf/f_z6y6n959193x3csh_pcb0m0000gn/T/tmp.Zjp9bgzU/sam-app/hello_world as /tmp/samcli/source:ro,delegated inside runtime container

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource

Expected result:

"Valid cache found" message on second run

❯ sam build -cu
Starting Build use cache
Starting Build inside a container
Cache is invalid, running build and copying resources to function build definition of 72cbac78-847c-4b54-9953-5fe0d300b926
Building codeuri: /private/var/folders/bf/f_z6y6n959193x3csh_pcb0m0000gn/T/tmp.Zjp9bgzU/sam-app/hello_world runtime: python3.8 metadata: {} functions: ['HelloWorldFunction']

Fetching public.ecr.aws/sam/build-python3.8:latest Docker container image......
Mounting /private/var/folders/bf/f_z6y6n959193x3csh_pcb0m0000gn/T/tmp.Zjp9bgzU/sam-app/hello_world as /tmp/samcli/source:ro,delegated inside runtime container

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

Running PythonPipBuilder:ResolveDependencies
Running PythonPipBuilder:CopySource
❯ sam build -cu
Starting Build use cache
Starting Build inside a container
Valid cache found, copying previously built resources from function build definition of 72cbac78-847c-4b54-9953-5fe0d300b926

Build Succeeded

Built Artifacts  : .aws-sam/build
Built Template   : .aws-sam/build/template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Deploy: sam deploy --guided

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: macOS 11.3.1
2. sam --version: 1.23.0
3. AWS region: eu-west-1

debug.log

[wnkz]

This is very painful for "large" projects ; any plans / timeline on a fix?

[wnkz]

Tracked it down a bit more, it appears that when building with -e, those environment variables are stored in the build.toml file. On later builds, they are read again as if they where part of the FunctionBuildDefinition so the function is treated as unique because the real definition (template.yml) does not have those.

aws-sam-cli/samcli/lib/build/build_graph.py

Lines 189 to 207 in 199ded1

	if function_build_definition in self._function_build_definitions:
	previous_build_definition = self._function_build_definitions[
	self._function_build_definitions.index(function_build_definition)
	]
	LOG.debug(
	"Same function build definition found, adding function (Previous: %s, Current: %s, Function: %s)",
	previous_build_definition,
	function_build_definition,
	function,
	)
	previous_build_definition.add_function(function)
	else:
	LOG.debug(
	"Unique function build definition found, adding as new (Function Build Definition: %s, Function: %s)",
	function_build_definition,
	function,
	)
	function_build_definition.add_function(function)
	self._function_build_definitions.append(function_build_definition)

aws-sam-cli/samcli/lib/build/build_graph.py

Lines 442 to 469 in 199ded1

	def __eq__(self, other: Any) -> bool:
	"""
	Checks equality of the function build definition
	Parameters
	----------
	other: Any
	other function build definition to compare
	Returns
	-------
	bool
	True if both function build definitions has same following properties, False otherwise
	"""
	if not isinstance(other, FunctionBuildDefinition):
	return False
	# each build with custom Makefile definition should be handled separately
	if self.metadata and self.metadata.get("BuildMethod", None) == "makefile":
	return False
	return (
	self.runtime == other.runtime
	and self.codeuri == other.codeuri
	and self.packagetype == other.packagetype
	and self.metadata == other.metadata
	and self.env_vars == other.env_vars
	)

[mndeveci]

Thanks for reporting and thorough analysis @wnkz !

Indeed it is coming from invalid cache definition because our builders are adding additional environment variables before building and that was causing cache to be invalid all the time.

I raised the PR for the fix, and this should be resolved within next release after the PR is merged.

Thanks!

[qingchm]

This fix is now released with our newest version 1.25.0! Please update the CLI version to use cached build with container env vars! Closing this ticket, let me know if you have any questions ;)

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.