Sam build fails when template contains multiple resources using !If with !Ref AWS::NoValue

[danludwig]

Description

Sam build fails when more than one DynamoDB resource uses conditional !If intrinsic functions combined with !Ref AWS::NoValue

Steps to reproduce

1. Install aws sam cli version 0.21.0
2. git clone https://github.com/danludwig/aws-sam-cli-bug-reports.git
3. cd ./v0.21.0/two-dynamos-with-intrinsic-tags
4. sam build or sam build --use-container
5. Examine error stack trace

More info is available in the README.md file @ https://github.com/danludwig/aws-sam-cli-bug-reports/tree/master/v0.21.0/two-dynamos-with-intrinsic-tags

Observed result

sam build --debug
Using SAM Template at C:\Users\Dan\Code\sam-inits\aws-sam-cli-bug-reports\v0.21.0\two-dynamos-with-intrinsic-tags\template.yaml
Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics
'build' command is called
Collected default values for parameters: {'ProjectName': 'my-project', 'DataBackups': 'disabled'}
Unable to process properties of DynamoDbTableOne.AWS::DynamoDB::Table
Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 27, 'exitReason': 'AttributeError', 'exitCode': 255, 'requestId': '62f9ecb3-cc92-44cb-b345-6b9eb6793a46', 'installationId': '250b0f37-320c-4833-8754-63acdd61926a', 'sessionId': '6b27a339-2716-40e7-88ff-8827f832b039', 'executionEnvironment': 'CLI', 'pyversion': '3.6.7', 'samcliVersion': '0.21.0'}}]}
HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1)
Traceback (most recent call last):
  File "runpy.py", line 193, in _run_module_as_main
  File "runpy.py", line 85, in _run_code
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\__main__.py", line 12, in <module>
    cli(prog_name="sam")
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 717, in main
    rv = self.invoke(ctx)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metrics.py", line 94, in wrapped
    raise exception  # pylint: disable=raising-bad-type
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metrics.py", line 65, in wrapped
    return_value = func(*args, **kwargs)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 105, in cli
    skip_pull_image, parameter_overrides, mode)  # pragma: no cover
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 138, in do_cli
    mode=mode) as ctx:
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\build_context.py", line 65, in __enter__
    self._function_provider = SamFunctionProvider(self._template_dict, self._parameter_overrides)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\local\lib\sam_function_provider.py", line 46, in __init__
    self.template_dict = SamBaseProvider.get_template(template_dict, parameter_overrides)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\local\lib\sam_base_provider.py", line 53, in get_template
    template_dict = resolver.resolve_template(ignore_errors=True)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsic_property_resolver.py", line 240, in resolve_template
    processed_template["Resources"] = self.resolve_attribute(self._resources, ignore_errors)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsic_property_resolver.py", line 265, in resolve_attribute
    processed_resource = self.intrinsic_property_resolver(val, parent_function=processed_key)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsic_property_resolver.py", line 220, in intrinsic_property_resolver
    sanitized_val = self.intrinsic_property_resolver(val, parent_function=parent_function)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsic_property_resolver.py", line 220, in intrinsic_property_resolver
    sanitized_val = self.intrinsic_property_resolver(val, parent_function=parent_function)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsic_property_resolver.py", line 199, in intrinsic_property_resolver
    return [self.intrinsic_property_resolver(item) for item in intrinsic]
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsic_property_resolver.py", line 199, in <listcomp>
    return [self.intrinsic_property_resolver(item) for item in intrinsic]
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsic_property_resolver.py", line 213, in intrinsic_property_resolver
    return self.conditional_key_function_map.get(key)(intrinsic_value)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsic_property_resolver.py", line 753, in handle_fn_if
    intrinsic_value, parent_function=IntrinsicResolver.FN_IF
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsic_property_resolver.py", line 199, in intrinsic_property_resolver
    return [self.intrinsic_property_resolver(item) for item in intrinsic]
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsic_property_resolver.py", line 199, in <listcomp>
    return [self.intrinsic_property_resolver(item) for item in intrinsic]
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsic_property_resolver.py", line 210, in intrinsic_property_resolver
    return self.intrinsic_key_function_map.get(key)(intrinsic_value)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsic_property_resolver.py", line 659, in handle_fn_ref
    return self._symbol_resolver.resolve_symbols(arguments, IntrinsicResolver.REF)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsics_symbol_table.py", line 217, in resolve_symbols
    translated = self.get_translation(logical_id, resource_attribute)
  File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\intrinsic_resolver\intrinsics_symbol_table.py", line 323, in get_translation
    return logical_id_item.get(resource_attributes)
AttributeError: 'NoneType' object has no attribute 'get'

Expected result

$ sam build
Building resource 'HelloWorldFunction'
Running NodejsNpmBuilder:NpmPack
Running NodejsNpmBuilder:CopyNpmrc
Running NodejsNpmBuilder:CopySource
Running NodejsNpmBuilder:NpmInstall
Running NodejsNpmBuilder:CleanUpNpmrc

Build Succeeded

Built Artifacts  : .aws-sam\build
Built Template   : .aws-sam\build\template.yaml

Commands you can use next
=========================
[*] Invoke Function: sam local invoke
[*] Package: sam package --s3-bucket <yourbucket>

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10
2. sam --version: 0.21.0

[danludwig]

Some additional info copied over from the reproducible example code:

The bug does not seem to surface when the stack template only contains a single DynamoDB resource. For example, commenting out either table from the template will result in a successful build.

Furthermore, commenting out the sections which use the !If / !Ref AWS::NoValue blocks results in a successful build with both DynamoDB resources. For example, the following will result in a successful build:

  DynamoDbTableOne:
    Type: AWS::DynamoDB::Table
    Properties:
      # Tags:
      #   - Key: Project
      #     Value: !Ref ProjectName
      #   - !If
      #     - CreateBackups
      #     - Key: data:backup
      #       Value: ""
      #     - !Ref AWS::NoValue
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - KeyType: HASH
          AttributeName: id
      BillingMode: PAY_PER_REQUEST

  DynamoDbTableTwo:
    Type: AWS::DynamoDB::Table
    Properties:
      # Tags:
      #   - Key: Project
      #     Value: !Ref ProjectName
      #   - !If
      #     - CreateBackups
      #     - Key: data:backup
      #       Value: ""
      #     - !Ref AWS::NoValue
      AttributeDefinitions:
        - AttributeName: id
          AttributeType: S
      KeySchema:
        - KeyType: HASH
          AttributeName: id
      BillingMode: PAY_PER_REQUEST

[danludwig]

@jfuss no love for this issue? or is it possibly another duplicate like #1407 and #1408?

[sriram-mv]

@danludwig I started working on this issue with #1409 , But it does not go out right delete the property if no value was encountered, this just stops the exception. The PR needs some more work.

[danludwig]

@thesriram what PR needs some more work? #1410 ..?

[danludwig]

Ok I see, #1409 was the pull request... I thought that was a link to another issue.

[jfuss]

@danludwig I am not able to reproduce the failure anymore in v0.22.0. Can you upgrade and verify also? I put one of the DynamoDB tables with AWS::NoValue into a template and able to invoke the function. We are still not doing the right thing with AWS::NoValue but at least shouldn't break the CLI anymore.

[danludwig]

@jfuss verified, the latest version seems to build fine with the template where we have these 2 dynamo tables with conditional properties.

[sriram-mv]

Closing this issue, as we are doing a best effort intrinsic resolution and it doesnt break the CLI anymore.