troposphere 3.0 - futures

[markpeek]

Wanted to start an issue to discuss goals for troposphere 3.0:

• All core code gets generated from the resource specs
• Python 3+ (remove the EOL python2 support)
• Restructure in a way to allow intellisense to work with properties to make coding easier
• Maintain core backward compatibility

Implications:

• Validators and other non-generated code will be merged in (likely from separate files using object mixins)
• Align/rename with the AWS resource spec naming where possible
• Likely will have some breaking changes due to name changes (hence a major release)

[benbridts]

I would also consider using the CloudFormation Repository instead of the resource specification.

I'd also consider using the json booleans as a default instead of the 'true' and 'false' strings, so the output can be validated with cfn-lint and doesn't throw errors with both tools in their default configuration.

[markpeek]

Thanks @ikben. Can you provide a link to the CloudFormation Repository? Not sure I know that one.

Agreed on your other point:

• Switch to json booleans - remove TROPO_REAL_BOOL backward compatibility hack

[benbridts]

Documentation link: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/registry.html

For troposphere the interesting part is the type schema. You can develop your own, but the schema of all AWS resources is also available via the API: https://docs.aws.amazon.com/cloudformation-cli/latest/userguide/resource-type-schema.html or download publicly accessible versions: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/resource-type-schemas.html
full schema: https://github.com/aws-cloudformation/cloudformation-cli/blob/master/src/rpdk/core/data/schema/provider.definition.schema.v1.json

This is also the schema that is used for third party resources in AWS Config: https://aws.amazon.com/about-aws/whats-new/2019/11/aws-config-launches-support-third-party-resources/

I'd expect there would be more data that can be parsed from the registry than from the current schema.

[ns-cweber]

I would also suggest generating type annotations. Perhaps this is implied in the better intellisense support objective, but I think it's a worthy goal on its own for the growing mypy user community.

[MacHu-GWU]

Hi @markpeek

I have experienced in using code to generate code from AWS official Cloudformation documentation source

@ns-cweber Actually I had a personal project troposphere_mate-project that generate type hint from the troposphere source code, and enables property name hint, type hint. The idea is simple. Just use jinja2 template and use importlib recursively grab all AWSObject and its Properties. It is very easy if you see the code template class template and module template

I think I can do the same things that automatically generates code from aws docs. I can help with this part.

For "Restructure in a way to allow intellisense to work with properties to make coding easier", I have a lot of ideas and implemented in troposphere_mate-project, such as:

• Batch Tagging
• Auto Resource Inter-Reference
• Return Values Hint
• Remove Resource and auto-remove dependent resource and Output
• Partial Deployment
• Deploy from Python support nested stack one line deployment
• Reuse template via subclassing (I call it canned template).

Is there anyone you feel like could benefit the developer in productivity or maintainability?

Thank you

[0x4d4e]

Disclaimer: I'm a complete noob regarding trophosphere and InGraph, I just stumbled across this issue after playing around with both while attempting to re-build existing AWS infrastructure.

Maybe it's worth taking a look at InGraph for the generation of Objects based on the AWS specs? -> https://github.com/lifadev/ingraph-aws

Another thing that stuck out from playing around with InGraph was the use of Cidr for creating multiple subnet ranges for a VPC. In troposphere you more or less mirror the CF syntax in Python by using the Select and Cidr functions, in InGraph it's more 'pythonic' and can be accessed as a Python list that also 'hides' the use of Select.

Select(1, Cidr(VPC_CIDR, 3, 8)) vs cidr(block=vpc.CidrBlock, count=3, bits=8)[1]

[PatMyron]

• Validators and other non-generated code will be merged in (likely from separate files using object mixins)

Validators like AllowedValues lists should no longer be manually maintained:
instance types from pricing
others from botocore

Most property validators could be shared between multiple projects:
cfnlint/helpers.py
aws-cloudformation/cfn-lint#1867

Would focusing validators in cfn-lint and simply having Troposphere run cfn-lint make sense?

[michael-k]

I'd say the scope mentioned in the initial comment it to big. Why not target Python 3.6+ only with Troposphere 3.0. And then try to do as many of the other items in 3.1, 3.2, …

All core code gets generated from the resource specs

I haven't looked at the resource specs. But since no effort in this direction was successful so far, maybe it's better to start with generating some code from the specs and then go from there.

• Align/rename with the AWS resource spec naming where possible
• Likely will have some breaking changes due to name changes (hence a major release)

It's probably possible to support both names. The old ones should be properly deprecated and removed in Troposphere 4.0.

---

Dropping support for Python 2 in Troposphere would also make it easier to drop support for Python 2 in awacs. And it would make it easier for people to run the tests locally (no need to ensure that 2to3 was called).