[cfnspec] CloudFormation Registry Schemas translation to CFN Specifications is lossy

[rix0rrr]

The issue

This surfaced in #9452. We have seen this before with other teams, but never investigated enough to figure out the root cause.

The problem is that, as AWS teams are moving to the CloudFormation Registry (internal codename more well-known but I'm not going to reproduce that here) they are rewriting their property schemas in the new schema language.

That language is JSON Schema, and is more expressive than the CFN Spec. What seems to be done at the moment is that this schema is being converted back into the CFN spec on a best-effort basis. However, since the upstream schema is more expressive, some things cannot be translated, and so happen to be translated in a way that completely breaks their interpretation in the CFN schema.

Motivating example, ECS' Options type.

It used to be defined as (some details elided):

     "Properties": {
        "Options": {
          "PrimitiveItemType": "String",
          "Type": "Map",
        },
     },

In the new CloudFormation Registry schema language, it is now defined as:

        "Options": {
          "$ref": "#/definitions/Options"
        },

// ...

    "Options": {
      "type": "object",
      "patternProperties": {
        ".{1,}": {
          "type": "string"
        }
      }
    },

Which should be interpreted as "the property Options is of a new type Options, which is a Map<String> containing keys of at least one character.

This cannot be expressed in the CFN Schema for two reasons:

• Map<> types cannot be aliased.
• The constraint that keys should have at least one character cannot be expressed.

The currently generated spec looks like this:

     "Properties": {
        "Options": {
          "Type": "Options",
        },
     },

    "AWS::ECS::TaskDefinition.Options": {
    },

What ends up happening is that we generate the definition of the Options type, but the Options type itself ends up as a record with no fields. CDK then promptly drops all data that used to go there, and breaks the interface definitions.

Solutions

There are a number of ways to address this:

• patching it by hand: will become untenable as more and more teams will move to CloudFormation Registry implementations, and involves a lot of work on our part.
• interpreting empty property types specially: we just assume that types that end up with zero properties were intended to be aliases for Map<>, so we just generate those. We won't be able to derive the element type for the map though, as that information is lost.
• have CloudFormation update their CFN Spec generation logic: it is still true that JSON schema can express more than the CFN Spec can express, but commonly occurring cases such as this could be handled "more correctly" than they are being handled right now.
• start parsing the new schema: we start parsing the new schema ourselves, probably in addition to the old schema. We need to set up tooling to import the new schema and since it's a lot more flexible, also harder to parse (and parse correctly), so a pretty big undertaking.

This is blocking/hampering CFNSpec imports until resolved.

---

This is 🐛 Bug Report

[rix0rrr]

Internal reference: D17415152

[github-actions]

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.