Get value constraints from AWS registry types #1867

kddejong · 2021-01-12T21:33:45Z

Issue #, if available: #1277

Description of changes:

Use the Registry to get AWS property constraints
(in addition to already existing AWS constraints manually maintained and pulled from boto and 3rd party constraints from Registry)

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

PatMyron · 2021-01-12T21:45:11Z

scripts/update_from_registry.py

+    """ main function """
+    configure_logging()
+
+    types = client.list_types(


comment to remember pagination if calling list_types() (+1 to boto/botocore#2177)

These hosted resource provider schemas might be even better to avoid credential requirements and pagination

PatMyron · 2021-01-12T21:47:22Z

scripts/update_from_registry.py

@@ -0,0 +1,180 @@
+#!/usr/bin/env python


let's make sure this is run in #1792 as well

thoughts on just putting this in cfn-lint --update-specs like #1682 if we use the hosted resource provider schemas to avoid new credential requirements? Would be great if updates could be easily picked up without us needing to release

yea good question. Any thoughts on how this would scale and the speed of it? We could download the zips and read them instead of using the APIs (honestly didn't know that existed). Plus it would help with my rate limiting issues.

PatMyron · 2021-01-12T22:00:31Z

src/cfnlint/data/ExtendedSpecs/all/08_registry_value_types.json

@@ -0,0 +1,1883 @@
+[


thoughts on ditching these files + writing directly to CloudSpecs if we make this part of cfn-lint --update-specs?

think it's confusing that ExtendedSpecs/all is partially manually maintained and partially autogenerated

might be useful to include some metadata that the information is coming from the Registry though

agreed. The automation portions can get confusing. Let me see what I can do.

Current values that would need to get corrected in AWS::DataBrew::Job.DatasetName

DatasetName" : { "description" : "Dataset name", "type" : "string", "pattern" : "[ -퟿-�𐀀-\t]*", "minLength" : 1, "maxLength" : 255 },

found all non-ASCII characters in resource schemas (only pattern ones related to this PR):

$ brew install coreutils $ ggrep --color='auto' -P -n "[^\x00-\x7F]" * aws-appflow-connectorprofile.json:158: "description" : "A unique alphanumeric identiﬁer used to authenticate a user, developer, or calling program to your API.", aws-appflow-connectorprofile.json:171: "description" : "A unique alphanumeric identiﬁer used to authenticate a user, developer, or calling program to your API.", aws-appflow-connectorprofile.json:175: "description" : "Application keys, in conjunction with your API key, give you full access to Datadog’s programmatic API. Application keys are associated with the user account that created them. The application key is used to log all requests made to the API.", aws-appflow-connectorprofile.json:215: "description" : "The identiﬁer for the desired client.", aws-appflow-connectorprofile.json:245: "description" : "The identiﬁer for the user.", aws-appflow-connectorprofile.json:273: "description" : "The identiﬁer for the desired client.", aws-appflow-connectorprofile.json:327: "description" : "The object key for the destination bucket in which Amazon AppFlow will place the ﬁles.", aws-appflow-connectorprofile.json:398: "description" : "A unique alphanumeric identiﬁer used to authenticate a user, developer, or calling program to your API.", aws-appflow-connectorprofile.json:408: "description" : "The identiﬁer for the desired client.", aws-appflow-connectorprofile.json:454: "description" : "The name of the Snowﬂake warehouse.", aws-appflow-connectorprofile.json:458: "description" : "The name of the Amazon S3 stage that was created while setting up an Amazon S3 stage in the\nSnowﬂake account. This is written in the following format: < Database>< Schema><Stage Name>.", aws-appflow-connectorprofile.json:462: "description" : "The name of the Amazon S3 bucket associated with Snowﬂake.", aws-appflow-connectorprofile.json:466: "description" : "The bucket prefix that refers to the Amazon S3 bucket associated with Snowﬂake.", aws-appflow-connectorprofile.json:470: "description" : "The Snowﬂake Private Link service name to be used for private data transfers.", aws-appflow-connectorprofile.json:478: "description" : "The region of the Snowﬂake account.", aws-appflow-connectorprofile.json:522: "description" : "The identiﬁer for the desired client.", aws-databrew-job.json:9: "pattern" : "[ -퟿-�𐀀-�\t]*", aws-databrew-job.json:27: "pattern" : "[ -퟿-�𐀀-�\t]*", aws-databrew-job.json:72: "pattern" : "[ -퟿-�𐀀-�\t]*", aws-databrew-recipe.json:8: "pattern" : "[ -퟿-�𐀀-�\t]*", aws-databrew-recipe.json:16: "pattern" : "^[ -.0-^`-퟿-�]*$", aws-databrew-schedule.json:37: "pattern" : "[ -퟿-�𐀀-�\t]*", aws-gamelift-gameservergroup.json:244: "pattern" : "[ -퟿-�𐀀-�\r\n\t]*", aws-kinesis-stream.json:26: "description" : "An arbitrary set of tags (key–value pairs) to associate with the Kinesis stream.", aws-kinesis-stream.json:73: "description" : "An arbitrary set of tags (key–value pairs) to associate with the Kinesis stream.", aws-mediapackage-packagingconfiguration.json:100: "description" : "This setting controls how ad markers are included in the packaged OriginEndpoint. “NONE” will omit all SCTE-35 ad markers from the output. “PASSTHROUGH” causes the manifest to contain a copy of the SCTE-35 ad markers (comments) taken directly from the input HTTP Live Streaming (HLS) manifest. “SCTE35_ENHANCED” generates ad markers and blackout tags based on SCTE-35 messages in the input source.", aws-mediapackage-packagingconfiguration.json:141: "description" : "The Dynamic Adaptive Streaming over HTTP (DASH) profile type. When set to “HBBTV_1_5”, HbbTV 1.5 compliant output is enabled.", aws-mediapackage-packagingconfiguration.json:217: "description" : "A list of triggers that controls when the outgoing Dynamic Adaptive Streaming over HTTP (DASH) Media Presentation Description (MPD) will be partitioned into multiple periods. If empty, the content will not be partitioned into more than one period. If the list contains “ADS”, new periods will be created where the Asset contains SCTE-35 ad markers.",

https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-appflow/pull/6#discussion_r562217193
aws-cloudformation/aws-cloudformation-resource-providers-databrew#2 (comment)
https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-gamelift/pull/13#discussion_r562220060
https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-kinesis/pull/6#discussion_r562226794
https://github.com/aws-cloudformation/aws-cloudformation-resource-providers-mediapackage/pull/3#discussion_r562228692
(in TODO list in aws-cloudformation/cloudformation-cli#663 / aws-cloudformation/cloudformation-cli#459)

PatMyron · 2021-01-12T22:08:30Z

Noting constraints applied last overwrite any already existing constraints (seems okay, just noteworthy)

PatMyron · 2021-01-12T22:13:47Z

Do you know how this handles combined JSON schemas? Fine to not handle them yet since there are so few resource provider schemas using them so far, just want to make sure they're at least not handled incorrectly:

grep -l -E -i '(oneOf|anyOf|allOf)' *
aws-applicationinsights-application.json
aws-cloudformation-moduledefaultversion.json
aws-cloudformation-stackset.json
aws-cloudwatch-metricstream.json
aws-databrew-dataset.json
aws-databrew-recipe.json
aws-gamelift-alias.json
aws-iotwireless-wirelessdevice.json
aws-kendra-datasource.json
aws-rds-globalcluster.json
aws-s3-storagelens.json
aws-sagemaker-pipeline.json
aws-synthetics-canary.json

scripts/update_from_registry.py

PatMyron · 2021-01-12T22:55:28Z

This is awesome btw, this alone is pulling in significantly more constraints:

AllowedValues: 315 new
NumberMin/NumberMax: 49 new
AllowedPatternRegex: 773 new
StringMin/StringMax: 662 new

than we had before

PatMyron · 2021-01-13T00:08:57Z

src/cfnlint/data/ExtendedSpecs/all/08_registry_value_types.json

+    "op": "add",
+    "path": "/ValueTypes/AWS::SageMaker::ModelBiasJobDefinition.EndpointInput.EndTimeOffset",
+    "value": {
+      "Allowed


Similar to aws-cloudformation/cloudformation-cli#459, we need to think about how to verify regular expressions are valid in Java and Python
(some don't seem to be valid in either though 🤔)

Switched to the regex package that supports unicode. That helps bring Python closer to Java but there are still some expressions that are not compliant. Example: ^[a-zA-Z0-9-]+{1,255}$

ERROR: Failed building wheel for regex

src/cfnlint/rules/resources/ectwo/Ebs.py

kddejong · 2021-02-11T21:27:10Z

This is going to add a little time to build update-specs process because the amount of jsonpatches. I think the better long term solution is to separate out our spec update process into another repo that we can build and release them.

jlongtine · 2021-02-11T21:49:24Z

This is going to add a little time to build update-specs process because the amount of jsonpatches. I think the better long term solution is to separate out our spec update process into another repo that we can build and release them.

I like this idea. Separating this from the python codebase would help me with cfn-cue.

* Get value constraints from registry types * Eliminate unicode strings from being allowed for patterns

PatMyron

minItems/maxItems could be translated to ListMin/ListMax as well

currently 468 usages of those 2 keywords in AWS resource schemas:

CloudformationSchema $ grep -Eho '"m\w\wItems"' * | sort | uniq -c | sort -nr
 257 "maxItems"
 211 "minItems"

PatMyron · 2021-05-13T20:37:03Z

src/cfnlint/maintenance.py

+                        results['.'.join(names + [propname])] = {}
+                if propdetails.get('pattern'):
+                    p = propdetails.get('pattern')
+                    if '.'.join(names + [propname]) == 'AWS::OpsWorksCM::Server.CustomPrivateKey':


linking pattern for reference

PatMyron · 2021-05-13T20:44:24Z

src/cfnlint/maintenance.py

+                if propdetails.get('enum'):
+                    results['.'.join(names + [propname])].update({
+                        'AllowedValues': propdetails.get('enum')
+                    })


Could grab enums for types other than string as well

Especially since we seem to already be handling more types in the rule itself:
https://github.com/aws-cloudformation/cfn-python-lint/blob/853522ff0e4c977331ac677a47059cf96b6f3189/src/cfnlint/rules/resources/properties/AllowedValue.py#L38

PatMyron · 2021-05-13T20:47:34Z

src/cfnlint/maintenance.py

+                        'NumberMin': propdetails.get('minimum'),
+                        'NumberMax': propdetails.get('maximum'),


Could use exclusiveMinimum/exclusiveMaximum as well

Although no AWS resource schemas use these keywords yet anyway

Could even just set NumberMin/NumberMax for these for now even though those are inclusive since that'd still be better than not supporting it at all

kddejong force-pushed the feature/valuecontraintsfromregistry branch from d453491 to 22a86e1 Compare January 12, 2021 21:35

PatMyron reviewed Jan 12, 2021

View reviewed changes

scripts/update_from_registry.py Outdated Show resolved Hide resolved

kddejong force-pushed the feature/valuecontraintsfromregistry branch from 22a86e1 to 86a6cf0 Compare January 12, 2021 22:26

PatMyron reviewed Jan 12, 2021

View reviewed changes

scripts/update_from_registry.py Outdated Show resolved Hide resolved

PatMyron reviewed Jan 13, 2021

View reviewed changes

PatMyron changed the title ~~Get value constraints from registry types~~ Get value constraints from AWS registry types Jan 13, 2021

kddejong force-pushed the feature/valuecontraintsfromregistry branch 2 times, most recently from 1d8726a to 265a207 Compare January 14, 2021 18:04

PatMyron reviewed Jan 14, 2021

View reviewed changes

src/cfnlint/rules/resources/ectwo/Ebs.py Outdated Show resolved Hide resolved

This was referenced Jan 15, 2021

Onboard AWS Glue DataBrew Resources to CloudFormation. aws-cloudformation/aws-cloudformation-resource-providers-databrew#2

Merged

catch common resource schema issues in cfn validate aws-cloudformation/cloudformation-cli#663

Merged

kddejong force-pushed the feature/valuecontraintsfromregistry branch 4 times, most recently from 2c70c72 to 30dd5c2 Compare January 27, 2021 22:34

PatMyron mentioned this pull request Jan 28, 2021

catch common resource schema issues in cfn validate aws-cloudformation/cloudformation-cli#675

Merged

kddejong force-pushed the feature/valuecontraintsfromregistry branch 9 times, most recently from 80a511f to d4fd6f8 Compare January 28, 2021 20:24

kddejong added 6 commits February 10, 2021 09:25

reload specs based on what is compliant with the new regex package

873b81a

revert back to the re package

c49af17

Switch to add registry schema as part of udpate-specs

5cea4f0

fix linting issues

76ab29b

update tests for registy types

eb0939f

elimate unicode strings from being allowed for patterns

46ce4d9

kddejong force-pushed the feature/valuecontraintsfromregistry branch from fe4d594 to 1a3e7f3 Compare February 11, 2021 21:25

Fixing a few issues

776f7fb

kddejong force-pushed the feature/valuecontraintsfromregistry branch from 1a3e7f3 to 776f7fb Compare February 11, 2021 22:22

kddejong merged commit df9b214 into aws-cloudformation:master Feb 11, 2021

kddejong deleted the feature/valuecontraintsfromregistry branch February 11, 2021 22:43

PatMyron mentioned this pull request Feb 15, 2021

Faster --update-specs #1249

Closed

PatMyron mentioned this pull request Mar 11, 2021

CloudFormation Registry resource schema support (--registry-schemas) #1732

Merged

16 tasks

PatMyron mentioned this pull request Mar 12, 2021

Adding AnomalyMonitor and AnomalySubscription resources aws-cloudformation/aws-cloudformation-resource-providers-cost-explorer#8

Merged

direvus pushed a commit to direvus/cfn-python-lint that referenced this pull request Apr 12, 2021

Get value constraints from AWS registry types (aws-cloudformation#1867)

eb21272

* Get value constraints from registry types * Eliminate unicode strings from being allowed for patterns

PatMyron reviewed May 13, 2021

View reviewed changes

PatMyron mentioned this pull request Jul 12, 2021

cfn-lint -u fails with "FutureWarning: Possible nested set at position 7 re.compile(p, re.UNICODE)" #2077

Closed

PatMyron mentioned this pull request Dec 22, 2021

troposphere 3.0 - futures cloudtools/troposphere#1558

Open

PatMyron mentioned this pull request Jan 7, 2022

mapping aws_acmpca_certificate and aws_acmpca_certificate_authority_certificate terraform-linters/tflint-ruleset-aws#218

Merged

PatMyron mentioned this pull request Feb 22, 2022

Tag values are not validated against permitted character regex #2219

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get value constraints from AWS registry types #1867

Get value constraints from AWS registry types #1867

kddejong commented Jan 12, 2021 •

edited by PatMyron

Loading

PatMyron Jan 12, 2021 •

edited

Loading

PatMyron Jan 12, 2021

PatMyron Jan 12, 2021 •

edited

Loading

PatMyron Jan 12, 2021 •

edited

Loading

kddejong Jan 12, 2021

PatMyron Jan 12, 2021 •

edited

Loading

kddejong Jan 13, 2021

kddejong Jan 15, 2021

PatMyron Jan 21, 2021 •

edited

Loading

PatMyron commented Jan 12, 2021 •

edited

Loading

PatMyron commented Jan 12, 2021 •

edited

Loading

PatMyron commented Jan 12, 2021 •

edited

Loading

PatMyron Jan 13, 2021 •

edited

Loading

kddejong Jan 13, 2021 •

edited

Loading

PatMyron Jan 13, 2021

kddejong commented Feb 11, 2021

jlongtine commented Feb 11, 2021

PatMyron left a comment •

edited

Loading

PatMyron May 13, 2021 •

edited

Loading

PatMyron May 13, 2021

PatMyron May 13, 2021 •

edited

Loading

		'NumberMin': propdetails.get('minimum'),
		'NumberMax': propdetails.get('maximum'),

Get value constraints from AWS registry types #1867

Get value constraints from AWS registry types #1867

Conversation

kddejong commented Jan 12, 2021 • edited by PatMyron Loading

PatMyron Jan 12, 2021 • edited Loading

Choose a reason for hiding this comment

PatMyron Jan 12, 2021

Choose a reason for hiding this comment

PatMyron Jan 12, 2021 • edited Loading

Choose a reason for hiding this comment

PatMyron Jan 12, 2021 • edited Loading

Choose a reason for hiding this comment

kddejong Jan 12, 2021

Choose a reason for hiding this comment

PatMyron Jan 12, 2021 • edited Loading

Choose a reason for hiding this comment

kddejong Jan 13, 2021

Choose a reason for hiding this comment

kddejong Jan 15, 2021

Choose a reason for hiding this comment

PatMyron Jan 21, 2021 • edited Loading

Choose a reason for hiding this comment

PatMyron commented Jan 12, 2021 • edited Loading

PatMyron commented Jan 12, 2021 • edited Loading

PatMyron commented Jan 12, 2021 • edited Loading

PatMyron Jan 13, 2021 • edited Loading

Choose a reason for hiding this comment

kddejong Jan 13, 2021 • edited Loading

Choose a reason for hiding this comment

PatMyron Jan 13, 2021

Choose a reason for hiding this comment

kddejong commented Feb 11, 2021

jlongtine commented Feb 11, 2021

PatMyron left a comment • edited Loading

Choose a reason for hiding this comment

PatMyron May 13, 2021 • edited Loading

Choose a reason for hiding this comment

PatMyron May 13, 2021

Choose a reason for hiding this comment

PatMyron May 13, 2021 • edited Loading

Choose a reason for hiding this comment

kddejong commented Jan 12, 2021 •

edited by PatMyron

Loading

PatMyron Jan 12, 2021 •

edited

Loading

PatMyron Jan 12, 2021 •

edited

Loading

PatMyron Jan 12, 2021 •

edited

Loading

PatMyron Jan 12, 2021 •

edited

Loading

PatMyron Jan 21, 2021 •

edited

Loading

PatMyron commented Jan 12, 2021 •

edited

Loading

PatMyron commented Jan 12, 2021 •

edited

Loading

PatMyron commented Jan 12, 2021 •

edited

Loading

PatMyron Jan 13, 2021 •

edited

Loading

kddejong Jan 13, 2021 •

edited

Loading

PatMyron left a comment •

edited

Loading

PatMyron May 13, 2021 •

edited

Loading

PatMyron May 13, 2021 •

edited

Loading