Faster --update-specs

[ghost]

cfn-lint version: 0.25.5

Right now, cfn-lint --update-specs doesn't really do any checking to see if the existing specs are up to date and naively patches all of them. It would be beneficial to have some sort of local checksum which would prevent this, to increase overall speed when there are no updates.

[PatMyron]

using the zipped resource specifications should've helped a bit, but there's definitely more we could do. Will also look into updating the resource specifications in parallel across regions here:

https://github.com/aws-cloudformation/cfn-python-lint/blob/52864add9eac85a42cc6866cdf05c4d3b4dcd149/src/cfnlint/maintenance.py#L50

[kddejong]

@PatMyron do you know if there are any hashes for those specs that we could use to compare before downloading?

[PatMyron]

@PatMyron do you know if there are any hashes for those specs that we could use to compare before downloading?

aws s3api head-object provides "An ETag is an opaque identifier assigned by a web server to a specific version of a resource found at a URL.", but I'd rather not require credentials

[ghost]

Honestly, I was thinking that there would be a single file that the updater would download first that lists the versions of each spec and checksums.

[kddejong]

I'm curios @clar-cmp and @Tro95. Are you doing the --update-specs as part of CI/CD? Are you testing your templates across all regions? The other option for speeding this up even more may be to use the region parameter with the --update-specs From my basic testing this PR does a pretty good job speeding things up thought... so just curious.

[Tro95]

I am not using cfn-lint as part of any CI/CD flow, and nor do I necessarily need to check if my templates work across all regions. My main use cases are for assisting manual dev effort, and ensuring basic compatibility with all of the eu-* regions and us-east-1.

For context, slow updates don't cause me any pain. I worked on this PR simply because it was something to do on a weekend, and it seemed more interesting than writing a couple of additional rules.

[PatMyron]

#1682 / #1867 will slow down --update-specs by a minute or two again sometimes in order to update much more information

this should currently be infrequent since nothing is currently updated if there are not newer Resource Specifications

[PatMyron]

Not run by customers like --update-specs, but I'd love for scripts/update_specs_from_pricing.py to be faster too