adding Los Angeles, ElastiCache CacheNodeType, Elasticsearch InstanceType, and fixing Neptune InstanceClass pricing information #1535

PatMyron · 2020-05-17T20:08:37Z

Offer files for reference: (jq 'del(.terms)' helps for larger files)
https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/index.json
AmazonEC2 (3GB+)
AmazonRedshift
AmazonDAX
AmazonMQ
AmazonRDS
AmazonNeptune
AmazonDocDB
AmazonElastiCache
AmazonES

unrelated to this change, m6g instance types launched this week

moved US West (Los Angeles) from region_exceptions to region_map for us-west-2

which adds i3en.metal-2tb EC2 instance type to src/cfnlint/data/ExtendedSpecs/us-west-2/05_pricing_property_values.json and 37 lines to src/cfnlint/data/AdditionalSpecs/RdsProperties.json

switched AmazonNeptune from parsing usagetype to using instanceType which appends arge to db.r5.\d*xl, which seems correct:

Resources:
  DBCluster:
    Type: AWS::Neptune::DBCluster
  DBInstance:
    Type: AWS::Neptune::DBInstance
    Properties:
      DBInstanceClass: db.r5.xl
      DBClusterIdentifier: !Ref DBCluster

Invalid DB Instance class: db.r5.xl (Service: AmazonNeptune; Status Code: 400; Error Code: InvalidParameterValue)

Resources:
  DBCluster:
    Type: AWS::Neptune::DBCluster
  DBInstance:
    Type: AWS::Neptune::DBInstance
    Properties:
      DBInstanceClass: db.r5.xlarge
      DBClusterIdentifier: !Ref DBCluster

added AmazonElastiCache information for:

AWS::ElastiCache::CacheCluster.CacheNodeType

AWS::ElastiCache::ReplicationGroup.CacheNodeType

added AmazonES information for:

AWS::Elasticsearch::Domain.ElasticsearchClusterConfig.InstanceType

how to test:

brew install github/gh/gh # https://github.com/cli/cli
gh pr checkout 1535
pip3 install -e .
scripts/update_specs_from_pricing.py # requires Boto3 and Credentials, takes forever
cfn-lint --update-specs # https://github.com/aws-cloudformation/cfn-python-lint/pull/1383#issuecomment-629891506

Resources:
  Domain:
    Type: AWS::Elasticsearch::Domain
    Properties:
      ElasticsearchClusterConfig:
        InstanceType: r3.large.elasticsearch # change this for E3030 errors 🎉
  CacheCluster:
    Type: AWS::ElastiCache::CacheCluster
    Properties:
      NumCacheNodes: 1
      Engine: redis
      CacheNodeType: CacheNodeType
  ReplicationGroup:
    Type: AWS::ElastiCache::ReplicationGroup
    Properties:
      ReplicationGroupDescription: ReplicationGroupDescription
      CacheNodeType: CacheNodeType

future work:

more services:
~~nodetype~~

instanceclass:

AWSDatabaseMigrationSvc 'Replication Server' for AWS::DMS::ReplicationInstance.ReplicationInstanceClass (dms. prefix missing from instanceType, could parse usagetype)

instancetype:

~~ElasticMapReduce, ManagedBlockchain, GameLift, AppStream~~, AmazonSageMaker, AmazonMSK, etc.

~~cn-north-1 and cn-northwest-1 didn't seem to be picking up any pricing information until #1635~~

PatMyron · 2020-05-17T21:34:14Z

scripts/update_specs_from_pricing.py

                    results[region_map[product.get('attributes').get('location')]].add(
-                        usage_type
+                        product.get('attributes').get('usagetype').split(':')[1]


instanceType is missing the dax. prefix, so usagetype is parsed

PatMyron · 2020-05-17T21:51:31Z

scripts/update_specs_from_pricing.py

-                    usage_type = product.get('attributes').get('usagetype').split(':')[1]
-                    results[region_map[product.get('attributes').get('location')]].add(usage_type)


switched Neptune from parsing usage_type to using instanceType

PatMyron · 2020-05-17T22:01:13Z

scripts/update_specs_from_pricing.py

@@ -145,14 +115,13 @@ def get_mq_pricing():
        'mq.m5.2xl': 'mq.m5.2xlarge',
        'mq.m5.4xl': 'mq.m5.4xlarge'


mq.m5.2xl and mq.m5.4xl are the only ones present so far, but wondering if every AmazonMQ instanceType ending in xl should have arge appended like Neptune

PatMyron · 2020-05-17T22:03:39Z

scripts/update_specs_from_pricing.py

    results = {}
    for page in get_paginator('AmazonMQ'):
        for price_item in page.get('PriceList', []):
            products = json.loads(price_item)
            product = products.get('product', {})
            if product:
-                if product.get('productFamily') == 'Broker Instances':
+                if product.get('productFamily') in ['Broker Instances']:
                    if not results.get(region_map[product.get('attributes').get('location')]):
                        results[region_map[product.get('attributes').get('location')]] = set()
                    usage_type = product.get('attributes').get('usagetype').split(':')[1]


AmazonMQ instanceType is not present, so usagetype must be parsed

#1535 https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json

#1535 https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonManagedBlockchain/current/index.json

#1535 https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonGameLift/current/index.json

#1535 https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAppStream/current/index.json

PatMyron added 3 commits May 17, 2020 14:29

consolidating scripts/update_specs_from_pricing.py

b641386

removing region_exceptions by mapping Los Angeles to us-west-2

952e90f

calling AmazonElastiCache

da9cbbf

PatMyron commented May 17, 2020

View reviewed changes

formatting

c46e489

PatMyron commented May 17, 2020

View reviewed changes

applying AmazonElastiCache CacheNodeType pricing information

f30ee0d

PatMyron changed the title ~~Pricing~~ adding Los Angeles and ElastiCache CacheNodeType and fixing Neptune pricing information May 17, 2020

PatMyron requested review from fatbasstard, kddejong, miparnisari and johnttompkins May 17, 2020 22:49

PatMyron marked this pull request as ready for review May 17, 2020 22:50

PatMyron marked this pull request as draft May 17, 2020 23:08

PatMyron marked this pull request as ready for review May 18, 2020 00:11

Elasticsearch InstanceType

a122737

PatMyron changed the title ~~adding Los Angeles and ElastiCache CacheNodeType and fixing Neptune pricing information~~ adding Los Angeles, ElastiCache CacheNodeType, Elasticsearch InstanceType, and fixing Neptune pricing information May 18, 2020

PatMyron changed the title ~~adding Los Angeles, ElastiCache CacheNodeType, Elasticsearch InstanceType, and fixing Neptune pricing information~~ adding Los Angeles, ElastiCache CacheNodeType, Elasticsearch InstanceType, and fixing Neptune InstanceClass pricing information May 18, 2020

This was referenced May 18, 2020

eu-south-1 (Milan) #1496

Merged

Description of cfn-lint development environment and build steps #903

Closed

miparnisari approved these changes May 18, 2020

View reviewed changes

kddejong merged commit 8d70feb into aws-cloudformation:master May 18, 2020

PatMyron added a commit that referenced this pull request May 19, 2020

ElasticMapReduce InstanceType

916a3bf

#1535 https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/ElasticMapReduce/current/index.json

PatMyron added a commit that referenced this pull request May 19, 2020

ManagedBlockchain InstanceType

f44ce69

#1535 https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonManagedBlockchain/current/index.json

PatMyron added a commit that referenced this pull request May 19, 2020

GameLift InstanceType

c92907c

#1535 https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonGameLift/current/index.json

PatMyron added a commit that referenced this pull request May 19, 2020

AppStream InstanceType

d6a4b94

#1535 https://pricing.us-east-1.amazonaws.com/offers/v1.0/aws/AmazonAppStream/current/index.json

PatMyron deleted the pricing branch June 4, 2020 16:24

This was referenced Aug 18, 2020

CloudFormation Resource Specification 16.3.0 #1635

Merged

consolidating scripts/update_specs_from_pricing.py #1503

Merged

PatMyron mentioned this pull request Nov 20, 2020

automating maintenance with Github actions #1792

Merged

This was referenced Feb 10, 2022

Update CloudFormation spec to 41.2.0 #2119

Merged

Elasticsearch instance type productFamily switched to OpenSearch #2214

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding Los Angeles, ElastiCache CacheNodeType, Elasticsearch InstanceType, and fixing Neptune InstanceClass pricing information #1535

adding Los Angeles, ElastiCache CacheNodeType, Elasticsearch InstanceType, and fixing Neptune InstanceClass pricing information #1535

PatMyron commented May 17, 2020 •

edited

Loading

PatMyron May 17, 2020 •

edited

Loading

PatMyron May 17, 2020 •

edited

Loading

PatMyron May 17, 2020 •

edited

Loading

PatMyron May 17, 2020 •

edited

Loading

		usage_type = product.get('attributes').get('usagetype').split(':')[1]
		results[region_map[product.get('attributes').get('location')]].add(usage_type)

		@@ -145,14 +115,13 @@ def get_mq_pricing():
		'mq.m5.2xl': 'mq.m5.2xlarge',
		'mq.m5.4xl': 'mq.m5.4xlarge'

adding Los Angeles, ElastiCache CacheNodeType, Elasticsearch InstanceType, and fixing Neptune InstanceClass pricing information #1535

adding Los Angeles, ElastiCache CacheNodeType, Elasticsearch InstanceType, and fixing Neptune InstanceClass pricing information #1535

Conversation

PatMyron commented May 17, 2020 • edited Loading

how to test:

future work:

PatMyron May 17, 2020 • edited Loading

Choose a reason for hiding this comment

PatMyron May 17, 2020 • edited Loading

Choose a reason for hiding this comment

PatMyron May 17, 2020 • edited Loading

Choose a reason for hiding this comment

PatMyron May 17, 2020 • edited Loading

Choose a reason for hiding this comment

PatMyron commented May 17, 2020 •

edited

Loading

PatMyron May 17, 2020 •

edited

Loading

PatMyron May 17, 2020 •

edited

Loading

PatMyron May 17, 2020 •

edited

Loading

PatMyron May 17, 2020 •

edited

Loading