Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: fetch DynamoDB costs for RCUs and WCUs #1111

Merged
merged 14 commits into from
Oct 19, 2023
Merged

feat: fetch DynamoDB costs for RCUs and WCUs #1111

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
fb0bf02
feat: fetch DynamoDB costs for RCUs and WCUs
Traxmaxx Oct 17, 2023
9c70b10
chore: cleanup and add util test
Traxmaxx Oct 17, 2023
4d876d6
fix: address linter errors
Traxmaxx Oct 17, 2023
454aa9f
fix: set correct expect values
Traxmaxx Oct 17, 2023
cbe7a79
fix: fix log
Traxmaxx Oct 17, 2023
3065969
fix: set region to US before initializing pricing client
Traxmaxx Oct 18, 2023
400fc26
fix: address review comment and move util into utils and also the tes…
Traxmaxx Oct 18, 2023
dc5b786
fix: add instance name to awscli util
Traxmaxx Oct 18, 2023
53419b7
fix: address review comment and improve error handling
Traxmaxx Oct 18, 2023
70f2b2f
Merge branch 'develop' of github.com:tailwarden/komiser into feature/…
Traxmaxx Oct 18, 2023
8bb4655
fix: fix typo
Traxmaxx Oct 18, 2023
d0b4ca4
fix: address review comment
Traxmaxx Oct 19, 2023
cdca2ad
fix: adress review comment and update pricing table unit
Traxmaxx Oct 19, 2023
fbd297a
Merge branch 'develop' into feature/tech-1702
mlabouardy Oct 19, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
57 changes: 56 additions & 1 deletion providers/aws/dynamodb/tables.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,54 @@ import (

log "github.com/sirupsen/logrus"

"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/service/dynamodb"
"github.com/aws/aws-sdk-go-v2/service/pricing"
"github.com/aws/aws-sdk-go-v2/service/pricing/types"
"github.com/aws/aws-sdk-go-v2/service/sts"
awsUtils "github.com/tailwarden/komiser/providers/aws/utils"
. "github.com/tailwarden/komiser/models"
. "github.com/tailwarden/komiser/providers"
)


func Tables(ctx context.Context, client ProviderClient) ([]Resource, error) {
resources := make([]Resource, 0)
var config dynamodb.ListTablesInput
dynamodbClient := dynamodb.NewFromConfig(*client.AWSClient)

var monthlyCost float64 = 0.0
// there is something strange going on when using pricing client with regions other than us-east-1
// https://discord.com/channels/932683789384183808/1117721764957536318/1162338171435090032
Comment on lines +27 to +28
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/using-price-list-query-api.html#price-list-query-api-endpoints

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean there could be potentially different prices returned per region? 🤔

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the same place they say

The AWS Region is the API endpoint for the Price List Query API. The endpoints aren't related to product or service attributes.

So, IDTS but it's worth checking out

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

evem the cost explorer gives only one endpoint https://docs.aws.amazon.com/cost-management/latest/userguide/ce-api.html

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I talked with @mlabouardy and I will create a helper to take care of this in a new PR in the next days.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's worth fixing as the refactor work @AvineshTripathi is doing will probably address this as well. Since the client will be created only once.

Copy link
Contributor Author

@Traxmaxx Traxmaxx Oct 19, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think it's worth fixing as the refactor work @AvineshTripathi is doing will probably address this as well. Since the client will be created only once.

Awesome, good to know thanks for the heads-up! I think the PR is good to go from my side if you agree as well. Thanks a lot for the thorough review!

oldRegion := client.AWSClient.Region
client.AWSClient.Region = "us-east-1"
pricingClient := pricing.NewFromConfig(*client.AWSClient)
client.AWSClient.Region = oldRegion

pricingOutput, err := pricingClient.GetProducts(ctx, &pricing.GetProductsInput{
ServiceCode: aws.String("AmazonDynamoDB"),
Filters: []types.Filter{
{
Field: aws.String("regionCode"),
Value: aws.String(client.AWSClient.Region),
Type: types.FilterTypeTermMatch,
},
},
})

if err != nil {
log.Errorf("ERROR: Couldn't fetch pricing info for AWS DynamoDB: %v", err)
}

priceMap, err := awsUtils.GetPriceMap(pricingOutput, "group")

if err != nil {
log.Errorf("ERROR: Failed to fetch pricing map: %v", err)
}


output, err := dynamodbClient.ListTables(ctx, &config)

if err != nil {
return resources, err
}
Expand Down Expand Up @@ -47,14 +84,32 @@ func Tables(ctx context.Context, client ProviderClient) ([]Resource, error) {
}
}

tableDetails, err := dynamodbClient.DescribeTable(ctx, &dynamodb.DescribeTableInput{
TableName: aws.String(table),
})

if err != nil {
log.Errorf("ERROR: Failed to query DynamoDB table details: %v", err)
}

if tableDetails.Table != nil && tableDetails.Table.ProvisionedThroughput != nil {
provisionedRCUs := tableDetails.Table.ProvisionedThroughput.ReadCapacityUnits
provisionedWCUs := tableDetails.Table.ProvisionedThroughput.WriteCapacityUnits

RCUCharges := awsUtils.GetCost(priceMap["DDB-ReadUnits"], awsUtils.Int64PtrToFloat64(provisionedRCUs))
WCUCharges := awsUtils.GetCost(priceMap["DDB-WriteUnits"], awsUtils.Int64PtrToFloat64(provisionedWCUs))

monthlyCost = RCUCharges + WCUCharges
}

resources = append(resources, Resource{
Provider: "AWS",
Account: client.Name,
Service: "DynamoDB",
ResourceId: resourceArn,
Region: client.AWSClient.Region,
Name: table,
Cost: 0,
Cost: monthlyCost,
Tags: tags,
FetchedAt: time.Now(),
Link: fmt.Sprintf("https://%s.console.aws.amazon.com/dynamodbv2/home?region=%s#table?initialTagKey=&name=%s", client.AWSClient.Region, client.AWSClient.Region, table),
Expand Down
7 changes: 7 additions & 0 deletions providers/aws/utils/utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,3 +91,10 @@ func GetPriceMap(pricingOutput *pricing.GetProductsOutput, field string) (map[st

return priceMap, nil
}

func Int64PtrToFloat64(i *int64) float64 {
if i == nil {
return 0.0 // or any default value you prefer
}
return float64(*i)
}
21 changes: 21 additions & 0 deletions providers/aws/utils/utils_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,3 +234,24 @@ func TestGetPriceMap_NoPricingOutput(t *testing.T) {
t.Errorf("Expected an empty priceMap, but got %v", priceMap)
}
}

func TestInt64PtrToFloat64_ValidInput(t *testing.T) {
var number int64 = 1
pointer := &number

returnValue := Int64PtrToFloat64(pointer)
var expected float64 = 1.0
if returnValue != expected {
t.Errorf("Expected return value: %f, but got: %f", expected, returnValue)
}
}

func TestInt64PtrToFloat64_NilInput(t *testing.T) {
// nil input
returnValue := Int64PtrToFloat64(nil)
var expected float64 = 0.0
if returnValue != expected {
t.Errorf("Expected return value: %f, but got: %f", expected, returnValue)
}
}