Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

r/kinesis_analytics_application: new resource #5456

Merged
merged 30 commits into from
Nov 7, 2018

Conversation

kl4w
Copy link
Contributor

@kl4w kl4w commented Aug 5, 2018

Still work in progress right now, still need to support Outputs: https://docs.aws.amazon.com/kinesisanalytics/latest/dev/API_CreateApplication.html#analytics-CreateApplication-request-Outputs

Fixes #536

Changes proposed in this pull request:

  • create new kinesis_analytics_application resource

Output from acceptance testing:

make testacc TEST=./aws TESTARGS='-run=TestAccAWSKinesisAnalyticsApplication_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccAWSKinesisAnalyticsApplication_ -timeout 120m
=== RUN   TestAccAWSKinesisAnalyticsApplication_basic
--- PASS: TestAccAWSKinesisAnalyticsApplication_basic (13.58s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_update
--- PASS: TestAccAWSKinesisAnalyticsApplication_update (21.21s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_addCloudwatchLoggingOptions
--- PASS: TestAccAWSKinesisAnalyticsApplication_addCloudwatchLoggingOptions (67.57s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_updateCloudwatchLoggingOptions
--- PASS: TestAccAWSKinesisAnalyticsApplication_updateCloudwatchLoggingOptions (70.95s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_inputsKinesisStream
--- PASS: TestAccAWSKinesisAnalyticsApplication_inputsKinesisStream (131.00s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_inputUpdateKinesisStream
--- PASS: TestAccAWSKinesisAnalyticsApplication_inputUpdateKinesisStream (212.85s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	517.185s

@ghost ghost added size/XXL Managed by automation to categorize the size of a PR. dependencies Used to indicate dependency changes. labels Aug 5, 2018
CurrentApplicationVersionId: aws.Int64(int64(version)),
CloudWatchLoggingOption: cloudwatchLoggingOption,
}
conn.AddApplicationCloudWatchLoggingOption(addOpts)
Copy link
Contributor Author

@kl4w kl4w Aug 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really sure the best way to handle this but adding new cloudwatch logging options cannot be part of the UpdateApplication call hence a separate call

`, streamName, rInt)
}

func fulfillSleep() resource.TestCheckFunc {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I put in a sleep here because I found that the trust policy for the IAM role needs to propagate before referencing the role in the tests

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @kl4w 👋 We prefer that the resource usually handles this type of logic by retrying on the specific error message(s) that are returned for a minute or two. e.g.

// Handle IAM eventual consistency
err := resource.Retry(1*time.Minute, func() *resource.RetryError {
  // CreateApplication as an example
  err := conn.CreateApplication(input)
  if err != nil {
    // kinesisanalytics.ErrCodeInvalidArgumentException as an example
    if isAWSErr(err, kinesisanalytics.ErrCodeInvalidArgumentException, "some IAM-related error message, e.g. not authorized") {
      return resource.RetryableError(err)
    }
    return resource.NonRetryableError(err)
  }
  return nil
})
if err != nil {
  return fmt.Errorf("error creating application: %s", err)
}

@ghost ghost added size/XXL Managed by automation to categorize the size of a PR. dependencies Used to indicate dependency changes. labels Aug 5, 2018
@@ -750,6 +750,14 @@
"version": "v1.14.33",
"versionExact": "v1.14.33"
},
{
"checksumSHA1": "nUdxsOW9jg+m+sWZYPu7oX9rZo8=",
"path": "github.com/aws/aws-sdk-go/service/kinesisanalytics",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please separate the vendoring into its own PR? We do this because these dependencies are a fast moving target, it makes the PRs a little more manageable, and we can quickly merge it in. FYI master is already AWS SDK Go v1.15.4.

https://github.com/terraform-providers/terraform-provider-aws/blob/2c3eaaef70430fa462a5750825b2a339e910348a/vendor/vendor.json#L41-L48

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Absolutely. I've created the PR here: #5461

@bflad bflad added new-resource Introduces a new resource. service/kinesisanalytics Issues and PRs that pertain to the kinesisanalytics service. labels Aug 6, 2018
@kl4w kl4w mentioned this pull request Aug 6, 2018
@kl4w kl4w force-pushed the kinesis-analytics-application branch from 8e6da57 to 3bc0d53 Compare August 6, 2018 15:32
@ghost ghost added size/XXL Managed by automation to categorize the size of a PR. dependencies Used to indicate dependency changes. labels Aug 6, 2018
@kl4w kl4w force-pushed the kinesis-analytics-application branch from 3bc0d53 to 193b409 Compare August 6, 2018 15:33
@ghost ghost added size/XXL Managed by automation to categorize the size of a PR. and removed dependencies Used to indicate dependency changes. labels Aug 6, 2018
@kl4w kl4w changed the title WIP: r/kinesis_analytics_application: new resource r/kinesis_analytics_application: new resource Aug 7, 2018
@kl4w
Copy link
Contributor Author

kl4w commented Aug 7, 2018

make testacc TEST=./aws TESTARGS='-run=TestAccAWSKinesisAnalyticsApplication_'
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -run=TestAccAWSKinesisAnalyticsApplication_ -timeout 120m
=== RUN   TestAccAWSKinesisAnalyticsApplication_basic
--- PASS: TestAccAWSKinesisAnalyticsApplication_basic (10.75s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_update
--- PASS: TestAccAWSKinesisAnalyticsApplication_update (14.77s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_addCloudwatchLoggingOptions
--- PASS: TestAccAWSKinesisAnalyticsApplication_addCloudwatchLoggingOptions (25.60s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_updateCloudwatchLoggingOptions
--- PASS: TestAccAWSKinesisAnalyticsApplication_updateCloudwatchLoggingOptions (37.05s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_inputsKinesisStream
--- PASS: TestAccAWSKinesisAnalyticsApplication_inputsKinesisStream (86.94s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_inputsAdd
--- PASS: TestAccAWSKinesisAnalyticsApplication_inputsAdd (94.82s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_inputsUpdateKinesisStream
--- PASS: TestAccAWSKinesisAnalyticsApplication_inputsUpdateKinesisStream (170.57s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_outputsKinesisStream
--- PASS: TestAccAWSKinesisAnalyticsApplication_outputsKinesisStream (88.25s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_outputsAdd
--- PASS: TestAccAWSKinesisAnalyticsApplication_outputsAdd (94.47s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_outputsUpdateKinesisStream
--- PASS: TestAccAWSKinesisAnalyticsApplication_outputsUpdateKinesisStream (182.43s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_referenceDataSource
--- PASS: TestAccAWSKinesisAnalyticsApplication_referenceDataSource (38.51s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_referenceDataSourceUpdate
--- PASS: TestAccAWSKinesisAnalyticsApplication_referenceDataSourceUpdate (59.21s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	903.415s

Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Phew, @kl4w, what a complicated service and resource! 😅 Great work so far. Please see below for some initial feedback and please reach out if you have any questions or do not have time to implement the items. 👍

}
resp, err := conn.DescribeApplication(describeOpts)
if err != nil {
if awsErr, ok := err.(awserr.Error); ok {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This error handling logic can be simplified with:

if isAWSErr(err, kinesisanalytics.ErrCodeResourceNotFoundException, "") {
  log.Printf("[WARN] Kinesis Analytics Application (%d) not found, removing from state", d.Id())
  d.SetId("")
  return nil
}
if err != nil {
  return fmt.Errorf("error reading Kinesis Analytics Application (%s): %s", d.Id(), err)
}

return err
}

d.SetId(aws.StringValue(resp.ApplicationDetail.ApplicationARN))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

d.SetId() should be called during the Create function unless the Update function will change the ARN.


if v, ok := d.GetOk("cloudwatch_logging_options"); ok {
clo := v.([]interface{})[0].(map[string]interface{})
cloudwatchLoggingOption := createCloudwatchLoggingOption(clo)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this project uses a very flat Go package structure, we prefer to include the AWS service name in the function name. We also generally call these functions "expand" functions. e.g. expandKinesisAnalyticsCloudwatchLoggingOption

Same for the below createX function names.

d.Set("status", aws.StringValue(resp.ApplicationDetail.ApplicationStatus))
d.Set("version", int(aws.Int64Value(resp.ApplicationDetail.ApplicationVersionId)))

if err := d.Set("cloudwatch_logging_options", getCloudwatchLoggingOptions(resp.ApplicationDetail.CloudWatchLoggingOptionDescriptions)); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this project uses a very flat Go package structure, we prefer to include the AWS service name in the function name. We also generally call these functions "flatten" functions. e.g. flattenKinesisAnalyticsCloudwatchLoggingOptions

Same for the below getX function names.

d.Set("version", int(aws.Int64Value(resp.ApplicationDetail.ApplicationVersionId)))

if err := d.Set("cloudwatch_logging_options", getCloudwatchLoggingOptions(resp.ApplicationDetail.CloudWatchLoggingOptionDescriptions)); err != nil {
return err
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Its generally helpful to provide the operator (and code maintainers) some context about the error message they are receiving, e.g. return fmt.Errorf("error setting cloudwatch_logging_options: %s", err)

Same for below d.Set() error handlers as well. 👍


func createCloudwatchLoggingOption(clo map[string]interface{}) *kinesisanalytics.CloudWatchLoggingOption {
cloudwatchLoggingOption := &kinesisanalytics.CloudWatchLoggingOption{
LogStreamARN: aws.String(clo["log_stream"].(string)),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should prefer to stick to the API (and more clear) attribute naming for any of these ARN parameters that accept an ARN, e.g. log_stream_arn

This applies to a large amount of the attributes below as well so I'll withhold from commenting on each of them.

testAccCheckKinesisAnalyticsApplicationExists(resName, &application),
resource.TestCheckResourceAttr(resName, "version", "2"),
resource.TestCheckResourceAttr(resName, "cloudwatch_logging_options.#", "1"),
resource.TestMatchResourceAttr(resName, "cloudwatch_logging_options.0.log_stream", streamRe),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of regexp matching the log stream ARN, you may find it easier to directly check the resource information via:

resource.TestCheckResourceAttrPair(resName, "cloudwatch_logging_options.0.log_stream_arn", "aws_cloudwatch_log_stream.test", "arn"),

This applies to the other below checks as well 😄

return fmt.Errorf("Error: Application still exists")
}
}
return nil
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should remove this return nil to ensure this function can check all configured aws_kinesis_analytics_application resources.

}
resp, err := conn.DescribeApplication(describeOpts)
if err == nil {
if resp.ApplicationDetail != nil && *resp.ApplicationDetail.ApplicationStatus != kinesisanalytics.ApplicationStatusDeleting {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the application does not delete immediately, should we be waiting for the deletion to complete in the Delete function? 🤔

@bflad bflad added the waiting-response Maintainers are waiting on response from community or contributor. label Oct 15, 2018
@kl4w kl4w force-pushed the kinesis-analytics-application branch from 5f82d55 to 5e9b5b8 Compare October 24, 2018 15:43
@ghost ghost added documentation Introduces or discusses updates to documentation. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure. labels Oct 24, 2018
@kl4w
Copy link
Contributor Author

kl4w commented Oct 24, 2018

@bflad made changes as requested. test output:

make testacc TEST=./aws TESTARGS='-run=TestAccAWSKinesisAnalyticsApplication_'
==> Fixing source code with gofmt...
gofmt -s -w ./aws
==> Checking that code complies with gofmt requirements...
TF_ACC=1 go test ./aws -v -parallel 20 -run=TestAccAWSKinesisAnalyticsApplication_ -timeout 120m
=== RUN   TestAccAWSKinesisAnalyticsApplication_basic
--- PASS: TestAccAWSKinesisAnalyticsApplication_basic (11.24s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_update
--- PASS: TestAccAWSKinesisAnalyticsApplication_update (17.09s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_addCloudwatchLoggingOptions
--- PASS: TestAccAWSKinesisAnalyticsApplication_addCloudwatchLoggingOptions (26.86s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_updateCloudwatchLoggingOptions
--- PASS: TestAccAWSKinesisAnalyticsApplication_updateCloudwatchLoggingOptions (37.94s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_inputsKinesisStream
--- PASS: TestAccAWSKinesisAnalyticsApplication_inputsKinesisStream (88.39s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_inputsAdd
--- PASS: TestAccAWSKinesisAnalyticsApplication_inputsAdd (96.66s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_inputsUpdateKinesisStream
--- PASS: TestAccAWSKinesisAnalyticsApplication_inputsUpdateKinesisStream (183.16s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_outputsKinesisStream
--- PASS: TestAccAWSKinesisAnalyticsApplication_outputsKinesisStream (89.69s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_outputsAdd
--- PASS: TestAccAWSKinesisAnalyticsApplication_outputsAdd (96.81s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_outputsUpdateKinesisStream
--- PASS: TestAccAWSKinesisAnalyticsApplication_outputsUpdateKinesisStream (173.10s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_referenceDataSource
--- PASS: TestAccAWSKinesisAnalyticsApplication_referenceDataSource (28.78s)
=== RUN   TestAccAWSKinesisAnalyticsApplication_referenceDataSourceUpdate
--- PASS: TestAccAWSKinesisAnalyticsApplication_referenceDataSourceUpdate (49.67s)
PASS
ok  	github.com/terraform-providers/terraform-provider-aws/aws	899.442s

@bflad bflad removed the waiting-response Maintainers are waiting on response from community or contributor. label Nov 7, 2018
Copy link
Contributor

@bflad bflad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, @kl4w! 🚀 Sorry for the delay in re-review after your updates! 😅 With such a complex resource like this is probably best to let folks start playing with it to tease out anything the testing missed -- LGTM though.

--- PASS: TestAccAWSKinesisAnalyticsApplication_basic (11.06s)
--- PASS: TestAccAWSKinesisAnalyticsApplication_update (17.23s)
--- PASS: TestAccAWSKinesisAnalyticsApplication_addCloudwatchLoggingOptions (25.38s)
--- PASS: TestAccAWSKinesisAnalyticsApplication_referenceDataSource (30.47s)
--- PASS: TestAccAWSKinesisAnalyticsApplication_updateCloudwatchLoggingOptions (38.74s)
--- PASS: TestAccAWSKinesisAnalyticsApplication_referenceDataSourceUpdate (54.39s)
--- PASS: TestAccAWSKinesisAnalyticsApplication_outputsKinesisStream (87.98s)
--- PASS: TestAccAWSKinesisAnalyticsApplication_inputsKinesisStream (88.00s)
--- PASS: TestAccAWSKinesisAnalyticsApplication_outputsAdd (94.81s)
--- PASS: TestAccAWSKinesisAnalyticsApplication_inputsAdd (110.65s)
--- PASS: TestAccAWSKinesisAnalyticsApplication_outputsUpdateKinesisStream (172.40s)
--- PASS: TestAccAWSKinesisAnalyticsApplication_inputsUpdateKinesisStream (172.42s)

@bflad bflad added this to the v1.43.0 milestone Nov 7, 2018
@bflad bflad merged commit ff41de4 into hashicorp:master Nov 7, 2018
bflad added a commit that referenced this pull request Nov 7, 2018
@kl4w kl4w deleted the kinesis-analytics-application branch November 7, 2018 16:20
@bflad
Copy link
Contributor

bflad commented Nov 7, 2018

This has been released in version 1.43.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

@arturoherrero
Copy link

Hello @kl4w @bflad. Thank you very much for this contribution!

I created a aws_kinesis_analytics_application resource and everything is working as expected. The only problem I can't solve is how to start the Kinesis Analytics application.
https://docs.aws.amazon.com/kinesisanalytics/latest/dev/API_StartApplication.html

At the moment I can go directly to the UI and click the button to start the application:

screenshot 2019-01-29 at 09 51 28

I guess I can start the application using Terraform, maybe I have to configure that action or something else. Am I missing something?

@kl4w
Copy link
Contributor Author

kl4w commented Jan 29, 2019

@arturoherrero you are correct, there's no functionality to start the application right now on creation. This probably could implemented by changing the status attribute: https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_kinesis_analytics_application.go#L55-L58 and calling StartApplication post-creation though. Mind submitting a feature request if there's not one already?

@netrebel
Copy link

netrebel commented Feb 11, 2019

Hi @kl4w,

I'm trying to add an additional output for the error_stream created by Kinesis Analytics by default to handle runtime errors.

The block in aws_kinesis_analytics_application looks like this:

outputs = [{
            name = "DESTINATION_SQL_STREAM"
            schema {
                record_format_type = "JSON"
            }
            lambda {
                resource_arn = "${aws_lambda_function.estimator.arn}"
                role_arn = "${aws_iam_role.lambda_execution.arn}"
            }
        },
        {
            name = "error_stream"
            schema {
                record_format_type = "JSON"
            }
            lambda {
                resource_arn = "${aws_lambda_function.splunk-logger.arn}"
                role_arn = "${aws_iam_role.lambda_execution.arn}"
            }
        }
    ]

But the corresponding block in the state file does not reflect that stream:

outputs.#: "1",
outputs.0.id: "1.1",
outputs.0.kinesis_firehose.#: "0",
outputs.0.kinesis_stream.#: "0",
outputs.0.lambda.#: "1",
outputs.0.lambda.0.resource_arn: "arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:some-lambda",
outputs.0.lambda.0.role_arn: "arn:aws:iam::XXXXXXXXXXXX:role/some-lambda",
outputs.0.name: "DESTINATION_SQL_STREAM",
outputs.0.schema.#: "1",
outputs.0.schema.0.record_format_type: "JSON",

I'm wondering if there's something wrong in my configuration or it's not possible to create the connection to a stream created by Kinesis Analytics by default. Thanks!

Using terraform 0.11.8 and aws provider 1.52

@kl4w
Copy link
Contributor Author

kl4w commented Feb 11, 2019

@netrebel there's nothing wrong with your configuration. For some reason, I didn't iterate over the list of outputs and only captures the first output. Can you please submit a new Issue if not yet created?

@netrebel
Copy link

@kl4w Before creating the issue I updated to the latest terraform (0.11.11) and aws provider (1.58) and when I ran it again and I got this in the log, which I had not seen before:

aws_kinesis_analytics_application.detection: Modifying... (ID: arn:aws:kinesisanalytics:us-west-2:XXXX...XXX:application/tf-vee2-vee2-detection)
  outputs.#:                             "1" => "2"
  outputs.1.lambda.#:                    "0" => "1"
  outputs.1.lambda.0.resource_arn:       "" => "arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:some-lambda"
  outputs.1.lambda.0.role_arn:           "" => "arn:aws:iam::XXXXXXXXXXXX:role/some-lambda-execution"
  outputs.1.name:                        "" => "error_stream"
  outputs.1.schema.#:                    "0" => "1"
  outputs.1.schema.0.record_format_type: "" => "JSON"

So I thought it had been fixed already, but looking at the AWS console I still don't see the connection to the error_stream.

I think I see what you mean with taking only the first element:
https://github.com/kl4w/terraform-provider-aws/blob/ff41de43022c1f402f636829dddf9f5ce9d951ef/aws/resource_aws_kinesis_analytics_application.go#L1390-L1391

func flattenKinesisAnalyticsOutputs(outputs []*kinesisanalytics.OutputDescription) []interface{} {
	s := []interface{}{}

	if len(outputs) > 0 {
		id := outputs[0]
...

I'll create the Issue. Thanks for the quick response!

@ghost
Copy link

ghost commented Apr 1, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Apr 1, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
documentation Introduces or discusses updates to documentation. new-resource Introduces a new resource. service/kinesisanalytics Issues and PRs that pertain to the kinesisanalytics service. size/XXL Managed by automation to categorize the size of a PR. tests PRs: expanded test coverage. Issues: expanded coverage, enhancements to test infrastructure.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for AWS Kinesis Analytics
4 participants