Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

F/aws_sns_topic_subscription: Provide full support to HTTP/HTTPS/EMAIL/EMAIL-JSON protocols / SQS Subscription without Assume Role in both Accounts #15633

Closed
wants to merge 11 commits into from
Closed

F/aws_sns_topic_subscription: Provide full support to HTTP/HTTPS/EMAIL/EMAIL-JSON protocols / SQS Subscription without Assume Role in both Accounts #15633

Show file tree
Hide file tree
Changes from 9 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
79fd3e3
Provide full support to HTTP/HTTPS/EMAIL/EMAIL-JSON protocols / SQS S…
smailli-moraes Oct 14, 2020
e18130c
terrafmt fix
smailli-moraes Oct 14, 2020
a26741e
Enhancement
smailli-moraes Oct 14, 2020
18e0d55
Update documentation
smailli-moraes Oct 14, 2020
734291b
terrafmt fix
smailli-moraes Oct 14, 2020
f061efc
Remove Update Timeout
smailli-moraes Oct 14, 2020
c00e508
Deprecated endpoint_auto_confirms and confirmation_timeout_in_minutes
smailli-moraes Oct 14, 2020
c00fbeb
Using aws_iam_policy_document to generate access policy
smailli-moraes Oct 14, 2020
926ff7d
Merge branch 'master' of github.com:smailli/terraform-provider-aws in…
smailli-moraes Oct 14, 2020
c199ae7
Merge branch 'main' into f/aws_sns_topic_subscription
smailli Feb 12, 2021
f297121
Merge branch 'main' into f/aws_sns_topic_subscription
smailli Oct 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 17 additions & 16 deletions aws/resource_aws_sns_topic_subscription.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,19 +33,24 @@ func resourceAwsSnsTopicSubscription() *schema.Resource {
State: schema.ImportStatePassthrough,
},

Timeouts: &schema.ResourceTimeout{
Create: schema.DefaultTimeout(5 * time.Minute),
},

Schema: map[string]*schema.Schema{
"protocol": {
Type: schema.TypeString,
Required: true,
ForceNew: true,
ValidateFunc: validation.StringInSlice([]string{
// email and email-json not supported
"application",
"http",
"https",
"lambda",
"sms",
"sqs",
"email",
"email-json",
}, true),
},
"endpoint": {
Expand All @@ -54,14 +59,16 @@ func resourceAwsSnsTopicSubscription() *schema.Resource {
ForceNew: true,
},
"endpoint_auto_confirms": {
Type: schema.TypeBool,
Optional: true,
Default: false,
Type: schema.TypeBool,
Optional: true,
Default: false,
Deprecated: "endpoint_auto_confirms exists for historical compatibility and should not be used.",
},
"confirmation_timeout_in_minutes": {
Type: schema.TypeInt,
Optional: true,
Default: 1,
Type: schema.TypeInt,
Optional: true,
Default: 1,
Deprecated: "confirmation_timeout_in_minutes exists for historical compatibility and should not be used.",
},
"topic_arn": {
Type: schema.TypeString,
Expand Down Expand Up @@ -229,14 +236,8 @@ func subscribeToSNSTopic(d *schema.ResourceData, snsconn *sns.SNS) (output *sns.
protocol := d.Get("protocol").(string)
endpoint := d.Get("endpoint").(string)
topic_arn := d.Get("topic_arn").(string)
endpoint_auto_confirms := d.Get("endpoint_auto_confirms").(bool)
confirmation_timeout_in_minutes := d.Get("confirmation_timeout_in_minutes").(int)
attributes := getResourceAttributes(d)

if strings.Contains(protocol, "http") && !endpoint_auto_confirms {
return nil, fmt.Errorf("Protocol http/https is only supported for endpoints which auto confirms!")
}

log.Printf("[DEBUG] SNS create topic subscription: %s (%s) @ '%s'", endpoint, protocol, topic_arn)

req := &sns.SubscribeInput{
Expand All @@ -253,11 +254,11 @@ func subscribeToSNSTopic(d *schema.ResourceData, snsconn *sns.SNS) (output *sns.

log.Printf("[DEBUG] Finished subscribing to topic %s with subscription arn %s", topic_arn, *output.SubscriptionArn)

if strings.Contains(protocol, "http") && subscriptionHasPendingConfirmation(output.SubscriptionArn) {
if subscriptionHasPendingConfirmation(output.SubscriptionArn) {

log.Printf("[DEBUG] SNS create topic subscription is pending so fetching the subscription list for topic : %s (%s) @ '%s'", endpoint, protocol, topic_arn)

err = resource.Retry(time.Duration(confirmation_timeout_in_minutes)*time.Minute, func() *resource.RetryError {
err = resource.Retry(d.Timeout(schema.TimeoutCreate), func() *resource.RetryError {

subscription, err := findSubscriptionByNonID(d, snsconn)

Expand All @@ -266,7 +267,7 @@ func subscribeToSNSTopic(d *schema.ResourceData, snsconn *sns.SNS) (output *sns.
}

if subscription == nil {
return resource.RetryableError(fmt.Errorf("Endpoint (%s) did not autoconfirm the subscription for topic %s", endpoint, topic_arn))
return resource.RetryableError(fmt.Errorf("Endpoint (%s) did not confirm the subscription for topic %s", endpoint, topic_arn))
}

output.SubscriptionArn = subscription.SubscriptionArn
Expand Down
192 changes: 37 additions & 155 deletions website/docs/r/sns_topic_subscription.html.markdown
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,14 +13,6 @@ This resource allows you to automatically place messages sent to SNS topics in S
to a given endpoint, send SMS messages, or notify devices / applications. The most likely use case for Terraform users will
probably be SQS queues.

~> **NOTE:** If the SNS topic and SQS queue are in different AWS regions, it is important for the "aws_sns_topic_subscription" to use an AWS provider that is in the same region of the SNS topic. If the "aws_sns_topic_subscription" is using a provider with a different region than the SNS topic, terraform will fail to create the subscription.

~> **NOTE:** Setup of cross-account subscriptions from SNS topics to SQS queues requires Terraform to have access to BOTH accounts.

~> **NOTE:** If SNS topic and SQS queue are in different AWS accounts but the same region it is important for the "aws_sns_topic_subscription" to use the AWS provider of the account with the SQS queue. If "aws_sns_topic_subscription" is using a Provider with a different account than the SQS queue, terraform creates the subscriptions but does not keep state and tries to re-create the subscription at every apply.

~> **NOTE:** If SNS topic and SQS queue are in different AWS accounts and different AWS regions it is important to recognize that the subscription needs to be initiated from the account with the SQS queue but in the region of the SNS topic.

## Example Usage

You can directly supply a topic and ARN by hand in the `topic_arn` property along with the queue ARN:
Expand Down Expand Up @@ -53,97 +45,20 @@ resource "aws_sns_topic_subscription" "user_updates_sqs_target" {

You can subscribe SNS topics to SQS queues in different Amazon accounts and regions:

```hcl
variable "sns" {
default = {
account-id = "111111111111"
role-name = "service/service-hashicorp-terraform"
name = "example-sns-topic"
display_name = "example"
region = "us-west-1"
}
}

variable "sqs" {
default = {
account-id = "222222222222"
role-name = "service/service-hashicorp-terraform"
name = "example-sqs-queue"
region = "us-east-1"
}
}

data "aws_iam_policy_document" "sns-topic-policy" {
policy_id = "__default_policy_ID"

statement {
actions = [
"SNS:Subscribe",
"SNS:SetTopicAttributes",
"SNS:RemovePermission",
"SNS:Receive",
"SNS:Publish",
"SNS:ListSubscriptionsByTopic",
"SNS:GetTopicAttributes",
"SNS:DeleteTopic",
"SNS:AddPermission",
]

condition {
test = "StringEquals"
variable = "AWS:SourceOwner"
-> NOTE:
Terraform must be run on each account individually.
SQS in account `222222222222` and region `us-east-1`
SNS topic and Subscription in account `111111111111` and region `us-west-1`

values = [
var.sns["account-id"],
]
}
### SQS Queue (Account Id: 222222222222 / Region: us-east-1)

effect = "Allow"

principals {
type = "AWS"
identifiers = ["*"]
}

resources = [
"arn:aws:sns:${var.sns["region"]}:${var.sns["account-id"]}:${var.sns["name"]}",
]

sid = "__default_statement_ID"
}

statement {
actions = [
"SNS:Subscribe",
"SNS:Receive",
]

condition {
test = "StringLike"
variable = "SNS:Endpoint"

values = [
"arn:aws:sqs:${var.sqs["region"]}:${var.sqs["account-id"]}:${var.sqs["name"]}",
]
}

effect = "Allow"

principals {
type = "AWS"
identifiers = ["*"]
}

resources = [
"arn:aws:sns:${var.sns["region"]}:${var.sns["account-id"]}:${var.sns["name"]}",
]

sid = "__console_sub_0"
}
```hcl
resource "aws_sqs_queue" "this" {
name = "example-sqs-queue"
}

data "aws_iam_policy_document" "sqs-queue-policy" {
policy_id = "arn:aws:sqs:${var.sqs["region"]}:${var.sqs["account-id"]}:${var.sqs["name"]}/SQSDefaultPolicy"
policy_id = "${aws_sqs_queue.this.arn}/SQSDefaultPolicy"

statement {
sid = "example-sns-topic"
Expand All @@ -159,71 +74,37 @@ data "aws_iam_policy_document" "sqs-queue-policy" {
]

resources = [
"arn:aws:sqs:${var.sqs["region"]}:${var.sqs["account-id"]}:${var.sqs["name"]}",
aws_sqs_queue.this.arn
]

condition {
test = "ArnEquals"
variable = "aws:SourceArn"

values = [
"arn:aws:sns:${var.sns["region"]}:${var.sns["account-id"]}:${var.sns["name"]}",
"arn:aws:sns:us-west-1:111111111111:example-sns-topic",
]
}
}
}

# provider to manage SNS topics
provider "aws" {
alias = "sns"
region = var.sns["region"]

assume_role {
role_arn = "arn:aws:iam::${var.sns["account-id"]}:role/${var.sns["role-name"]}"
session_name = "sns-${var.sns["region"]}"
}
}

# provider to manage SQS queues
provider "aws" {
alias = "sqs"
region = var.sqs["region"]

assume_role {
role_arn = "arn:aws:iam::${var.sqs["account-id"]}:role/${var.sqs["role-name"]}"
session_name = "sqs-${var.sqs["region"]}"
}
}

# provider to subscribe SQS to SNS (using the SQS account but the SNS region)
provider "aws" {
alias = "sns2sqs"
region = var.sns["region"]

assume_role {
role_arn = "arn:aws:iam::${var.sqs["account-id"]}:role/${var.sqs["role-name"]}"
session_name = "sns2sqs-${var.sns["region"]}"
}
resource "aws_sqs_queue_policy" "this" {
queue_url = aws_sqs_queue.this.id
policy = data.aws_iam_policy_document.sqs-queue-policy.json
}
```

resource "aws_sns_topic" "sns-topic" {
provider = "aws.sns"
name = var.sns["name"]
display_name = var.sns["display_name"]
policy = data.aws_iam_policy_document.sns-topic-policy.json
}
### SNS Topic and Subscription (Account Id: 111111111111 / Region: us-west-1)

resource "aws_sqs_queue" "sqs-queue" {
provider = "aws.sqs"
name = var.sqs["name"]
policy = data.aws_iam_policy_document.sqs-queue-policy.json
```hcl
resource "aws_sns_topic" "this" {
name = "example-sns-topic"
}

resource "aws_sns_topic_subscription" "sns-topic" {
provider = "aws.sns2sqs"
topic_arn = aws_sns_topic.sns-topic.arn
resource "aws_sns_topic_subscription" "this" {
topic_arn = aws_sns_topic.this.arn
protocol = "sqs"
endpoint = aws_sqs_queue.sqs-queue.arn
endpoint = "arn:aws:sqs:us-east-1:222222222222:example-sqs-queue"
}
```

Expand All @@ -232,14 +113,23 @@ resource "aws_sns_topic_subscription" "sns-topic" {
The following arguments are supported:

* `topic_arn` - (Required) The ARN of the SNS topic to subscribe to
* `protocol` - (Required) The protocol to use. The possible values for this are: `sqs`, `sms`, `lambda`, `application`. (`http` or `https` are partially supported, see below) (`email` is an option but is unsupported, see below).
* `protocol` - (Required) The protocol to use, see below. Refer to the [SNS API docs](https://docs.aws.amazon.com/sns/latest/api/API_Subscribe.html) for more details.
* `endpoint` - (Required) The endpoint to send data to, the contents will vary with the protocol. (see below for more information)
* `endpoint_auto_confirms` - (Optional) Boolean indicating whether the end point is capable of [auto confirming subscription](http://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html#SendMessageToHttp.prepare) e.g., PagerDuty (default is false)
* `confirmation_timeout_in_minutes` - (Optional) Integer indicating number of minutes to wait in retying mode for fetching subscription arn before marking it as failure. Only applicable for http and https protocols (default is 1 minute).
* `endpoint_auto_confirms` - (Optional, **DEPRECATED**) Boolean indicating whether the end point is capable of [auto confirming subscription](http://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp.html#SendMessageToHttp.prepare) e.g., PagerDuty (default is false)
* `confirmation_timeout_in_minutes` - (Optional, **DEPRECATED**) Integer indicating number of minutes to wait in retying mode for fetching subscription arn before marking it as failure. Only applicable for http and https protocols (default is 1 minute).
* `raw_message_delivery` - (Optional) Boolean indicating whether or not to enable raw message delivery (the original message is directly passed, not wrapped in JSON with the original message in the message property) (default is false).
* `filter_policy` - (Optional) JSON String with the filter policy that will be used in the subscription to filter messages seen by the target resource. Refer to the [SNS docs](https://docs.aws.amazon.com/sns/latest/dg/message-filtering.html) for more details.
* `delivery_policy` - (Optional) JSON String with the delivery policy (retries, backoff, etc.) that will be used in the subscription - this only applies to HTTP/S subscriptions. Refer to the [SNS docs](https://docs.aws.amazon.com/sns/latest/dg/DeliveryPolicies.html) for more details.

### Timeouts

Refer to the [AWS SNS docs](https://docs.aws.amazon.com/sns/latest/dg/sns-send-message-to-sqs-cross-account.html) for more details.

The `timeouts` block allows you to specify [timeouts](https://www.terraform.io/docs/configuration/resources.html#timeouts) for certain actions:

* `create` - (Defaults to 5 mins) - You should receive a confirmation message at the configured endpoint and validate the subscription.


### Protocols supported

Supported SNS protocols include:
Expand All @@ -248,20 +138,11 @@ Supported SNS protocols include:
* `sqs` -- delivery of JSON-encoded message to an Amazon SQS queue
* `application` -- delivery of JSON-encoded message to an EndpointArn for a mobile app and device
* `sms` -- delivery text message

Partially supported SNS protocols include:

* `http` -- delivery of JSON-encoded messages via HTTP. Supported only for the end points that auto confirms the subscription.
* `https` -- delivery of JSON-encoded messages via HTTPS. Supported only for the end points that auto confirms the subscription.

Unsupported protocols include the following:

* `http` -- delivery of JSON-encoded messages via HTTP.
* `https` -- delivery of JSON-encoded messages via HTTPS.
* `email` -- delivery of message via SMTP
* `email-json` -- delivery of JSON-encoded message via SMTP

These are unsupported because the endpoint needs to be authorized and does not
generate an ARN until the target email address has been validated. This breaks
the Terraform model and as a result are not currently supported.

### Specifying endpoints

Expand All @@ -287,3 +168,4 @@ SNS Topic Subscriptions can be imported using the `subscription arn`, e.g.
```
$ terraform import aws_sns_topic_subscription.user_updates_sqs_target arn:aws:sns:us-west-2:0123456789012:my-topic:8a21d249-4329-4871-acc6-7be709c6ea7f
```