Add SNS subscriptions to queues

[jesusch]

I assum this could be a common usecase

constructs:
  my-queue:
    type: queue
    subscriptions:
    - arn:aws:sns:us-east-1:1234567890:my-topic
    - arn:aws:sns:eu-west-1:1234567890:my-othertopic

[mnapoli]

What would be the use case?

[jesusch]

Subscribe sqs queues to sns topics

…

On Fri, 16 Sept 2022, 09:18 Matthieu Napoli, ***@***.***> wrote: What would be the use case? — Reply to this email directly, view it on GitHub <#266 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AASA3LTOJZUFIB6IXZF56N3V6RQRJANCNFSM6AAAAAAQNNFOXA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Agrumas]

I also have a similar need in event-driven architecture.
For example, a service sends events to SNS topic and other services have their own SQS queue to process messages from other services. A subscription is needed to filter and get events from other services.

[otaviodantas]

I have the same need as @Agrumas :/

[ryanwalters]

What would be the use case?

A common pattern, usually called "fanout", is to subscribe SQS queues to SNS topics. AWS documentation has a basic example of the pattern here: https://aws.amazon.com/getting-started/hands-on/send-fanout-event-notifications/

Here's an example of what the CloudFormation configuration looks like (there's probably a CDK example out there too): https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-sns-subscription.html#aws-resource-sns-subscription--examples--Create_a_subscription_with_optional_attributes--yaml

Note the protocol set to sqs.

[cmcnicholas]

What would be the use case?

A common pattern, usually called "fanout", is to subscribe SQS queues to SNS topics. AWS documentation has a basic example of the pattern here: https://aws.amazon.com/getting-started/hands-on/send-fanout-event-notifications/

Here's an example of what the CloudFormation configuration looks like (there's probably a CDK example out there too): https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-sns-subscription.html#aws-resource-sns-subscription--examples--Create_a_subscription_with_optional_attributes--yaml

Note the protocol set to sqs.

Exactly this, I'm experiencing this on a project at the moment and have seen this used before in a few places, it's quite common for allowing resilient simplified async processing of independent pipelines e.g.

[cmcnicholas]

I've had a stab at implementing this, see #344

I would appreciate any help simplifying or helping to resolve the issue about having to use ARN strings instead of Refs.

[cmcnicholas]

sorry to chase up on this, after doing a little bit of digging I still can't find the "plugin way" to resolve refs in the linked PR e.g.

constructs:
  ## adds a simple queue and unfiltered subscription
  my-queue:
    type: queue
    worker:
      handler: worker.handler
    subscriptions:
      - topic: !Ref MyTopic # this doesn't work

does anyone have any reference material for working with refs that need to be resolved dynamically with lift?

[cmcnicholas]

I've opened a discussion for this too #345

[cmcnicholas]

Managed to resolve it, PR now ready #344

[johnrobertcobbold]

@cmcnicholas it seems that your PR never got merged? It's such a nice feature

[cmcnicholas]

@cmcnicholas it seems that your PR never got merged? It's such a nice feature

Yer is a shame, I feel like this is a really common pattern.

I've forked and been using it in a live project for a few months now so feel free to do the same until it maybe ends up in lift natively.

[johnrobertcobbold]

@mnapoli Would you consider merging/looking into @cmcnicholas's PR? The fanout pattern described above is very useful and without this PR, we are currently having to manually create the SNS subscription for the SQS queue created with lift.

[madCodes]

@mnapoli , this pattern is almost universal. Not sure, why this is not getting the love it deserves :)