New component: AWS SQS Receiver

[yash025]

The purpose and use-cases of the new component

The AWS SQS receiver is designed to implement a mechanism for retrieving and processing telemetry data triggered by event notifications for S3 uploads, integrating directly with the OpenTelemetry Collector. This receiver will collect messages from Amazon Simple Queue Service (SQS) associated with S3 upload events and forward these to the OpenTelemetry Collector for further processing and exporting.

Example configuration for the component

    receivers:
      awssqsreceiver:
        queue: "https://sqs.us-west-2.amazonaws.com/xxxx/fci-raw-spans-queue-qal-usw2"
        region: "us-west-2"
        visibility_timeout: 120
        worker_pool: 2
        wait_timeout: 15

Telemetry data types supported

• traces
• metrics
• logs

Code Owner(s)

@yash025, @igrishi @MaheshGPai

Sponsor (optional)

No response

Additional context

No response

[yash025]

This has been useful for scaling our backend servers, which collect raw spans. We have one layer of lightweight collectors that collects the spans and exports them to S3 using the s3 exporter, while the second layer performs the heavy work of processing using different processors The event-driven approach has significantly helped us reduce data loss, manage retries, and better scaling.

[yash025]

#30750 (comment)

[VihasMakwana]

@yash025 #36315 seems related? Can you confirm and let me know?

[yash025]

@VihasMakwana , thanks for looking into this. Yes, it seems related, the discussion there is about updating the existing S3 receiver to listen to the SQS queue. In my opinion, updating the S3 receiver would complicate things. Since this is a common use case, does it make sense to have a separate receiver? What do you think?

[VihasMakwana]

If we add an awssqsreceiver, we shouldn't restrict it to only listening for S3 events. The receiver should be capable of handling all events from an SQS queue.

I believe it makes sense to create a new receiver specifically for reading events from SQS.

If you're willing to contribute this receiver, you'll need a sponsor. I recommend joining the OTeL Collector SIG and proposing this new component there. Additionally, it would be a good idea to reach out on Slack if possible.

[yash025]

Yes, I was thinking the same thing, to support the payload being sent directly in the message body. Let me post a message on slack.