feat(jmespath): new built-in envelopes to unwrap S3 events

[leandrodamascena]

Issue number: #2168

Summary

Changes

In this pull request, 5 new envelopes were created to make the developer's experience simpler.

User experience

BEFORE

from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor
from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord
from aws_lambda_powertools.utilities.jmespath_utils import extract_data_from_envelope

processor = BatchProcessor(event_type=EventType.SQS)

def record_handler(record: SQSRecord):
    s3_record = extract_data_from_envelope(data=record.raw_event, envelope="powertools_json(body).powertools_json(Message).Records[0]")

    return True

@batch_processor(record_handler=record_handler, processor=processor)
def lambda_handler(event: dict, context: LambdaContext):
    return processor.response()

AFTER

from aws_lambda_powertools.utilities.typing import LambdaContext
from aws_lambda_powertools.utilities.batch import BatchProcessor, EventType, batch_processor
from aws_lambda_powertools.utilities.data_classes.sqs_event import SQSRecord
from aws_lambda_powertools.utilities.jmespath_utils import (
    envelopes,
    extract_data_from_envelope,
)

processor = BatchProcessor(event_type=EventType.SQS)

def record_handler(record: SQSRecord):
    s3_record = extract_data_from_envelope(data=record.raw_event, envelope=envelope=envelopes.S3_SNS_SQS)

    return True

@batch_processor(record_handler=record_handler, processor=processor)
def lambda_handler(event: dict, context: LambdaContext):
    return processor.response()

Payloads to test

S3->SNS->SQS

{
    "Records":[
       {
          "messageId":"deefbac6-344a-4db5-b7dd-35894575e7d4",
          "receiptHandle":"AQEB0OWCS1/EYMFz92sKdu7g+DdjchivKC4bZ+sltCNq6ZodCecJ4VYtjvqilEP8D/gxbo4sEk8tTFPAZLoCHyat52E+vbPANwFkcAhObAQRmwWX7BRGzs/zFfqCW2nt/DwY2A+ElkYrdkmxRLGjQcCRekJ0hAaZrSWAmnWfO4siquB4lRsxlHL6TzLaYXTQlbQ/K/1NsUZ+Tnb7WSpUKs7GN44RjzJGbvZKK6C2Bg6BRs+S32to5mfIHFjPB4Mu1+8ao9Rt0ChintH/NQO+93lsgzCeLbokhKVqIMAwqgqAVG7l9Jk51hkvMT3zXk4jnwGFX7RfLmtFdU3O4MNizQVucQL6nrHj44Gb4ldvL5Jq6lSjz1BA+j0zwexh3EH4mj+R",
          "body":"{\n  \"Type\" : \"Notification\",\n  \"MessageId\" : \"1cd1d74a-f20d-5b05-b96b-8b897458b85b\",\n  \"TopicArn\" : \"arn:aws:sns:us-east-1:200984112386:SQS\",\n  \"Subject\" : \"Amazon S3 Notification\",\n  \"Message\" : \"{\\\"Records\\\":[{\\\"eventVersion\\\":\\\"2.1\\\",\\\"eventSource\\\":\\\"aws:s3\\\",\\\"awsRegion\\\":\\\"us-east-1\\\",\\\"eventTime\\\":\\\"2023-04-26T20:05:39.433Z\\\",\\\"eventName\\\":\\\"ObjectCreated:Put\\\",\\\"userIdentity\\\":{\\\"principalId\\\":\\\"A1YQ72UWCM96UF\\\"},\\\"requestParameters\\\":{\\\"sourceIPAddress\\\":\\\"93.108.161.96\\\"},\\\"responseElements\\\":{\\\"x-amz-request-id\\\":\\\"ZPVZFYFWS102RWZ7\\\",\\\"x-amz-id-2\\\":\\\"AuER6ogIVeGOPF7bGWzi+U441aSBpLFf+fObE3Iq7/EbPXX2tLPOR2V4xL6TeXYPzwDH8Qo4PmX0Aj5Idfm7fIj48lw5FgZJ\\\"},\\\"s3\\\":{\\\"s3SchemaVersion\\\":\\\"1.0\\\",\\\"configurationId\\\":\\\"SNS\\\",\\\"bucket\\\":{\\\"name\\\":\\\"xxx\\\",\\\"ownerIdentity\\\":{\\\"principalId\\\":\\\"A1YQ72UWCM96UF\\\"},\\\"arn\\\":\\\"arn:aws:s3:::xxx\\\"},\\\"object\\\":{\\\"key\\\":\\\"xxx.pdf\\\",\\\"size\\\":103424,\\\"eTag\\\":\\\"cd3ae8d2aca42cb951566a82c615b639\\\",\\\"versionId\\\":\\\"FbCq_E1uOATR01o.H2ewD8KfTlxqoHCm\\\",\\\"sequencer\\\":\\\"0064498413657AEFCF\\\"}}}]}\",\n  \"Timestamp\" : \"2023-04-26T20:05:41.101Z\",\n  \"SignatureVersion\" : \"1\",\n  \"Signature\" : \"RWcOmgOXLll01SCYMGHVL+Y5KtEDmFojlQ3DFYIDeAjr7Y2uQZm90hftD+OcYkLucRg1hEXljJAZnhBT5dTi8M7PVvAm+MEND0T8/s9JHidd1/TR3bB5BBPRkr0qe5D9bVsIZV/dZ1tAbOHPHYT8oBwXYeJ0CjUVKoOKb9TT6asvnx8nxq0V9qlY1b+hGojVtS92VMBXIj4WJRWI1XwSk2WRd+RYxRPuD4b9517SEgCdzdF+vuq/5FV/jAJudw0mT1oHE+hkZ315k5mTO3rIh9xiLQaKVUqTF01e16fnzLz798JYv3sazAydXjRkBhMP+h9Nj8hxadEZWMkPqryroA==\",\n  \"SigningCertURL\" : \"https://sns.us-east-1.amazonaws.com/SimpleNotificationService-56e67fcb41f6fec09b0196692625d385.pem\",\n  \"UnsubscribeURL\" : \"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:200984112386:SQS:4a2496a5-7b8b-4039-959a-3d3729b1d790\"\n}",
          "attributes":{
             "ApproximateReceiveCount":"1",
             "SentTimestamp":"1682539541154",
             "SenderId":"AIDAIT2UOQQY3AUEKVGXU",
             "ApproximateFirstReceiveTimestamp":"1682539561154"
          },
          "messageAttributes":{
             
          },
          "md5OfBody":"19bc26cd974eae7362ec27e31a9eefc2",
          "eventSource":"aws:sqs",
          "eventSourceARN":"arn:aws:sqs:us-east-1:xxx:SQS",
          "awsRegion":"us-east-1"
       }
    ]
 }

S3->SNS(raw delivery message enabled)->SQS

{
    "Records":[
       {
          "messageId":"fc1d0d34-a507-4e10-b3e6-b501eb8f200c",
          "receiptHandle":"AQEBlkznH+C0zHuv6y453tltB30zTv+MpfH5Le3FSI2spABjWQmQXuMKBsE4ps/Fgh5tQ1VDnavknuW/dr5LcdCUc00E0wYF0sm9SiR35PxUs6Ybf1BzcGUxIAVPL+QV1/fBMRA3nvqY5B/gBoHvJX2uKcs9FNid0PtnOb11DQYh7FtkvzjufRXyTs0znLM2t7E1PJtwEghJr5z6yfiOlB1xw4K18av6RWOkG4dWIVzrNnHJKrmAUyHutZ6p+SdCMjucmR3MEEL848FPVUkUDlpg+tTCIQOMxjSr81FQFn0CJLaAFuoStN2+HoqfL+Uu78j1N1uCJHRIHUe0ZPa861nYtm0zcPX3Sr9Gswbku7bTnrkkd3VBgBQPl7t6+6kt1zGR",
          "body":"{\"Records\":[{\"eventVersion\":\"2.1\",\"eventSource\":\"aws:s3\",\"awsRegion\":\"us-east-1\",\"eventTime\":\"2023-02-14T16:48:40.076Z\",\"eventName\":\"ObjectCreated:Put\",\"userIdentity\":{\"principalId\":\"A1YQ72UWCM96UF\"},\"requestParameters\":{\"sourceIPAddress\":\"93.108.161.96\"},\"responseElements\":{\"x-amz-request-id\":\"APCDMT4D1SBRGWJW\",\"x-amz-id-2\":\"dC8zU2g7TFS8p24lRvIgcFBAd1snHSs/1mrZ94oexCqDAzbVigpkUxrwbtzFU2IP0TcHq+8LIgRmFPhohYMn4KuS1DFXKHKSKruegMu42xM=\"},\"s3\":{\"s3SchemaVersion\":\"1.0\",\"configurationId\":\"SNS\",\"bucket\":{\"name\":\"xxx\",\"ownerIdentity\":{\"principalId\":\"A1YQ72UWCM96UF\"},\"arn\":\"arn:aws:s3:::xxx\"},\"object\":{\"key\":\"xxx.pdf\",\"size\":683156,\"eTag\":\"01409bb790f925321c619170a4002cd6\",\"versionId\":\"6hIIuveLWIMHVD5qszL7MaDlsOxHOvVM\",\"sequencer\":\"0063EBBB67F02C246E\"}}}]}",
          "attributes":{
             "ApproximateReceiveCount":"1",
             "SentTimestamp":"1676393321478",
             "SenderId":"AIDAIT2UOQQY3AUEKVGXU",
             "ApproximateFirstReceiveTimestamp":"1676393321485"
          },
          "messageAttributes":{
             
          },
          "md5OfBody":"c17a39d9858961037e8d748de4c39f9d",
          "eventSource":"aws:sqs",
          "eventSourceARN":"arn:aws:sqs:us-east-1:xxx:SQS",
          "awsRegion":"us-east-1"
       }
    ]
 }

S3->SNS->Kinesis Firehose

{
    "invocationId":"064dbbd7-d5b9-42b4-8548-e2c87b49ad68",
    "deliveryStreamArn":"arn:aws:firehose:us-east-1:xxxx:deliverystream/PUT-S3-yltjg",
    "region":"us-east-1",
    "records":[
       {
          "recordId":"49640211073817327783046060903306588296500113445104910338000000",
          "approximateArrivalTimestamp":1682550045572,
          "data":"eyJUeXBlIjoiTm90aWZpY2F0aW9uIiwiTWVzc2FnZUlkIjoiZGUyOWE1NmItZDhhYy01NDQwLWI4NDctZjlhODc3ZjFiNTFjIiwiVG9waWNBcm4iOiJhcm46YXdzOnNuczp1cy1lYXN0LTE6MjAwOTg0MTEyMzg2OlNRUyIsIlN1YmplY3QiOiJBbWF6b24gUzMgTm90aWZpY2F0aW9uIiwiTWVzc2FnZSI6IntcIlJlY29yZHNcIjpbe1wiZXZlbnRWZXJzaW9uXCI6XCIyLjFcIixcImV2ZW50U291cmNlXCI6XCJhd3M6czNcIixcImF3c1JlZ2lvblwiOlwidXMtZWFzdC0xXCIsXCJldmVudFRpbWVcIjpcIjIwMjMtMDQtMjZUMjM6MDA6NDQuMjYwWlwiLFwiZXZlbnROYW1lXCI6XCJPYmplY3RDcmVhdGVkOlB1dFwiLFwidXNlcklkZW50aXR5XCI6e1wicHJpbmNpcGFsSWRcIjpcIkExWVE3MlVXQ005NlVGXCJ9LFwicmVxdWVzdFBhcmFtZXRlcnNcIjp7XCJzb3VyY2VJUEFkZHJlc3NcIjpcIjkzLjEwOC4xNjEuOTZcIn0sXCJyZXNwb25zZUVsZW1lbnRzXCI6e1wieC1hbXotcmVxdWVzdC1pZFwiOlwiMDBYQ0FWNEpaTVlHMjlEWlwiLFwieC1hbXotaWQtMlwiOlwiUHI2U2U5eGdSaEYvMFY5OFhzZlMvMENCaStxczlubDN5OXhXVW9FOUlBRmtMc0pld3ZVYUUrTGtqSSsvTmg5VTN6cVBXQlZGQ25VWDhOR3NpcTg5RnIzZDJmaUw2amkzXCJ9LFwiczNcIjp7XCJzM1NjaGVtYVZlcnNpb25cIjpcIjEuMFwiLFwiY29uZmlndXJhdGlvbklkXCI6XCJkZGRkXCIsXCJidWNrZXRcIjp7XCJuYW1lXCI6XCJhYWFhYmJjY2NkZGRkXCIsXCJvd25lcklkZW50aXR5XCI6e1wicHJpbmNpcGFsSWRcIjpcIkExWVE3MlVXQ005NlVGXCJ9LFwiYXJuXCI6XCJhcm46YXdzOnMzOjo6YWFhYWJiY2NjZGRkZFwifSxcIm9iamVjdFwiOntcImtleVwiOlwibGVvL2UyZS91dGlscy9fX3B5Y2FjaGVfXy9hdXRoLmNweXRob24tMzkucHljXCIsXCJzaXplXCI6NzMxLFwiZVRhZ1wiOlwiNWMxZGE3ODYzNDYzMGRjNTU2YmI1MDJkNWNmOGRmZGJcIixcInZlcnNpb25JZFwiOlwiWllaaWhMa2g3X2V4V2paT1d3dmhremZscWFBY2w4aTNcIixcInNlcXVlbmNlclwiOlwiMDA2NDQ5QUQxQzM1RjJCOThDXCJ9fX1dfSIsIlRpbWVzdGFtcCI6IjIwMjMtMDQtMjZUMjM6MDA6NDUuNTExWiIsIlVuc3Vic2NyaWJlVVJMIjoiaHR0cHM6Ly9zbnMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vP0FjdGlvbj1VbnN1YnNjcmliZSZTdWJzY3JpcHRpb25Bcm49YXJuOmF3czpzbnM6dXMtZWFzdC0xOjIwMDk4NDExMjM4NjpTUVM6ZjIwNmI0ZmItNmQ3YS00NmQxLTlhODktNGU2ZjRkOGJjYTNkIn0K"
       }
    ]
 }

S3->SNS(raw delivery message enabled)->Kinesis Firehose

{
   "invocationId":"2b93ed6a-4f9a-4b9e-80a8-bbaa78167ef9",
   "deliveryStreamArn":"arn:aws:firehose:us-east-1:200984112386:deliverystream/PUT-S3-yltjg",
   "region":"us-east-1",
   "records":[
      {
         "recordId":"49640211073817327783046060904329339539894136249992085506000000",
         "approximateArrivalTimestamp":1682550723287,
         "data":"eyJSZWNvcmRzIjpbeyJldmVudFZlcnNpb24iOiIyLjEiLCJldmVudFNvdXJjZSI6ImF3czpzMyIsImF3c1JlZ2lvbiI6InVzLWVhc3QtMSIsImV2ZW50VGltZSI6IjIwMjMtMDQtMjZUMjM6MTI6MDIuNDQzWiIsImV2ZW50TmFtZSI6Ik9iamVjdENyZWF0ZWQ6UHV0IiwidXNlcklkZW50aXR5Ijp7InByaW5jaXBhbElkIjoiQTFZUTcyVVdDTTk2VUYifSwicmVxdWVzdFBhcmFtZXRlcnMiOnsic291cmNlSVBBZGRyZXNzIjoiOTMuMTA4LjE2MS45NiJ9LCJyZXNwb25zZUVsZW1lbnRzIjp7IngtYW16LXJlcXVlc3QtaWQiOiI3MzkyQVo0OE5NSDRKOUVZIiwieC1hbXotaWQtMiI6ImZNeHNZaTFtYjdjZ0ozNDUvNHhqMEdtRzBHd2pDTFhsOXYreHNWOFRWZTNUU3pkZ2JmYm9OQ0Vla1pZNzhhUkhwMU9QN2NoR0VYYzVkUUppa1JpUnVoNzBFYmtnYmgvWSJ9LCJzMyI6eyJzM1NjaGVtYVZlcnNpb24iOiIxLjAiLCJjb25maWd1cmF0aW9uSWQiOiJkZGRkIiwiYnVja2V0Ijp7Im5hbWUiOiJhYWFhYmJjY2NkZGRkIiwib3duZXJJZGVudGl0eSI6eyJwcmluY2lwYWxJZCI6IkExWVE3MlVXQ005NlVGIn0sImFybiI6ImFybjphd3M6czM6OjphYWFhYmJjY2NkZGRkIn0sIm9iamVjdCI6eyJrZXkiOiJsZW8vZTJlL3V0aWxzL2RhdGFfZmV0Y2hlci9fX3B5Y2FjaGVfXy9tZXRyaWNzLmNweXRob24tMzkucHljIiwic2l6ZSI6MjM4NCwiZVRhZyI6IjgwYzM5ZjYzYWM5ZWE1MWRlZjJhMzk4MGM0NDhlNjRiIiwidmVyc2lvbklkIjoiWVB2TlVtUVIuRTVqWVY1eVd1NTQzNXB0OFVySEhMalciLCJzZXF1ZW5jZXIiOiIwMDY0NDlBRkMyNjFGNEYzNTkifX19XX0="
      }
   ]
}

S3->EventBridge->SQS

{
    "Records":[
       {
          "messageId":"8a39d5ec-3a82-4ffb-9c2b-8c1506983f98",
          "receiptHandle":"AQEBwgzrAtZhxdYYHNQdbUK9bHVQrk/bO/kXqdxuJ8IyfBstaTo2yigXmvuebqxAlbkgYLfvrQZl36JfZ4ZJ/oe4MKRY94H0HfN7K5byxQvGvNtXZ4LMEYuGa1CTL5zgS28jihj/hcfdt6OOHbmnxVlJz07nTGf+QH3wFTAuVEMDE96aEX2YUtP8TtX1c3+UxbaVRWuXWu41aoq6G10G0Nez1XkAUmUFsn5cTrGURNXksIVbccOLdGNG8coXaH1vwqxCz6ExavLXE5UegMoppLty2sSFB8LSzBVdFkDcQ0g4cSkJsrd/GaQwZN5E8fKm0lcIKN+tDNIDAoJE4G16ylAVTtBla+vI1n7tImQpvP4BeLZ0e7dXKPtIC7ArFqKu5ZYQ",
          "body":"{\"version\":\"0\",\"id\":\"680166b5-82a6-f60a-d99f-71a9d41dbb33\",\"detail-type\":\"Object Created\",\"source\":\"aws.s3\",\"account\":\"200984112386\",\"time\":\"2023-04-26T21:18:05Z\",\"region\":\"us-east-1\",\"resources\":[\"arn:aws:s3:::xxx\"],\"detail\":{\"version\":\"0\",\"bucket\":{\"name\":\"aaaabbcccdddd\"},\"object\":{\"key\":\"xx-xxx.pdf\",\"size\":9468,\"etag\":\"5f9ca96e2ed74b596b3754325c069812\",\"version-id\":\"_S90fg_wD3dgMCVp5xcLdlwEFYtCIVpM\",\"sequencer\":\"006449950D7FBE1F45\"},\"request-id\":\"71F7NXFJ1RA274BQ\",\"requester\":\"200984112386\",\"source-ip-address\":\"93.108.161.96\",\"reason\":\"PutObject\"}}",
          "attributes":{
             "ApproximateReceiveCount":"1",
             "SentTimestamp":"1682543887329",
             "SenderId":"AIDAJXNJGGKNS7OSV23OI",
             "ApproximateFirstReceiveTimestamp":"1682543907329"
          },
          "messageAttributes":{
             
          },
          "md5OfMessageAttributes":"None",
          "md5OfBody":"3bffe7eef50ac2a32e5eb4725e8093d5",
          "eventSource":"aws:sqs",
          "eventSourceARN":"arn:aws:sqs:us-east-1:xxx:SNS",
          "awsRegion":"us-east-1"
       }
    ]
 }

Checklist

If your change doesn't seem to apply, please leave them unchecked.

• Meet tenets criteria
• I have performed a self-review of this change
• Changes have been tested
• Changes are documented
• PR title follows conventional commit semantics

Is this a breaking change?

RFC issue number:

Checklist:

• Migration process documented
• Implement warnings (if it can live side by side)

Acknowledgment

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Disclaimer: We value your time and bandwidth. As such, any pull requests created on non-triaged issues might not be successful.

[heitorlessa]

Looks great!!! THANK YOU for that @leandrodamascena ;)

One necessary change, and two suggestions to make this better.

[heitorlessa]

Niiiice!! You can make it slightly better by:

• Adding a title so ESL readers know upfront if they should care or not e.g., "Using SNS?"
• Educate the reader that if they do need SNS event metadata they don't need to enable it

Write, review, cut the unnecessary words without losing its original intent, and ship it <3

[leandrodamascena]

• Added the title
• I thought of the following mental model to improve this text, and see if it makes sense.
  • Benefits - Reduce payload size
  • How to do - enabling raw message
  • why? - architectures that require sending event notifications from S3 to SNS but not require SNS metadata

[heitorlessa]

Consider rewriting the PR title so anyone reading later know what this provides; play around with words

[codecov-commenter]

Codecov Report

Patch and project coverage have no change.

Comparison is base (4663745) 97.46% compared to head (a5152dc) 97.46%.

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #2169   +/-   ##
========================================
  Coverage    97.46%   97.46%           
========================================
  Files          147      147           
  Lines         6872     6872           
  Branches       505      505           
========================================
  Hits          6698     6698           
  Misses         137      137           
  Partials        37       37           

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

[leandrodamascena]

Looks great!!! THANK YOU for that @leandrodamascena ;)

One necessary change, and two suggestions to make this better.

Thank you, @heitorlessa! Can you check again pls?