Skip to content

feat(data-classes): Add KinesisFirehoseEvent #1059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
michaelbrewer opened this issue Mar 6, 2022 · 1 comment
Closed

feat(data-classes): Add KinesisFirehoseEvent #1059

michaelbrewer opened this issue Mar 6, 2022 · 1 comment
Labels
feature-request feature request

Comments

@michaelbrewer
Copy link
Contributor

michaelbrewer commented Mar 6, 2022

Is your feature request related to a problem? Please describe.

AWS Lambda Powertools provides a nearly complete set of utilities for batch processing, but not Kinesis Data Firehose. Within our org having support for this in the Python version would be very useful. And in the long run including support for a unified batch processing.

Describe the solution you'd like

Add support for a KinesisFirehoseEvent data class for request and response. As well has some payload decoding utilities

  • data - can include support for payload decoding and caching
  • invocationId - can be added as a correlation path
  • result - in the response can be an enum ( 'Ok', 'Dropped', 'ProcessingFailed')

Describe alternatives you've considered

Building an unofficial data class, which does not make sense as their are already official ones for Java and Go.

Additional context

  • Amazon Kinesis Data Firehose Request message event
{
  "invocationId": "invoked123",
  "deliveryStreamArn": "aws:lambda:events",
  "region": "us-west-2",
  "records": [
    {
      "data": "SGVsbG8gV29ybGQ=",
      "recordId": "record1",
      "approximateArrivalTimestamp": 1510772160000,
      "kinesisRecordMetadata": {
        "shardId": "shardId-000000000000",
        "partitionKey": "4d1ad2b9-24f8-4b9d-a088-76e9947c317a",
        "approximateArrivalTimestamp": "2012-04-23T18:25:43.511Z",
        "sequenceNumber": "49546986683135544286507457936321625675700192471156785154",
        "subsequenceNumber": ""
      }
    },
    {
      "data": "SGVsbG8gV29ybGQ=",
      "recordId": "record2",
      "approximateArrivalTimestamp": 151077216000,
      "kinesisRecordMetadata": {
        "shardId": "shardId-000000000001",
        "partitionKey": "4d1ad2b9-24f8-4b9d-a088-76e9947c318a",
        "approximateArrivalTimestamp": "2012-04-23T19:25:43.511Z",
        "sequenceNumber": "49546986683135544286507457936321625675700192471156785155",
        "subsequenceNumber": ""
      }
    }
  ]
}
  • Amazon Kinesis Data Firehose Request ApacheLog
{
  "invocationId": "invocationIdExample",
  "region": "us-east-1",
  "records": [
    {
      "recordId": "49546986683135544286507457936321625675700192471156785154",
      "approximateArrivalTimestamp": 1495072949453,
      "data": "NjQuMjQyLjg4LjEwIC0gLSBbMDcvTWFyLzIwMDQ6MTY6MTA6MDIgLTA4MDBdICJHRVQgL21haWxtYW4vbGlzdGluZm8vaHNkaXZpc2lvbiBIVFRQLzEuMSIgMjAwIDYyOTE=="
    }
  ]
}
  • Response example
{
  "records": [
    {
      "recordId": "49572672223665514422805246926656954630972486059535892482",
      "result": "Ok",
      "data": "SEVMTE8gV09STEQ=",
      "metadata": {
        "partitionKeys": {
          "customerId": "customerId",
          "year": "2017",
          "month": "01",
          "date": "01",
          "hour": "01",
          "minute": "01"
        }
      }
    }
  ]
}

Other language data classes:

@michaelbrewer michaelbrewer added feature-request feature request triage Pending triage from maintainers labels Mar 6, 2022
@sthulb sthulb closed this as completed Jun 14, 2022
@github-actions
Copy link
Contributor

Comments on closed issues are hard for our team to see.

@heitorlessa heitorlessa removed the triage Pending triage from maintainers label Feb 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request feature request
Projects
None yet
Development

No branches or pull requests

3 participants