generated from kestra-io/plugin-template
-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Amazon Kinesis #43
Labels
enhancement
New feature or request
Comments
Proposed task: Proposed syntax: id: kinesis_data_streams_producer
namespace: dev
tasks:
- id: records
type: io.kestra.plugin.aws.kinesisdatastreams.PutRecords
streamName: required string
records: # required list of maps or JSON string in a structure as below:
- data: my custom blob object e.g. "user sign-in event"
partitionKey: required string for sharding e.g. "user1"
explicitHashKey: optional hash value overriding the partition key
- data: user sign-out event
partitionKey: user1
records2: | # only to show JSON-string payload to "records"
[
{
"Data": "user sign-in event",
"PartitionKey": "user1"
},
{
"Data": "user sign-out event",
"PartitionKey": "user1"
}
]
in theory, we could offer streamArn: |
required string if streamName is not provided.
There must always be either streamName or StreamArn set.
If this is too difficult to handle, we can provide just the "stream" property and detect whether this is ARN based on the pattern arn:aws.*:kinesis:.*:\d{12}:stream/\S+ Links: |
moadibfr
pushed a commit
that referenced
this issue
Sep 25, 2023
moadibfr
pushed a commit
that referenced
this issue
Sep 25, 2023
moadibfr
pushed a commit
that referenced
this issue
Sep 25, 2023
moadibfr
pushed a commit
that referenced
this issue
Sep 26, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Example scenario: consume streams from a Kinesis Data Stream, filter/manipulate data and publish into another Kinesis stream using a produce(r) task in a similar way how AWS EventBridge Pipes does that with Lambda functions https://aws.amazon.com/eventbridge/pipes/
The text was updated successfully, but these errors were encountered: