Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(kinesisfirehose-destinations): add support for prefixes in the S3 destination #15552

Merged
merged 121 commits into from
Jul 28, 2021

Conversation

madeline-k
Copy link
Contributor

Closes #15551


By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license

@gitpod-io
Copy link

gitpod-io bot commented Jul 14, 2021

@mergify mergify bot added the contribution/core This is a PR that came from AWS. label Jul 14, 2021
@BenChaimberg BenChaimberg requested a review from skinny85 July 28, 2021 03:23
BenChaimberg
BenChaimberg previously approved these changes Jul 28, 2021
skinny85
skinny85 previously approved these changes Jul 28, 2021
packages/@aws-cdk/aws-kinesisfirehose/README.md Outdated Show resolved Hide resolved
@skinny85 skinny85 changed the title feat(aws-kinesisfirehose-destinations): support prefixes for S3 destination feat(kinesisfirehose-destinations): support prefixes for S3 destination Jul 28, 2021
Base automatically changed from madeline-k/feat/kinesisfirehose-compression to master July 28, 2021 21:58
@mergify mergify bot dismissed stale reviews from skinny85 and BenChaimberg via e0f7901 July 28, 2021 21:58
@madeline-k madeline-k removed the pr/do-not-merge This PR should not be merged at this time. label Jul 28, 2021
@skinny85 skinny85 changed the title feat(kinesisfirehose-destinations): support prefixes for S3 destination feat(kinesisfirehose-destinations): add support for prefixes in the S3 destination Jul 28, 2021
@aws-cdk-automation
Copy link
Collaborator

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject89A8053A-LhjRyN9kxr8o
  • Commit ID: 929f9bb
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@mergify mergify bot merged commit d227e48 into master Jul 28, 2021
@mergify mergify bot deleted the madeline-k/feat/kinesisfirehose-prefixes branch July 28, 2021 22:45
@mergify
Copy link
Contributor

mergify bot commented Jul 28, 2021

Thank you for contributing! Your pull request will be updated from master and then merged automatically (do not update manually, and be sure to allow changes to be pushed to your fork).

TikiTDO pushed a commit to TikiTDO/aws-cdk that referenced this pull request Aug 3, 2021
…3 destination (aws#15552)

Closes aws#15551 

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
hollanddd pushed a commit to hollanddd/aws-cdk that referenced this pull request Aug 26, 2021
…3 destination (aws#15552)

Closes aws#15551 

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@jakejscott
Copy link

@madeline-k

I needed to add property overrides to the ExtendedS3DestinationConfiguration because the dataOutputPrefix and errorOutputPrefix props contained partitionKeyFromQuery fragments.

const s3Destination = new destinations.S3Bucket(s3DestinationBucket, {
  logging: true,
  logGroup: s3DestinationLogGroup,
  bufferingInterval: cdk.Duration.seconds(60),
  bufferingSize: cdk.Size.mebibytes(64),
  compression: undefined,
  dataOutputPrefix:
    "user/!{partitionKeyFromQuery:userId}/order/!{partitionKeyFromQuery:orderId}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/!{timestamp:HH}/",
  errorOutputPrefix:
    "failed/!{firehose:error-output-type}/!{timestamp:yyyy}/!{timestamp:MM}/!{timestamp:dd}/!{timestamp:HH}/",
});

const s3DeliveryStream = new kinesisfirehose.DeliveryStream(
  this,
  "S3DeliveryStream",
  {
    destinations: [s3Destination],
  }
);

s3DeliveryStreamEscapeHatch.addPropertyOverride(
  "ExtendedS3DestinationConfiguration.DynamicPartitioningConfiguration",
  {
    Enabled: true,
    RetryOptions: {
      DurationInSeconds: 300,
    },
  }
);

s3DeliveryStreamEscapeHatch.addPropertyOverride(
  "ExtendedS3DestinationConfiguration.ProcessingConfiguration",
  {
    Enabled: true,
    Processors: [
      {
        Type: "MetadataExtraction",
        Parameters: [
          {
            ParameterName: "MetadataExtractionQuery",
            ParameterValue: "{userId:.userId,orderId:.orderId}",
          },
          {
            ParameterName: "JsonParsingEngine",
            ParameterValue: "JQ-1.6",
          },
        ],
      },
    ],
  }
);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contribution/core This is a PR that came from AWS.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

(aws-kinesisfirehose): prefixes for delivery stream S3 destination
5 participants