Skip to content

A collection of examples built with AWS DataOps Development Kit (DDK)

License

Notifications You must be signed in to change notification settings

aws-samples/aws-ddk-examples

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

AWS DDK Examples

This repository contains a list of example projects for the AWS DataOps Development Kit (DDK).

Table of Contents

  1. About this Repo
  2. Examples
  3. Learning Resources
  4. License

About this Repo

This repository holds our official list of DDK examples code. They aim to showcase the different ways you can leverage the DDK to help with your DataOps workloads, including building a data pipeline to ingest SaaS data or to create your private code artifactory, for example.

We welcome contributions to this repository in the form of enhancements to existing examples or the addition of new ones. For more information on contributing, please see the CONTRIBUTING guide.

It is assumed that you are already somewhat familiar with the AWS DDK. If not, we strongly recommend that you go through our Learning Resources first, in particular the DDK workshop and documentation.

If you would like to start your journey by looking at an example, we recommend you to start with Kinesis to S3 Data Pipeline example from the list below.

Helper scipt for aws-ddk-examples

Create this optional directory for DDK patterns

mkdir directory_name
cd directory_name

Download the helper script using the below command

curl -LJO https://raw.githubusercontent.com/aws-samples/aws-ddk-examples/main/cli_helper.py

Once the script is downloaded, execute the script using the below command

pip install urllib3

For help regarding usage

python3 cli_helper.py -h

For list existing patterns

python3 cli_helper.py -t "list"

To init an available pattern into your directory

python3 cli_helper.py -t "init" -p "sdlf-ddk-lightweight" -l "python"

Earlier Versions of DDK See here for examples using the library before the 1.0.0 major version release.

Examples

Examples
Kinesis to S3 Data Pipeline
Google Analytics Appflow Data Pipeline
Athena Query Execution Pipeline
Athena Views Pipeline
DataBrew Athena Pipeline
Private Artifactory
Cross-Account / Cross-Region Data Pipelines
Data Validation & Cataloging Pipeline
SDLF DDK Lightweight
Simple Glue Tranform
File Standardization Pipeline

Learning Resources

Beyond this repository, there are other resources that can be referenced to assist with your learning/development process.

Official Resources

License

This library is licensed under the Apache 2.0 License.

About

A collection of examples built with AWS DataOps Development Kit (DDK)

Resources

License

Code of conduct

Security policy

Stars

Watchers

Forks