Tutorials and examples of how to deploy Presto and connect it to different data sources

This project offers a robust data pipeline solution designed to efficiently extract, transform, and load (ETL) Reddit data into a Redshift data warehouse. Leveraging a blend of industry-standard tools and services, the pipeline ensures seamless data processing and integration.

This project demonstrates how you can build downstream data pipeline using dbt in athena

This project focuses on real-time data streaming with Kinesis, using Flink for advanced processing and OpenSearch for analytics. This architecture has succinctly handled the complete lifecycle of data from ingestion to actionable insights, making it a comprehensive solution.

This projects uses ETL (Extract, Transform and Load) pipeline to extract data from Spotify using its API and loads the data to a data source(AWS Athena). The entire pipeline will be built using Amazon Web Services (AWS).

In this project I have used the Trending YouTube Video Statistics data from Kaggle to analyze and prepare it for usage.

This project showcases a data transformation pipeline utilizing AWS Glue and Amazon Athena to process Spotify data from CSV files. It involves loading, transforming, and storing data in an S3 datawarehouse, enabling seamless querying through Amazon Athena.

Big data and Cloud Deployment

Data Engineering Specialization offered by Joe Reis in partnership with DeepLearning.AI through Coursera...

ETL pipeline that transforms publicly available id304b covered entity (CE) data and loads to a relational database with change data capture. Uses AWS services to automate the pipeline.

Transformed YouTube’s raw JSON data to parquet & loaded it in an S3 bucket, used Glue Data Catalog for storing metadata & Athena to query the cleaned data. Developed an ETL process using a Lambda job that would be triggered when raw data is loaded into an S3 bucket, processed, and stored for analytical purposes in an S3 bucket.

This project sets up a real-time data pipeline to fetch data from Reddit, transform it using AWS Glue, and store it in Amazon S3. This involves data streaming, cloud storage, ETL (Extract, Transform, Load) processes, and orchestration using Apache Airflow.

This project demonstrates the use of Amazon Web Services (AWS) to analyze superstore sales data. The analysis was performed using AWS S3 for data storage, AWS Glue for data cataloging, AWS Athena for SQL-based serverless data querying, and AWS Quick Sight for visualization. The project’s objective was to provide actionable insights into sales trend

AWS S3 & Sentiment Analysis, Basic Plotting with Matplotlib, & Supervised Learning & Machine Learning with Sklearn.

A real-time stock market data analysis project using Apache Kafka⚡, Python, and AWS. It streams live stock data, processes it in real-time, and stores it in AWS S3 for further analysis using AWS Glue and Athena. This project showcases a scalable, cloud-native architecture for handling financial data efficiently

In this project we can run an ETL in AWS Glue by Orchestrating it with Airflow. This project we create a Docker Compose to raise the services as Airflow, Redis and PostgreSQL. PostgreSQL was use in this project to save metadata get of Airflow

This project aims to securely manage, process, and analyze structured and semi-structured YouTube data based on video categories and trending metrics. The architecture leverages AWS services to ingest, store, transform, analyze, and visualize data efficiently and at scale.

Leverages Apache Kafka to facilitate streaming real time data generated by Python to upload data into S3 using s3fs

Create Glue table using CI -CD

Incremental Data Load from S3 Bucket to Amazon Redshift Using AWS Glue