Skip to content

Workshop to show case data ingestion from multiple sources mostly tagged to retail industry , processing and visualization

Notifications You must be signed in to change notification settings

PoonamSampat/BigDataWorkshop

Repository files navigation

Big data and visualization

Use case -

Convert unstructured raw data, into more conformed business schema and answer business questions.

We will be ingesting data from two different sources:

  • Magento
  • Shopify

Creating a data pipeline to store the raw data, process the data and store the data in SnowFlakes as single fact and dimension. Then we will visualize the data to understand the ordering pattern from these two channels.

February 2020

Target audience

  • Data engineers
  • Data architects

Abstracts

Workshop

At the end of this workshop, you will be better able to build a complete data pipeline and visualize the data. In addition, you will learn how to ingest the data, use Azure Data Factory (ADF) for data movement and operationalizing the data pipeline, how to store the data in raw format, learn about Azure Data Lake Gen 2, Process the data using Databricks ,learn about Databricks auto clustering ,summarize data with Azure Databricks and PySpark, store the data into SnowFlake using Azure DataBricks Connector and visualize the data using Power BI.

Hands-on lab

This hands-on lab is designed to provide exposure to many of Microsoft's transformative line of business applications built using Microsoft big data and advanced analytics.

By the end of the lab, you will be able to show an end-to-end solution, leveraging many of these technologies, but not necessarily doing work in every component possible.

Azure services and related products

  • Azure Databricks
  • Azure Data Factory (ADF)
  • Azure Storage
  • Power BI Desktop
  • SnowFlake (Optional) can use a SQL Store to store structured data.

About

Workshop to show case data ingestion from multiple sources mostly tagged to retail industry , processing and visualization

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages