Skip to content

Latest commit

 

History

History
118 lines (79 loc) · 4.97 KB

README.md

File metadata and controls

118 lines (79 loc) · 4.97 KB
Raw

Taming Pyspark

Completely the initial learning of running PySpark jobs locally and in a cluser

Environment Variables

BASE_DATA_PATH= # data directory relative to root project directory (ex. BASE_DATA_PATH=taming_pyspark/data)
MOVIE_LENS= 
FRIENDS=
TEMP_1800S=
WORD_COUNT=
CONSUMER_SPENDING=
HEROES=
  • Based on the folder names in the data.zip file
  • See config.py

Navigating taming_pyspark directory

  • Section describes what each folder focuses on

taming_pyspark/utils

Objective:

  • Location for functions used in multiple locations

taming_pyspark/book

Objective: Count unique words in the word_count/Book.txt file in data.zip

Files Topic Additional Notes
word_count_rdd.py - RDD
- flatMap		 - Data in the word_count in the data.zip file
- Word count includes special characters
word_count_df.py - Dataframes
- RDD to Dataframes
- Sorting		 - Data in the word_count in the data.zip file
- Word count includes special characters
word_count_regex_rdd.py - RDD
- Regular Expressions
- Sorting
- re module		 - Normalize words and remove special characters
word_count_regex_df.py - Dataframes
- RDD to Dataframes
- Regular Expressions
- Sorting
- re module		 - Normalize words and remove special characters

taming_pyspark/consumer_spending

Objective:

  • Use the customer-purchases/customer_orders.csv file in data.zip
  • Find the total amount of spend by consumer id
  • Sort by consumer id total spend
Files Topic Additional Notes
spending_df.py - Dataframes
- Aggregrate Functions
spending_rdd.py - RDDs
- reduceByKey

taming_pyspark/friends

Objective:

  • Use the friends-dataset/fakefriends.csv file in data.zip
  • Psuedo friends/peers dataset
  • Use pyspark to determine number of friends/peers by age
Files Topic Additional Notes
fake_friends_df.py - Dataframes
- Aggregrate Functions
fake_friends_rdd.py - RDDs
- reduceByKey
- Text to CSV Parsing

taming_pyspark/heroes

Objective:

  • Use the friends-dataset/fakefriends.csv file in data.zip
  • Determines the most popular hero based on how many uniquely different hours they have had an encounter with
Files Topic Additional Notes
most_popular_hero.py - udf (user-defined functions)
- Dataframes
- Adding Columns (withColumns)
- RDD to DF

taming_pyspark/movies

Objective:

  • Uses the data found in the ml-100k/ folder in data.zip
  • Find the most watched movies
  • Count and group movies by rating
  • Recommend movie based on previous users using least squared algorithm
Files Topic Additional Notes
ratings_counter.py - RDD
- Text to CSV Parsing
- Ordered Dictionary
- countByValue
most_watched.py - DataFrames
- udf (user defined functions)
- udf with multiple inputs
- Passing Dictionary to UDF
recommendation.py - Dataframes
- Complex Aggregations
- Class Based PySpark Runs

taming_pyspark/streams

Objective:

  • Introduction to PySpark Structured Streaming
  • Log Ingestion for HTTP Requests
  • Uses the streams/logs.txt file in data.zip to be the sample logs
  • Run the stream and add a copy of the logs.txt file to the logs directory that gets created when running the script
Files Topic Additional Notes
stream_logs.py - Structured Streaming

taming_pyspark/temp_1800s

Objective:

  • Shows min weather temperature observed for each weather station by default
Files Topic Additional Notes
min_max_temp_df.py - DataFrames
- filter
min_max_temp_rdd.py - RDDs
- filter
- reduceByKey