Skip to content

NVIDIA/spark-rapids-examples

Repository files navigation

spark-rapids-examples

This is the RAPIDS Accelerator for Apache Spark examples repo. RAPIDS Accelerator for Apache Spark accelerates Spark applications with no code changes. You can download the latest version of RAPIDS Accelerator here. This repo contains examples and applications that showcases the performance and benefits of using RAPIDS Accelerator in data processing and machine learning pipelines. There are broadly five categories of examples in this repo:

  1. SQL/Dataframe
  2. Spark XGBoost
  3. Machine Learning/Deep Learning
  4. RAPIDS UDF
  5. Databricks Tools demo notebooks

For more information on each of the examples please look into respective categories.

Here is the list of notebooks in this repo:

Category Notebook Name Description
1 SQL/DF Microbenchmark Spark SQL operations such as expand, hash aggregate, windowing, and cross joins with up to 20x performance benefits
2 SQL/DF Customer Churn Data federation for modeling customer Churn with a sample telco customer data
3 XGBoost Agaricus (Scala) Uses XGBoost classifier function to create model that can accurately differentiate between edible and poisonous mushrooms with the agaricus dataset
4 XGBoost Mortgage (Scala) End-to-end ETL + XGBoost example to predict mortgage default with Fannie Mae Single-Family Loan Performance Data
5 XGBoost Taxi (Scala) End-to-end ETL + XGBoost example to predict taxi trip fare amount with NYC taxi trips data set
6 ML/DL PCA Spark-Rapids-ML based PCA example to train and transform with a synthetic dataset
7 ML/DL DL Inference 11 notebooks demonstrating distributed model inference on Spark using the predict_batch_udf across various frameworks: PyTorch, HuggingFace, and TensorFlow

Here is the list of Apache Spark applications (Scala and PySpark) that can be built for running on GPU with RAPIDS Accelerator in this repo:

Category Notebook Name Description
1 XGBoost Agaricus (Scala) Uses XGBoost classifier function to create model that can accurately differentiate between edible and poisonous mushrooms with the agaricus dataset
2 XGBoost Mortgage (Scala) End-to-end ETL + XGBoost example to predict mortgage default with Fannie Mae Single-Family Loan Performance Data
3 XGBoost Taxi (Scala) End-to-end ETL + XGBoost example to predict taxi trip fare amount with NYC taxi trips data set
4 ML/DL PCA Spark-Rapids-ML based PCA example to train and transform with a synthetic dataset
5 UDF URL Decode Decodes URL-encoded strings using the Java APIs of RAPIDS cudf
6 UDF URL Encode URL-encodes strings using the Java APIs of RAPIDS cudf
7 UDF CosineSimilarity Computes the cosine similarity between two float vectors using native code
8 UDF StringWordCount Implements a Hive simple UDF using native code to count words in strings