Skip to content

Latest commit

 

History

History
24 lines (21 loc) · 652 Bytes

README.md

File metadata and controls

24 lines (21 loc) · 652 Bytes

pyspark-profile-udf. Add simple udf profiler for pyspark.

  • Requirements

    1. flameprof
    pip install flameprof
  • How to use

    1. Rename this project to pyspark
    2. Compress this project to pyspark.zip: zip -r pyspark pyspark And move it to $SPARK_HOME/python/lib
    3. Enable pyspark profile in your application code, and set the profile dump path
    spark_session = SparkSession \
          .builder \
          .appName("Python Arrow-in-Spark profile") \
          .config("spark.python.profile", "true") \
          .config("spark.python.profile.dump", profile_dump_path) \
          .getOrCreate()