-
Requirements
- flameprof
pip install flameprof
-
How to use
- Rename this project to pyspark
- Compress this project to pyspark.zip:
zip -r pyspark pyspark
And move it to $SPARK_HOME/python/lib - Enable pyspark profile in your application code, and set the profile dump path
spark_session = SparkSession \ .builder \ .appName("Python Arrow-in-Spark profile") \ .config("spark.python.profile", "true") \ .config("spark.python.profile.dump", profile_dump_path) \ .getOrCreate()