docker-spark

A basic dockerfile for messing around with pyspark standalone.

• Spark version: 2.1.0
• Python version: 3.5

Launch spark shell

docker-compose run spark bin/spark-shell

Launch pyspark shell with ipython

docker-compose run -e PYSPARK_PYTHON=ipython spark bin/pyspark

Run spark job

docker-compose run spark bin/spark-submit scripts/word_count.py data/words.txt