Pinned Loading

1. Runs and distributes an activated py... Runs and distributes an activated python environment to the spark cluster

   1

   # set spark home to EMR default

   2

   export SPARK_HOME='/usr/lib/spark/'

   3

   4

   # if want to use jupyter instead can specify "jupyter" as first arg

   5

   if [ -z $1]; then

2. Run an activated python env on spark... Run an activated python env on spark cluster

   1

   import os

   2

   import argparse

   3

   import sys

   4

   import pathlib

   5

   import shutil