Magpie 3.2

Magpie 3.2

Add support for job submission via Flux.

• Note that Flux is currently in development and interfaces may vary
  between Flux versions. This support tested on flux-core v0.49.0.

Magpie 3.1

• Support new MAGPIE_HOSTNAME_CMD configuration, a simpler version of
  MAGPIE_HOSTNAME_CMD_MAP.
• Support Hadoop 3.2.4, 3.3.4, 3.3.5
  • Update Hadoop default to 3.3.5
• Support Spark 3.2.2, 3.2.3, 3.3.0, 3.3.1, 3.3.2
  • Update Spark default to 3.3.2-bin-hadoop3

Magpie 3.0

Magpie 3.0 includes some new software support, but primarily removes a lot of older support.

New Support

• Support Spark 3.2.1-bin-hadoop2.7, 3.2.1-bin-hadoop3.2, Spark 3.2.0-bin-hadoop2.7, 3.2.0-bin-hadoop3.2, Spark 3.1.3-bin-hadoop2.7, 3.1.3-bin-hadoop3.2, Spark 3.1.2-bin-hadoop2.7, 3.1.2-bin-hadoop3.2, Spark 3.1.1-bin-hadoop2.7, 3.1.1-bin-hadoop3.2
  • Update Spark default to 3.2.1-bin-hadoop3.2
• Support Hadoop 3.3.3, 3.3.2, 3.2.3
  • Update Hadoop default to 3.3.3

Update default MAGPIE_PYTHON to python3.

Legacy support Removed

• Spark 0.9.X support
• Hbase 0.98.X and 0.99.X support
• Mahout

Backwards Compatibility Notes

• Rename SPARK_SLAVE_COUNT to SPARK_WORKER_COUNT
• Rename SPARK_SLAVE_CORE_COUNT to SPARK_WORKER_CORE_COUNT
• Remove legacy support for HADOOP_SLAVE_COUNT and HADOOP_SLAVE_CORE_COUNT. HADOOP_WORKER_COUNT and HADOOP_WORKER_CORE_COUNT should be used.
• SPARK_JOB_CLASSPATH, SPARK_JOB_LIBRARY_PATH, and SPARK_JOB_JAVA_OPTS environment variables are removed. Users should use --driver-class-path, --driver-library-path, and --driver-java-options in spark-submit instead.
• Remove legacy support of STORM_WORKERS_COUNT (plural), users should use STORM_WORKER_COUNT.

Magpie 2.6

• Support Hadoop 3.1.4, 3.2.2, 3.3.0, 3.3.1
  • Update Hadoop default to 3.3.1
• Support Spark 3.0.3-bin-hadoop2.7, 3.0.3-bin-hadoop3.2, Spark 3.0.2-bin-hadoop2.7, 3.0.2-bin-hadoop3.2, Spark 3.0.1-bin-hadoop2.7, 3.0.1-bin-hadoop3.2, Spark 2.4.7-bin-hadoop2.6, 2.4.7-bin-hadoop2.7, Spark 2.4.8-bin-hadoop2.6, 2.4.8-bin-hadoop2.7
  • Update Spark default to 3.0.3-bin-hadoop3.2
• Misc fixes
• Documentation updates

Magpie 2.5

• Add Alluxio support.
• Support Spark 3.0.0-bin-hadoop2.7, 3.0.0-bin-hadoop3.2, Spark 2.4.6-bin-hadoop2.6, 2.4.6-bin-hadoop2.7
  • Update Spark default to 3.0.0-bin-hadoop3.2
• Support Hbase 1.2.7, 1.3.3, 1.3.4, 1.3.5, 1.5.0, 1.6.0.
  • Update Hbase default to 1.6.0

Magpie 2.4

• Support Hadoop 3.1.3, 3.2.1
  • Update Hadoop default to 3.2.1
• Add support for Storm 1.2.3.
  • Update default Storm to 1.2.3.
• Add support for Hbase 1.4.10, 1.4.13
  • Update default Hbase to 1.4.10, 1.4.13
• Add support for Zeppelin 0.8.2.
  • Update default Zeppelin to 0.8.2.
• Support Spark 2.3.3-bin-hadoop2.6, 2.3.3-bin-hadoop2.7, Spark 2.3.4-bin-hadoop2.6, 2.3.4-bin-hadoop2.7, Spark 2.4.5-bin-hadoop2.6, 2.4.5-bin-hadoop2.7
  • Update Spark default to 2.4.4-bin-hadoop2.7

Magpie 2.3

Add experimental support for Ray.

• Support Spark 2.4.2-bin-hadoop2.6, 2.4.2-bin-hadoop2.7, 2.4.3-bin-hadoop2.6, 2.4.3-bin-hadoop2.7
  • Update Spark default to 2.4.3-bin-hadoop2.7
• Add support for Zookeeper 3.4.14.
  • Update default Zookeeper to 3.4.14.

Magpie 2.2

• Support SPARK_IO_COMPRESSION_CODEC in Spark.
• Add new submission script type "sbatchmpirun"
• With sbatch & srun, add experimental support for distributed tensorflow support.
• With sbatch & mpirun, add experimental support for Tensorflow with Horovod
• Support new MAGPIE_PRE_EXECUTION_RUN and MAGPIE_POST_EXECUTION_RUN options. These options are similar to MAGPIE_PRE_JOB_RUN and MAGPIE_POST_JOB_RUN, but run before/after the entire software stack has been brought up / torn down.
• Support Spark 2.4.1-bin-hadoop2.6, 2.4.1-bin-hadoop2.7
• Update Spark default to 2.4.1-bin-hadoop2.7

Magpie 2.1

• Support SPARK_NETWORK_TIMEOUT configuration option.
• Support SPARK_RDD_COMPRESS configuration option.
• Fix various deprecated warnings.
• Fix broken Zeppelin dependencies.
• Alter default Spark webui ports to avoid conflicts with other web services.
• Support Hbase 1.4.7, 1.4.8, 1.4.9
  • Update Hbase default to 1.4.9
• Support Spark 2.4.0-bin-hadoop2.6, 2.4.0-bin-hadoop2.7, Spark 2.3.2-bin-hadoop2.6, 2.3.2-bin-hadoop2.7
  • Update Spark default to 2.4.0-bin-hadoop2.7
• Support Hadoop 2.7.6, 2.7.7, 2.8.5, 2.9.2, 3.1.2, 3.2.0
  • Update Hadoop default to 3.2.0
• Support Zeppelin 0.8.0, 0.8.1
  • Update Zeppelin default to 0.8.1
• Implement Shiro basic auth for Zeppelin, default requires a password being set

Magpie 2.0

Major new release of Magpie. Please see NEWS file for details. High level summary of changes.

• New feature highlights
  • Support Hadoop 3.X
  • Support MAGPIE_HOSTNAME_CMD_MAP and MAGPIE_HOSTNAME_SCHEDULER_MAP configuration to "map" hostnames/nodenames to alternate names (i.e. highspeed NIC).
  • MAGPIE_PRE_JOB_RUN and MAGPIE_POST_JOB_RUN can have multiple scripts listed by comma.
  • MAGPIE_PRE_JOB_RUN and MAGPIE_POST_JOB_RUN can have arguments passed to each script.
  • MAGPIE_SCRIPT_PATH and MAGPIE_SCRIPT_ARGS have been collapsed into one environment variable MAGPIE_JOB_SCRIPT.
• Magpie 1.0 submission scripts will not work with Magpie 2.0. Many configuration options have been changed or removed. Please see submission files for details on changes. Some examples of changes:
  • "setuponly" and "interactive" are now configured from MAGPIE_JOB_TYPE only.
  • "script" jobs are now configured from MAGPIE_JOB_TYPE only.
  • HADOOP_SETUP_TYPE is version agnostic
  • SPARK_USE_YARN is replaced with SPARK_SETUP_TYPE
  • HADOOP_PER_JOB_HDFS_PATH and ZOOKEEPER_PER_JOB_DATA_DIR have been removed, the behavior has been collapsed into MAGPIE_ONE_TIME_RUN.
  • Remove support of HADOOP_RAWNETWORKFS_BLOCKSIZE, users should use HADOOP_LOCAL_FILESYSTEM_BLOCKSIZE.
• Removed lots of legacy support including Hadoop 1.0, UDA, HDFS Federation, and Tachyon.