Skip to content

cerndb/SparkExecutorPlugins2.4

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

4 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

SparkExecutorPlugins for Spark 2.4

Code with examples of Spark Executor Plugins for Spark 2.4.x
Spark 3.0.0 is expected to have an extended and incompatible API, see Spark 3.0.0 examples
Contact: Luca.Canali@cern.ch

Build with: sbt +package

  • Executor plugins implement the org.apache.spark.ExecutorPlugin interface and can be used to run code at the startup of Spark executors. This feature is available in Spark 2.4.x. It can be useful for extending monitoring and configuration. Use as in the following example:
    /bin/spark-shell --master yarn --jars <path>/sparkexecutorplugins_2.11-0.1.jar \
    --conf spark.executor.plugins=ch.cern.ExecutorPluginJava.RunOSCommandPlugin 
    

Plugin code examples

This repo contains a list of example plugin classes, written in Scala and Java. Both languages work for this task, it is mostly a matter of taste.

  • Scala

    • DemoPlugin -> basic plugin for testing
  • Java

    • DemoPlugin -> basic plugin for testing
    • RunOSCommandPlugin -> plugin demonstrating how to run OS commands at executor startup

About

Spark Executor Plugins Examples for Spark 2.4

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published