Skip to content

Commit

Permalink
Minor updates to the README
Browse files Browse the repository at this point in the history
  • Loading branch information
LucaCanali committed Dec 1, 2023
1 parent fe1c643 commit c3964ab
Show file tree
Hide file tree
Showing 2 changed files with 8 additions and 9 deletions.
15 changes: 7 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@
![SparkPlugins CI](https://github.com/cerndb/SparkPlugins/workflows/SparkPlugins%20CI/badge.svg?branch=master&event=push)
[![Maven Central](https://maven-badges.herokuapp.com/maven-central/ch.cern.sparkmeasure/spark-plugins_2.12/badge.svg)](https://maven-badges.herokuapp.com/maven-central/ch.cern.sparkmeasure/spark-plugins_2.12)

This repository contains code and examples of how to use Apache Spark Plugins.
Spark plugins provide an interface,
and related configuration, for injecting custom code on executors as they are initialized.
This repository offers extensive resources for utilizing Plugins for Apache Spark, which are essential for incorporating custom code into executors during initialization.
The primary motivation behind Spark Plugins is to extend Spark's capabilities in terms of instrumentation and integration with various external systems.
nal systems, such as monitoring tools.

### Contents
- [Getting started](#getting-started)
Expand All @@ -16,14 +16,13 @@ and related configuration, for injecting custom code on executors as they are in
- [Experimental plugins](#experimental-plugins-for-io-time-instrumentation)

---
### Motivations
- One important use case for deploying Spark Plugins is extending Spark instrumentation with custom metrics.
- Other use cases include running custom actions when the executors start up, typically useful for integrating with
### Motivations and key use cases
- **Extending Spark instrumentation** with custom metrics
- **Running custom actions** when the executors start up, typically useful for integrating with
external systems, such as monitoring systems.
- This repo provides code and examples of plugins applied to measuring Spark on cluster resources (YARN, K8S, Standalone),
including measuring Spark I/O from cloud Filesystems, OS metrics, custom application metrics, and integrations with external systems like Pyroscope.
- Note: The code in this repo is for Spark 3.x.
For Spark 2.x, see instead [Executor Plugins for Spark 2.4](https://github.com/cerndb/SparkExecutorPlugins2.4)
- The code in this repo is for Spark 3.x. For Spark 2.x, see instead [Executor Plugins for Spark 2.4](https://github.com/cerndb/SparkExecutorPlugins2.4)

### Implementation Notes:
- Spark plugins implement the `org.apache.spark.api.Plugin` interface, they can be written in Scala or Java
Expand Down
2 changes: 1 addition & 1 deletion build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ licenses += ("Apache-2.0", url("http://www.apache.org/licenses/LICENSE-2.0"))

libraryDependencies += "io.dropwizard.metrics" % "metrics-core" % "4.2.19"
libraryDependencies += "org.apache.hadoop" % "hadoop-client-api" % "3.3.4"
libraryDependencies += "io.pyroscope" % "agent" % "0.12.0"
libraryDependencies += "io.pyroscope" % "agent" % "0.12.2"
libraryDependencies += "org.apache.spark" %% "spark-core" % "3.5.0"

// publishing to Sonatype Nexus repository and Maven
Expand Down

0 comments on commit c3964ab

Please sign in to comment.