Skip to content

SPARK-5358: Rework the classloader impelementation. #4166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
MattWhelan wants to merge 3 commits into from
Closed

SPARK-5358: Rework the classloader impelementation. #4166

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
bc15951
SPARK-5358: Rework the classloader impelementation.
Jan 22, 2015
9283a36
Added copyright header.
Jan 22, 2015
d6f3677
SPARK-5358: Rework the classloader impelementation.
Jan 22, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 33 additions & 38 deletions core/src/main/scala/org/apache/spark/executor/ExecutorURLClassLoader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,56 +19,51 @@ package org.apache.spark.executor

import java.net.{URLClassLoader, URL}

import org.apache.spark.util.ParentClassLoader
/**
* A ClassLoader trait that changes the normal delegation scheme.
* Normally, it's (cache, parent, self). GreedyClassLoader uses
* (cache, self, parent). This lets this CL "hide" or "override"
* class defs that also exist in the parent loader.
*/
private[spark] trait GreedyClassLoader extends ClassLoader {
Copy link
Contributor

@pwendell pwendell Feb 8, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rather than making this a trait, can you just modify ChildExecutorURLClassLoader to have this logic? We don't need it anywhere else at this point, and this is the main intended goal of ChildExecutorURLClassLoader (to do greedy loading like this).

override def loadClass(name: String, resolve: Boolean): Class[_] = {
this.synchronized {
// Check the cache
var c: Class[_] = findLoadedClass(name)
if (c == null) {
try {
// Try loading it ourselves
c = findClass(name)
} catch {
case ignored: ClassNotFoundException =>
}

if (c == null) {
// Finally, try delegating to the parent
c = getParent.loadClass(name)
}
}
if (resolve) {
resolveClass(c)
}
c
}
}
}

/**
* The addURL method in URLClassLoader is protected. We subclass it to make this accessible.
* We also make changes so user classes can come before the default classes.
*/

private[spark] trait MutableURLClassLoader extends ClassLoader {
def addURL(url: URL)
def addURL(url: URL) //XXX: this is sketchy and dangerous. ClassLoader instances aren't designed to be mutable.
def getURLs: Array[URL]
}

private[spark] class ChildExecutorURLClassLoader(urls: Array[URL], parent: ClassLoader)
extends MutableURLClassLoader {

private object userClassLoader extends URLClassLoader(urls, null){
override def addURL(url: URL) {
super.addURL(url)
}
override def findClass(name: String): Class[_] = {
super.findClass(name)
}
}

private val parentClassLoader = new ParentClassLoader(parent)

override def findClass(name: String): Class[_] = {
try {
userClassLoader.findClass(name)
} catch {
case e: ClassNotFoundException => {
parentClassLoader.loadClass(name)
}
}
}

def addURL(url: URL) {
userClassLoader.addURL(url)
}

def getURLs() = {
userClassLoader.getURLs()
}
extends URLClassLoader(urls, parent) with GreedyClassLoader with MutableURLClassLoader {
}

private[spark] class ExecutorURLClassLoader(urls: Array[URL], parent: ClassLoader)
extends URLClassLoader(urls, parent) with MutableURLClassLoader {

override def addURL(url: URL) {
super.addURL(url)
}
}