[SPARK-11421] [Core][Python][R] Added ability for addJar to augment the current classloader #15666

mariusvniekerk · 2016-10-28T00:37:03Z

What changes were proposed in this pull request?

Adds a flag to sc.addJar to add the jar to the current classloader

How was this patch tested?

Unit tests, manual tests

This is a continuation of the pull request in #9313 and is mostly a rebase of that moved to master with SparkR additions.

cc @holdenk

HyukjinKwon · 2016-10-28T00:55:16Z

cc @felixcheung too.

shivaram · 2016-10-29T04:14:46Z

Jenkins, ok to test

SparkQA · 2016-10-29T04:23:26Z

Test build #67744 has finished for PR 15666 at commit 2b1e98e.

This patch fails R style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-10-29T15:21:24Z

Test build #67759 has finished for PR 15666 at commit 7f37d3a.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-10-29T20:19:09Z

Test build #67773 has finished for PR 15666 at commit 9d838b3.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- |public class $className implements java.io.Serializable

SparkQA · 2016-10-29T23:08:59Z

Test build #67774 has finished for PR 15666 at commit d4416d9.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-10-30T02:33:23Z

Test build #67776 has finished for PR 15666 at commit fccb141.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2016-10-31T20:48:56Z

R/pkg/R/context.R

+
+#' Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.
+#'
+#' The `path` passed can be either a local file, a file in HDFS (or other Hadoop-supported


use \code{path} instead of path in R doc

felixcheung · 2016-10-31T20:50:17Z

core/src/main/scala/org/apache/spark/SparkContext.scala

+
+      val uri = if (path.contains("\\")) {
        // For local paths with backslashes on Windows, URI throws an exception
-        key = env.rpcEnv.fileServer.addJar(new File(path))


this seems to be changing existing behavior?

So this change gets the URI for the windows URI which is used later on to construct a File instance. That should allow the windows special case to work.

So if we construct file and then go with toURI its safe? Have we tested this on Windows?

felixcheung · 2016-10-31T20:52:15Z

core/src/main/scala/org/apache/spark/SparkContext.scala

+    addJar(path, false)
+  }
+
+  def addJar(path: String, addToCurrentClassLoader: Boolean) {


instead of changing SparkContext.addJar as a broad public API, would it be better to have a more narrow change in a Python/R only helper?

Keeping it in the Scala makes it simpler for other spark Scala interpeters (eg toree, zeppelin) to make use of this.

What about if we made the the Scala API as a Developer API since we might end up wanting to change the Scala side in the future?

SparkQA · 2016-11-01T04:25:01Z

Test build #67860 has finished for PR 15666 at commit 26b39de.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2016-11-01T07:01:21Z

Build started: [SparkR] ALL
Diff: master...spark-test:3930DC55-94D4-4307-8E77-BF16007C6AAD

mariusvniekerk · 2016-11-02T02:29:16Z

@HyukjinKwon there seems to be something weird with the appveyor checks?

HyukjinKwon · 2016-11-02T02:35:27Z

@mariusvniekerk Yes, it was (please refer #15709). There is a problem to re-trigger the test (it seems even committers are unable to do). So, I ran this via another account and left a comment above. So, I guess it is fine now. If you are worried of the failure mark, as a (a bit ugly) workaround, you could maybe close and re-open this to re-trigger. I just tested in spark-test#11.

mariusvniekerk · 2016-12-17T03:32:50Z

Rebased.

SparkQA · 2016-12-17T05:53:28Z

Test build #70293 has finished for PR 15666 at commit 1b1a0e9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk

Thanks for picking this up :) Would be good to have @JoshRosen or someone take a look as well. I'll try and do a more thorough read-through later on this week :)

holdenk · 2016-12-20T22:26:24Z

core/src/main/scala/org/apache/spark/TestUtils.scala

I know this is moved code, but could we just use createJarWithClasses or do the R tests require these inner functions?

If we do keep this though I'd give it a clearer name and description because without reading the code, createDummyJar and createJarWithClasses don't seem all that different.

Yeah when i wrote this that didn't exist yet. Changing.

The R tests do indeed verify that they can call the internal functions.

I can revert that part of the changes.

In R\pkg\inst\tests\testthat\jarTest.R

helloTest <- SparkR:::callJStatic("sparkrtest.DummyClass", "helloWorld", "Dave") stopifnot(identical(helloTest, "Hello Dave")) basicFunction <- SparkR:::callJStatic("sparkrtest.DummyClass", "addStuff", 2L, 2L) stopifnot(basicFunction == 4L)

SparkQA · 2016-12-23T07:20:43Z

Test build #70534 has finished for PR 15666 at commit 73df5a4.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mariusvniekerk · 2017-02-01T17:17:52Z

@holdenk Anything i can do from my side to help this guy along?

holdenk · 2017-02-01T17:25:56Z

Thanks for the ping, I was running into an issue this would have simplified yesterday so I was meaning to follow up. I'll see if maybe @felixcheung is around for Spark Summit East and we could go over together in person there (other wise I'll see if I can find another reviewer to rope in with me while I'm there :)).

mariusvniekerk · 2017-02-01T17:26:28Z

Yeah I'll be there

holdenk

Great, lets meetup and chat IRL. I've got some questions with regards to the windows handling. Maybe we can find @felixcheung at spark summit east too and get back up to speed quickly on the state of this because it would be nice to have :)

holdenk · 2017-02-01T17:29:35Z

core/src/main/scala/org/apache/spark/SparkContext.scala

+    addJar(path, false)
+  }
+
+  def addJar(path: String, addToCurrentClassLoader: Boolean) {


What about if we made the the Scala API as a Developer API since we might end up wanting to change the Scala side in the future?

holdenk · 2017-02-01T17:30:03Z

core/src/main/scala/org/apache/spark/SparkContext.scala

attempt to is interesting wording here - when would we expect this to fail and should we explicitly call that out?

Add to doc that already loaded urls will have no effect if a url is already present.

holdenk · 2017-02-01T17:35:41Z

core/src/main/scala/org/apache/spark/SparkContext.scala

+
+      val uri = if (path.contains("\\")) {
        // For local paths with backslashes on Windows, URI throws an exception
-        key = env.rpcEnv.fileServer.addJar(new File(path))


So if we construct file and then go with toURI its safe? Have we tested this on Windows?

holdenk · 2017-02-01T17:37:19Z

core/src/main/scala/org/apache/spark/SparkContext.scala

So I'm a bit confused, we already supposedly have a safe uri object for windows above, does it throw an exception on getPath? Or is it that \ is always local and getScheme doesn't return that?

If we have backslashes we are in a local path on windows.

holdenk · 2017-02-08T21:39:37Z

python/pyspark/context.py

Add a :param: annotation here as well for path & addToCurrentClassLoader

SparkQA · 2017-02-09T16:56:25Z

Test build #72650 has finished for PR 15666 at commit 21643b5.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-02-09T19:17:13Z

Test build #72657 has finished for PR 15666 at commit 9fbcd9b.

This patch fails Python style tests.
This patch merges cleanly.
This patch adds no public classes.

holdenk · 2017-02-13T19:03:31Z

Thanks for adding the R side tests :) Since this touches core (albeit as a developer API) we should probably also run this by someone who's worked on addJar (like @brkyvz ).

mariusvniekerk · 2017-03-06T14:13:10Z

Ah thanks

HyukjinKwon · 2017-03-06T14:48:39Z

R/pkg/inst/tests/testthat/test_context.R

+      destDir, "sparkrTests", "DummyClassForAddJarTest")
+
+  spark.addJar(jarName, addToCurrentClassLoader = TRUE)
+  testClass <- newJObject("sparkrTests.DummyClassForAddJarTest")


Hm.. it seems this line this time.

1. Error: add jar should work and allow usage of the jar on the driver node (@test_context.R#178) 1: newJObject("sparkrTests.DummyClassForAddJarTest") at C:/projects/spark/R/lib/SparkR/tests/testthat/test_context.R:178 2: invokeJava(isStatic = TRUE, className, methodName = "<init>", ...) 3: handleErrors(returnStatus, conn) 4: stop(readString(conn))

Maybe, helpful log:

ERROR RBackendHandler: <init> on sparkrTests.DummyClassForAddJarTest failed java.lang.ClassNotFoundException: sparkrTests.DummyClassForAddJarTest at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.spark.util.Utils$.classForName(Utils.scala:230) at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:143) at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:108) at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:40) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336) at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336) at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336) at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:293) at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:267) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:336) at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:357) at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:343) at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911) at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131) at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643) at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566) at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:745)

yeah i suspect that the windows path didn't make it properly into the classloader

HyukjinKwon

I just left few super minor comments. If commits should be pushed, maybe these could be improved.

Let me try to take a look on this weekend as I have an environment to test this on Windows.

HyukjinKwon · 2017-03-06T15:01:27Z

core/src/main/scala/org/apache/spark/TestUtils.scala

+    val excSource = new JavaSourceFromString(new File(srcDir, className).toURI.getPath,
+      s"""package $packageName;
+         |
+          |public class $className implements java.io.Serializable {


(This indentation seems inconsistent BTW)

HyukjinKwon · 2017-03-06T15:01:31Z

core/src/main/scala/org/apache/spark/TestUtils.scala

+         |  public static int addStuff(int arg1, int arg2) { return arg1 + arg2; }
+         |}
+        """.
+        stripMargin)


(We could make this lined.)

We could make this inlined.

HyukjinKwon · 2017-03-06T15:02:26Z

core/src/test/scala/org/apache/spark/SparkContextSuite.scala


+  for (
+    schedulingMode <- Seq("local_mode", "non_local_mode")
+  ) {


Just personal taste.. maybe

Seq("local_mode", "non_local_mode").foreach { schedulingMode => ... }

?

HyukjinKwon · 2017-03-06T15:04:10Z

core/src/main/scala/org/apache/spark/SparkContext.scala

+          val currentCL = Utils.getContextOrSparkClassLoader
+          currentCL match {
+            case cl: MutableURLClassLoader =>
+              cl.addURL(uri.toURL)


(We could make this inlined too..)

HyukjinKwon · 2017-03-06T15:07:35Z

Feel free to push any commit. Let me trigger the build with my account like a bot.

SparkQA · 2017-03-06T16:07:50Z

Test build #74006 has finished for PR 15666 at commit 8713be0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-06T21:06:24Z

Test build #74023 has finished for PR 15666 at commit 7a918a6.

This patch passes all tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
|public class $className implements java.io.Serializable

brkyvz · 2017-03-06T21:08:18Z

R/pkg/R/context.R

+#' spark.addJar("/path/to/something.jar", TRUE)
+#'}
+#' @note spark.addJar since 2.2.0
+spark.addJar <- function(path, addToCurrentClassLoader = FALSE) {


why don't we want to add it to the driver classpath by default?

Mostly for backwards compatibility.

brkyvz · 2017-03-06T21:09:53Z

R/pkg/inst/tests/testthat/test_context.R

+test_that("add jar should work and allow usage of the jar on the driver node", {
+  sparkR.sparkContext()
+
+  destDir <- paste0(tempdir(), "/", "testjar")


you may want to use https://stat.ethz.ch/R-manual/R-devel/library/base/html/file.path.html instead for windows compatibility

brkyvz · 2017-03-06T21:27:09Z

core/src/main/scala/org/apache/spark/SparkContext.scala

+
+      val uri = if (path.contains("\\")) {
+        // If we have backslashes here this is a windows path, and URI will throws an exception,
+        // so we use this constrcutor instead


constructor

brkyvz · 2017-03-06T21:28:12Z

core/src/main/scala/org/apache/spark/SparkContext.scala

+  /**
+   * Adds a JAR dependency for all tasks to be executed on this `SparkContext` in the future.
+   * @param path can be either a local file, a file in HDFS (or other Hadoop-supported filesystems),
+   * an HTTP, HTTPS or FTP URI, or local:/path for a file on every worker node.


nit: alignment

@param path can be either... blah more blah

felixcheung · 2017-03-07T05:36:35Z

R/pkg/R/context.R

+#' filesystems), an HTTP, HTTPS or FTP URI, or local:/path for a file on every worker node.
+#' If \code{addToCurrentClassLoader} is true, add the jar to the current threads' classloader. In
+#' general adding to the current threads' class loader will impact all other application threads
+#' unless they have explicitly changed their class loader.


should

add the jar to the current threads' classloader. In +#' general adding to the current threads' class loader will impact all other application threads +#' unless they have explicitly changed their class loader.

be reworded for R or Python? It should be the JVM. threads or classloader aren't familiar concepts to R users.

felixcheung · 2017-03-07T05:36:57Z

R/pkg/R/context.R

 }

+
+#' Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.


we don't really expose SparkContext in R actually

In that case do we want to bother having this method for R?

felixcheung · 2017-03-07T05:41:02Z

R/pkg/R/context.R

+spark.addJar <- function(path, addToCurrentClassLoader = FALSE) {
+  sc <- getSparkContext()
+  normalizedPath <- suppressWarnings(normalizePath(path))
+  scala_sc <- callJMethod(sc, "sc")


I'd just combine 342 & 344 and call it sc

sc <- callJMethod(getSparkContext(), "sc")

felixcheung · 2017-03-07T05:42:26Z

R/pkg/R/context.R

+#' @rdname spark.addJar
+#' @param path The path of the jar to be added
+#' @param addToCurrentClassLoader Whether to add the jar to the current driver classloader.
+#' Default is FALSE.


remove default value since it's in the signature already below

felixcheung · 2017-03-07T05:45:18Z

R/pkg/R/context.R

+#' spark.addJar("/path/to/something.jar", TRUE)
+#'}
+#' @note spark.addJar since 2.2.0
+spark.addJar <- function(path, addToCurrentClassLoader = FALSE) {


spark.addJar needs to be added to the NAMESPACE file, otherwise it won't be accessible. Tests are running within the same namespace so it's always able to access private methods.

HyukjinKwon

I am sorry that I could not check this in the past weekend. Honestly, after I updated my Mac OS, somehow all the VMs are broken and I have to rebuild the env... so I ran some tests with AppVeyor. I will update you when I finished this.

HyukjinKwon · 2017-03-13T17:37:51Z

R/pkg/inst/tests/testthat/test_context.R

+  sparkR.sparkContext()
+
+  destDir <- paste0(tempdir(), "/", "testjar")
+  jarName <- callJStatic("org.apache.spark.TestUtils", "createDummyJar",


I checked what it's got right after createDummyJar call. It was C:\Users\appveyor\AppData\Local\Temp\1\Rtmpg5tOYo/testjar

HyukjinKwon · 2017-03-13T17:39:10Z

R/pkg/R/context.R

+  sc <- getSparkContext()
+  normalizedPath <- suppressWarnings(normalizePath(path))
+  scala_sc <- callJMethod(sc, "sc")
+  invisible(callJMethod(scala_sc, "addJar", normalizedPath, addToCurrentClassLoader))


Right after this call, the path normalizedPath was C:\projects\spark\R\lib\SparkR\tests\testthat\file:\C:\Users\appveyor\AppData\Local\Temp\1\Rtmpg5tOYo\testjar\sparkrTests-DummyClassForAddJarTest-1489422615481.jar

(C:\projects\spark\R\lib\SparkR\tests\testthat\ was indeed printed together in the logs..).

why is normalizepath doing that to the url?

it doesn't handle file:\C: prefix - where is that coming from?

SparkQA · 2017-03-18T17:07:17Z

Test build #74781 has finished for PR 15666 at commit 8f956c5.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-03-18T17:15:43Z

Test build #74782 has finished for PR 15666 at commit e1a59c9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-06-17T18:01:18Z

Test build #78211 has finished for PR 15666 at commit d76880b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

mariusvniekerk · 2017-06-17T22:48:42Z

@HyukjinKwon Any hints what's needed to get the R stuff passing? I don't really have a windows testbed that I can use.

jiangxb1987 · 2017-10-03T14:48:53Z

Should we continue with this? @HyukjinKwon

HyukjinKwon · 2017-10-04T04:20:41Z

Thanks for asking this. I completely forgot this one. Will try to make some time to take a look within few days.

HyukjinKwon

Looks okay but I think I need another close look. Just left some comments while figuring out the R test failure on Windows.

HyukjinKwon · 2017-10-05T18:05:04Z

R/pkg/tests/fulltests/test_context.R

+
+  destDir <- file.path(tempdir(), "testjar")
+  jarName <- callJStatic("org.apache.spark.TestUtils", "createDummyJar",
+      destDir, "sparkrTests", "DummyClassForAddJarTest")


Ah, it looks the problem is here. createDummyJar returns URI string, for example,

> normalizePath("file:/C:/a/b/c") [1] "C:\\Users\\IEUser\\workspace\\spark\\file:\\C:\\a\\b\\c" Warning message: In normalizePath(path.expand(path), winslash, mustWork) : path[1]="file:/C:/a/b/c": The filename, directory name, or volume label syntax is incorrect

This looks ending up with an weird path like "C:\\Users\\IEUser\\workspace\\spark\\file:\\C:\\a\\b\\c".

HyukjinKwon · 2017-10-05T18:11:05Z

R/pkg/R/context.R

+}
+
+
+


little nit: I guess we just need a single newline.

HyukjinKwon · 2017-10-05T18:11:44Z

R/pkg/tests/fulltests/test_context.R

+test_that("add jar should work and allow usage of the jar on the driver node", {
+  sparkR.sparkContext()
+
+  destDir <- file.path(tempdir(), "testjar")


I'd remove this tempdir() btw (I mean removing the actual directory after this test).

HyukjinKwon · 2017-10-05T18:12:23Z

R/pkg/tests/fulltests/test_context.R

+
+  destDir <- file.path(tempdir(), "testjar")
+  jarName <- callJStatic("org.apache.spark.TestUtils", "createDummyJar",
+      destDir, "sparkrTests", "DummyClassForAddJarTest")


little nit:

jarName <- callJStatic("org.apache.spark.TestUtils", "createDummyJar", destDir, "sparkrTests", "DummyClassForAddJarTest")

HyukjinKwon · 2017-10-05T18:21:09Z

core/src/main/scala/org/apache/spark/SparkContext.scala

+                // For local paths with backslashes on Windows, URI throws an exception
+                new File(path).toURI
+              } else {
+                new URI(path)


Could we maybe just use Utils.resolveURI(path)? looks the logic is similar.

HyukjinKwon · 2017-10-05T18:21:37Z

core/src/main/scala/org/apache/spark/SparkContext.scala

+                new URI(path)
+              }
+              cl.addURL(uri.toURL)
+            case _ => logWarning(s"Unsupported cl $currentCL will not update jars thread cl")


I'd sayclass loader instead of cl.

HyukjinKwon · 2017-10-05T18:22:20Z

core/src/main/scala/org/apache/spark/TestUtils.scala

+         |  public static int addStuff(int arg1, int arg2) { return arg1 + arg2; }
+         |}
+        """.
+        stripMargin)


We could make this inlined.

HyukjinKwon · 2017-10-05T18:25:30Z

python/pyspark/context.py

+        Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.
+        The `path` passed can be either a local file, a file in HDFS (or other Hadoop-supported
+        filesystems), an HTTP, HTTPS or FTP URI, or local:/path for a file on every worker node.
+        If addToCurrentClassLoader is true, add the jar to the current threads'  classloader.


little nit: addToCurrentClassLoader ->`addToCurrentClassLoader` and ads' cl -> ads' cl.

HyukjinKwon · 2017-10-05T18:27:50Z

python/pyspark/tests.py

+        # We shouldn't be able to load anything from the package before it is added
+        self.assertFalse(isinstance(jvm.pysparktests.DummyClass, JavaClass))
+        # Generate and compile the test jar
+        destDir = os.path.join(SPARK_HOME, "python/test_support/jar")


I'd remove this directory too.

Instead you want to use a temp directory?

HyukjinKwon · 2017-10-05T18:30:01Z

I am sorry for this delay @mariusvniekerk. Would you have some time to proceed this one please?

AmplabJenkins · 2017-12-14T21:05:38Z

Can one of the admins verify this patch?

jiangxb1987 · 2018-04-04T13:35:57Z

ping @mariusvniekerk

mariusvniekerk · 2018-04-06T07:57:21Z

This was superceded by #19643

felixcheung reviewed Oct 31, 2016

View reviewed changes

mariusvniekerk closed this Dec 10, 2016

mariusvniekerk reopened this Dec 10, 2016

mariusvniekerk force-pushed the SPARK-11421 branch from 26b39de to 1b1a0e9 Compare December 17, 2016 03:31

holdenk reviewed Dec 20, 2016

View reviewed changes

mariusvniekerk force-pushed the SPARK-11421 branch from 1b1a0e9 to 73df5a4 Compare December 23, 2016 04:45

holdenk reviewed Feb 1, 2017

View reviewed changes

holdenk reviewed Feb 8, 2017

View reviewed changes

python/pyspark/context.py Outdated

Copy link

Contributor

holdenk Feb 8, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a :param: annotation here as well for path & addToCurrentClassLoader

HyukjinKwon reviewed Mar 6, 2017

View reviewed changes

brkyvz reviewed Mar 6, 2017

View reviewed changes

felixcheung requested changes Mar 7, 2017

View reviewed changes

HyukjinKwon reviewed Mar 13, 2017

View reviewed changes

mariusvniekerk force-pushed the SPARK-11421 branch from 7a918a6 to 8f956c5 Compare March 18, 2017 14:25

mariusvniekerk added 3 commits June 17, 2017 11:05

SPARK-11421 Squashed addjar pr

a6fb0c7

Addressed some review comments

962d1ff

Addressed some review comments

d76880b

mariusvniekerk force-pushed the SPARK-11421 branch from e1a59c9 to d76880b Compare June 17, 2017 15:10

HyukjinKwon reviewed Oct 5, 2017

View reviewed changes

HyukjinKwon mentioned this pull request Nov 2, 2017

[SPARK-11421][CORE][PYTHON][R] Added ability for addJar to augment the current classloader #19643

Closed

mariusvniekerk closed this Apr 6, 2018

		}


		#' Adds a JAR dependency for all tasks to be executed on this SparkContext in the future.

[SPARK-11421] [Core][Python][R] Added ability for addJar to augment the current classloader #15666

[SPARK-11421] [Core][Python][R] Added ability for addJar to augment the current classloader #15666

Uh oh!

Conversation

mariusvniekerk commented Oct 28, 2016

What changes were proposed in this pull request?

How was this patch tested?

Uh oh!

HyukjinKwon commented Oct 28, 2016

Uh oh!

shivaram commented Oct 29, 2016

Uh oh!

SparkQA commented Oct 29, 2016

Uh oh!

SparkQA commented Oct 29, 2016

Uh oh!

SparkQA commented Oct 29, 2016

Uh oh!

SparkQA commented Oct 29, 2016

Uh oh!

SparkQA commented Oct 30, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felixcheung Oct 31, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Nov 1, 2016

Uh oh!

HyukjinKwon commented Nov 1, 2016

Uh oh!

mariusvniekerk commented Nov 2, 2016

Uh oh!

HyukjinKwon commented Nov 2, 2016

Uh oh!

mariusvniekerk commented Dec 17, 2016

Uh oh!

SparkQA commented Dec 17, 2016

Uh oh!

holdenk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mariusvniekerk Dec 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Dec 23, 2016

Uh oh!

mariusvniekerk commented Feb 1, 2017

Uh oh!

holdenk commented Feb 1, 2017

Uh oh!

mariusvniekerk commented Feb 1, 2017

Uh oh!

holdenk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

felixcheung Oct 31, 2016 •

edited

Loading

mariusvniekerk Dec 23, 2016 •

edited

Loading

HyukjinKwon left a comment •

edited

Loading

HyukjinKwon Oct 5, 2017 •

edited

Loading