[SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System #23546

skonto · 2019-01-15T04:27:56Z

What changes were proposed in this pull request?

solves the current issue with --packages in cluster mode (there is no ticket for it). Also note of some issues of the past here when hadoop libs are used at the spark submit side.
supports spark.jars, spark.files, app jar.

It works as follows:
Spark submit uploads the deps to the HCFS. Then the driver serves the deps via the Spark file server.
No hcfs uris are propagated.

The related design document is here. the next option to add is the RSS but has to be improved given the discussion in the past about it (Spark 2.3).

How was this patch tested?

Run integration test suite.
Run an example using S3:

 ./bin/spark-submit \
...
 --packages com.amazonaws:aws-java-sdk:1.7.4,org.apache.hadoop:hadoop-aws:2.7.6 \
 --deploy-mode cluster \
 --name spark-pi \
 --class org.apache.spark.examples.SparkPi \
 --conf spark.executor.memory=1G \
 --conf spark.kubernetes.namespace=spark \
 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark-sa \
 --conf spark.driver.memory=1G \
 --conf spark.executor.instances=2 \
 --conf spark.sql.streaming.metricsEnabled=true \
 --conf "spark.driver.extraJavaOptions=-Divy.cache.dir=/tmp -Divy.home=/tmp" \
 --conf spark.kubernetes.container.image.pullPolicy=Always \
 --conf spark.kubernetes.container.image=skonto/spark:k8s-3.0.0 \
 --conf spark.kubernetes.file.upload.path=s3a://fdp-stavros-test \
 --conf spark.hadoop.fs.s3a.access.key=... \
 --conf spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem \
 --conf spark.hadoop.fs.s3a.fast.upload=true \
 --conf spark.kubernetes.executor.deleteOnTermination=false \
 --conf spark.hadoop.fs.s3a.secret.key=... \
 --conf spark.files=client:///...resolv.conf \
file:///my.jar **

Added integration tests based on Ceph nano. Looks very active.
Unfortunately minio needs hadoop >= 2.8.

SparkQA · 2019-01-15T04:32:24Z

Test build #101225 has finished for PR 23546 at commit 20603a4.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-15T04:44:06Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/7074/

SparkQA · 2019-01-15T04:56:49Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/7074/

SparkQA · 2019-01-15T05:00:44Z

Test build #101228 has finished for PR 23546 at commit 7597e03.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

skonto · 2019-01-15T05:03:41Z

Integration tests failed with:

Run SparkRemoteFileTest using a remote data file *** FAILED ***
The code passed to eventually never returned normally. Attempted 70 times over 2.00060388465 minutes. Last failure message: false was not true. (KubernetesSuite.scala:276)

This is most likely due to dns on the test node. I saw this with ubuntu 18.04 and contents of /etc/resolv.conf causing issues. Workaround was to use with minikube start: --extra-config=kubelet.resolv-conf=/run/systemd/resolve/resolv.conf

skonto · 2019-01-15T05:05:12Z

@erikerlandson, @liyinan926 , @felixcheung pls review if the approach is approved I can add integration tests etc.

SparkQA · 2019-01-15T08:05:01Z

Test build #101229 has finished for PR 23546 at commit bf5f3b1.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

skonto · 2019-01-15T11:38:59Z

jenkins test this please

SparkQA · 2019-01-15T11:55:18Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/7099/

SparkQA · 2019-01-15T12:07:44Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/7099/

skonto · 2019-01-15T13:04:29Z

Integration tests still fail for the same issue.

docs/running-on-kubernetes.md

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

...etes/core/src/main/scala/org/apache/spark/deploy/k8s/features/DriverCommandFeatureStep.scala

core/src/main/scala/org/apache/spark/util/Utils.scala

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

SparkQA · 2019-01-15T15:46:22Z

Test build #101254 has finished for PR 23546 at commit bf5f3b1.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-01-15T15:47:09Z

Test build #101255 has finished for PR 23546 at commit bf5f3b1.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

skonto · 2019-01-15T19:17:53Z

One python test is failing which is irrelevant:

" File "/home/jenkins/workspace/SparkPullRequestBuilder@2/python/pyspark/mllib/tests/test_streaming_algorithms.py", line 263, in condition
self.assertAlmostEqual(rel, 0.1, 1)
AssertionError: 0.25749106949322637 != 0.1 within 1 places"

skonto · 2019-01-15T20:43:59Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

@@ -330,6 +347,7 @@ private[spark] class SparkSubmit extends Logging {
      }
    }

+    // Fill in all spark properties of SparkConf


I underline this here because it should be a few lines above but not sure if it matters. The issue is a few lines above I have to use args.sparkProperties instead. The other problem is that at line 802 we re-fill the properties if missing... not sure why.

I'd just move this initialization to the top of the function, then code can be consistent in always using sparkConf.

And yes, this code is very confusing and probably does a lot of redundant things. Which I why I asked you to start with the plugin idea in the other PR, in the hopes that we can clean this up little by little.

I agree we should clean it up.

skonto · 2019-01-15T20:45:09Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

-        confKey = "spark.jars.repositories"),
-      OptionAssigner(args.ivyRepoPath, STANDALONE | MESOS, CLUSTER, confKey = "spark.jars.ivy"),
-      OptionAssigner(args.packagesExclusions, STANDALONE | MESOS,
+      OptionAssigner(args.packages, STANDALONE | MESOS | KUBERNETES,


Note: Packages in containers maybe slow if your net is slow since ivy cache will be empty. Users in practice should build their dependencies in the image or use a pre-populated cache.

This point might be a good addition to the docs

I will add it.

erikerlandson · 2019-01-15T21:05:57Z

docs/running-on-kubernetes.md

@@ -400,7 +415,6 @@ There are several Spark on Kubernetes features that are currently being worked o
 Some of these include:

 * Dynamic Resource Allocation and External Shuffle Service
-* Local File Dependency Management


is the fact that this is an hadoop (compatible) FS based solution imply there are use cases for local deps that aren't served by this PR?

Yes there might be cases like the RSS server where users want to upload to a file server within the cluster. I am covering the cases mentioned in the design document which provide an API to use out of the box. The RSS implementation AFAIK needs improvements so its open for now, but we can work on it next. I could add a note there instead of removing that part of the doc saying ("partially done"), but since it is a working solution, I thought I could remove that from future work.

Out of curiosity, does any s3 object store fit the HCFS category?

A quick example is minio another is swift.

I'd say that's broadly applicable enough to call it "done"

We have environments where there is nothing remotely HDFS like available and the systems are typically air-gapped so using external services like S3 isn't an option either. Primary storage is usually a high performance parallel file system (Lustre or IBM Spectrum Scale) which is just a POSIX compliant file system mounted to all nodes over the system interconnect.

Using hostPath volume mounts isn't a realistic option either because these environments have strict security requirements.

In my opinion we need both options: a) upload to some dfs/obect store service b) a file server.
We cannot possibly support all systems out there if they have their own client libs.

rvesse

Looks like a good first step to better supporting client dependencies, thanks for the hard work @skonto

core/src/main/scala/org/apache/spark/util/Utils.scala

docs/running-on-kubernetes.md

rvesse · 2019-01-17T10:47:44Z

docs/running-on-kubernetes.md

+The app jar file will be uploaded to the S3 and then when the driver is launched it will be downloaded
+to the driver pod and will be added to its classpath.
+
+The client scheme is supported for the application jar, and dependencies specified by proeprties `spark.jars` and `spark.files`.


Typo: proeprties -> properties

rvesse · 2019-01-17T11:00:41Z

docs/running-on-kubernetes.md

@@ -400,7 +415,6 @@ There are several Spark on Kubernetes features that are currently being worked o
 Some of these include:

 * Dynamic Resource Allocation and External Shuffle Service
-* Local File Dependency Management


We have environments where there is nothing remotely HDFS like available and the systems are typically air-gapped so using external services like S3 isn't an option either. Primary storage is usually a high performance parallel file system (Lustre or IBM Spectrum Scale) which is just a POSIX compliant file system mounted to all nodes over the system interconnect.

Using hostPath volume mounts isn't a realistic option either because these environments have strict security requirements.

core/src/main/scala/org/apache/spark/util/Utils.scala

liyinan926 · 2019-01-18T16:57:16Z

docs/running-on-kubernetes.md

+--conf spark.hadoop.fs.s3a.fast.upload=true
+--conf spark.hadoop.fs.s3a.secret.key=....
+ client:///full/path/to/app.jar
+```


How does the submission client the user's intention is to upload to S3 instead of say an HDFS cluster? I don't think this can be determined 100% sure only based on the present of those s3a options.

I saw you have spark.kubernetes.file.upload.path below, which should also be added here as an example.

The code is agnostic of the protocol. I am just using S3 as an example in the docs. If they dont put the properties submit will fail.

liyinan926 · 2019-01-18T17:00:27Z

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

@@ -289,6 +289,12 @@ private[spark] object Config extends Logging {
      .booleanConf
      .createWithDefault(true)

+  val KUBERNETES_FILE_UPLOAD_PATH =
+    ConfigBuilder("spark.kubernetes.file.upload.path")
+      .doc("HCFS path to upload files to, using the client scheme:// in cluster mode.")


HCFS path where files with the client:// scheme will be uploded to in cluster mode.

liyinan926 · 2019-01-18T17:02:12Z

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

+    if (fileScheme == "client") {
+      if (conf.get(KUBERNETES_FILE_UPLOAD_PATH).isDefined) {
+        val uploadPath = conf.get(KUBERNETES_FILE_UPLOAD_PATH).get
+        s"${uploadPath}/${fileUri.getPath.split("/").last}"


So a file client://path/to/app1.jar will be uploaded to ${uploadPath}/app1.jar? What if two client-local files at different local paths have the same file name?

I am currently not supporting that. I thought of that. People should make sure they dont create a conflict. Otherwise I will have to create random paths.

Please document this clearly, i.e., all client-side dependencies will be uploaded to the given path with a flat directory structure.

will do thanx.

liyinan926 · 2019-01-18T17:05:12Z

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

+                val uploadPath = sConf.get(KUBERNETES_FILE_UPLOAD_PATH).get
+                val fs = getHadoopFileSystem(Utils.resolveURI(uploadPath), hadoopConf)
+                val storePath = new Path(s"${uploadPath}/${fileUri.getPath.split("/").last}")
+                log.info(s"Uploading file: ${fileUri.getPath}...")


We should also mention the destination path in the log message.

liyinan926 · 2019-01-18T17:08:21Z

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

-  def resolveFileUri(uri: String): String = {
+  /**
+   * Get the final path for a client file, if not return the uri as is.
+   *


This empty comment line can be removed.

liyinan926 · 2019-01-18T17:09:56Z

docs/running-on-kubernetes.md

@@ -1010,6 +1024,15 @@ See the below table for the full list of pod specifications that will be overwri
    Spark will add additional labels specified by the spark configuration.
  </td>
 </tr>
+<tr>


This should be added to the table under the section Spark Properties.

liyinan926 · 2019-01-18T17:15:50Z

core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala

@@ -373,6 +407,49 @@ private[spark] class SparkSubmit extends Logging {
      localPyFiles = Option(args.pyFiles).map {
        downloadFileList(_, targetDir, sparkConf, hadoopConf, secMgr)
      }.orNull
+
+      if (isKubernetesClient &&
+        sparkConf.getBoolean("spark.kubernetes.submitInDriver", false)) {


What's the purpose of checking spark.kubernetes.submitInDriver, which AFAIK is used to indicate the cluster mode.

I need to make sure i run this only in cluster mode at the second submit time where we are using client mode.

See my previous comment about these names not being optimal in the context of how k8s works.

SparkQA · 2019-02-04T21:16:22Z

Test build #102043 has finished for PR 23546 at commit c885132.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-02-04T21:28:45Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/7710/

SparkQA · 2019-02-04T21:41:04Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/7710/

skonto · 2019-05-16T09:14:01Z

jenkins test this please

skonto · 2019-05-16T09:18:04Z

@erikerlandson @srowen I added support for random dirs, pls let me know if this is ok now to be merged. Updated the doc to indicate that user will not see conflicts when spark apps are run in parallel. Hope that helps.

SparkQA · 2019-05-16T09:27:49Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/10713/

SparkQA · 2019-05-16T09:44:48Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/10713/

SparkQA · 2019-05-16T11:33:46Z

Test build #105450 has finished for PR 23546 at commit ec2d66c.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

erikerlandson · 2019-05-17T17:00:04Z

@skonto that looks acceptable to me. Supporting additional directory semantics is not precluded by future PR.
@srowen github is showing unresolved reviews, are any of those still outstanding or can we mark them resolved?

srowen

Just a few nits here; I'd defer to @vanzin for comment on directory handling

srowen · 2019-05-17T19:03:41Z

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

+    if (isLocalAndResolvable(resource)) {
+      SparkLauncher.NO_RESOURCE
+    } else {
+    resource


Nit: indent

srowen · 2019-05-17T19:03:54Z

...ce-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesUtils.scala

+            case e: Exception =>
+              throw new SparkException(s"Uploading file ${fileUri.getPath} failed...", e)
+          }
+        }


Nit: pull the else up to this line

srowen

I resolved my previous comments that were out of date

skonto · 2019-05-17T19:16:05Z

@erikerlandson @srowen I fixed the two pending comments pls resolve the review.

SparkQA · 2019-05-17T19:26:54Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/10761/

SparkQA · 2019-05-17T19:40:11Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/10761/

SparkQA · 2019-05-17T21:39:09Z

Test build #105503 has finished for PR 23546 at commit 3c58f7b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

skonto · 2019-05-20T10:34:41Z

@srowen @erikerlandson gentle ping

skonto · 2019-05-20T18:42:35Z

@erikerlandson @vanzin are you ok with the current status of things? Should it be merged?

erikerlandson · 2019-05-20T20:46:02Z

If there are no further requests I will merge.

skonto · 2019-05-22T09:52:47Z

@erikerlandson from what I see there is no more activity, could you merge please?

koertkuipers · 2020-05-13T03:55:08Z

if i add local file on the client with ``--file somefile``` i expect it to be on classpath for the driver (on the cluster, using cluster deploy-mode). or at least i expect it to be in working directory for driver? but this does not seem to be the case? am i doing something wrong?

…tainer` ### What changes were proposed in this pull request? This PR aims to simply steps to re-write primary resource in k8s spark application. ### Why are the changes needed? Re-write primary resource uses `renameMainAppResource` twice. * First `renameMainAppResource` in `baseDriverContainer` in is introduced by #23546 * #25870 refactors `renameMainAppResource` and introduces `renameMainAppResource` in `configureForJava`. Refactoring and `renameMainAppResource` in `configureForJava` makes `renameMainAppResource` in `baseDriverContainer` useless. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? * Pass the GA. * Pass k8s IT. ``` $ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dtest.exclude.tags=r -Dspark.kubernetes.test.imageRepo=kubespark "kubernetes-integration-tests/test" [info] KubernetesSuite: [info] - Run SparkPi with no resources (17 seconds, 443 milliseconds) [info] - Run SparkPi with no resources & statefulset allocation (17 seconds, 858 milliseconds) [info] - Run SparkPi with a very long application name. (30 seconds, 450 milliseconds) [info] - Use SparkLauncher.NO_RESOURCE (18 seconds, 596 milliseconds) [info] - Run SparkPi with a master URL without a scheme. (18 seconds, 534 milliseconds) [info] - Run SparkPi with an argument. (21 seconds, 853 milliseconds) [info] - Run SparkPi with custom labels, annotations, and environment variables. (14 seconds, 285 milliseconds) [info] - All pods have the same service account by default (13 seconds, 800 milliseconds) [info] - Run extraJVMOptions check on driver (7 seconds, 825 milliseconds) [info] - Run SparkRemoteFileTest using a remote data file (15 seconds, 242 milliseconds) [info] - Verify logging configuration is picked from the provided SPARK_CONF_DIR/log4j2.properties (15 seconds, 491 milliseconds) [info] - Run SparkPi with env and mount secrets. (26 seconds, 967 milliseconds) [info] - Run PySpark on simple pi.py example (20 seconds, 318 milliseconds) [info] - Run PySpark to test a pyfiles example (25 seconds, 659 milliseconds) [info] - Run PySpark with memory customization (25 seconds, 608 milliseconds) [info] - Run in client mode. (14 seconds, 620 milliseconds) [info] - Start pod creation from template (19 seconds, 916 milliseconds) [info] - SPARK-38398: Schedule pod creation from template (19 seconds, 966 milliseconds) [info] - PVs with local hostpath storage on statefulsets (22 seconds, 380 milliseconds) [info] - PVs with local hostpath and storageClass on statefulsets (26 seconds, 935 milliseconds) [info] - PVs with local storage (30 seconds, 75 milliseconds) [info] - Launcher client dependencies (2 minutes, 48 seconds) [info] - SPARK-33615: Launcher client archives (1 minute, 26 seconds) [info] - SPARK-33748: Launcher python client respecting PYSPARK_PYTHON (1 minute, 47 seconds) [info] - SPARK-33748: Launcher python client respecting spark.pyspark.python and spark.pyspark.driver.python (1 minute, 51 seconds) [info] - Launcher python client dependencies using a zip file (1 minute, 51 seconds) [info] - Test basic decommissioning (59 seconds, 765 milliseconds) [info] - Test basic decommissioning with shuffle cleanup (1 minute, 3 seconds) [info] - Test decommissioning with dynamic allocation & shuffle cleanups (2 minutes, 58 seconds) [info] - Test decommissioning timeouts (58 seconds, 754 milliseconds) [info] - SPARK-37576: Rolling decommissioning (1 minute, 15 seconds) [info] Run completed in 29 minutes, 15 seconds. [info] Total number of tests run: 31 [info] Suites: completed 1, aborted 0 [info] Tests: succeeded 31, failed 0, canceled 0, ignored 0, pending 0 [info] All tests passed. [success] Total time: 2020 s (33:40), completed 2022-4-2 12:35:52 ``` PS. #23546 introduces deleted code and `DepsTestsSuite`. `DepsTestsSuite` can check re-write primary resource. This PR can pass `DepsTestsSuite`, which can prove deletion about `renameMainAppResource` in `baseDriverContainer` does not affect the process about re-write primary resource. Closes #36044 from dcoliversun/SPARK-38770. Authored-by: Qian.Sun <qian.sun2020@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

skonto changed the title ~~[SPARK-23153][K8s] Support client dependencies for HCFS~~ [SPARK-23153][K8s] Support client dependencies with a HCFS Jan 15, 2019

skonto force-pushed the support-client-deps branch from 20603a4 to 7597e03 Compare January 15, 2019 04:57

skonto force-pushed the support-client-deps branch from 7597e03 to bf5f3b1 Compare January 15, 2019 05:01

srowen requested changes Jan 15, 2019

View reviewed changes

skonto commented Jan 15, 2019

View reviewed changes

erikerlandson reviewed Jan 15, 2019

View reviewed changes

rvesse reviewed Jan 17, 2019

View reviewed changes

liyinan926 reviewed Jan 18, 2019

View reviewed changes

skonto mentioned this pull request Feb 1, 2019

[SPARK-24793][K8s] Enhance spark-submit for app management #23599

Closed

skonto changed the title ~~[SPARK-23153][K8s] Support client dependencies with a HCFS~~ [SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System Feb 4, 2019

skonto force-pushed the support-client-deps branch from bf5f3b1 to c885132 Compare February 4, 2019 21:12

skonto force-pushed the support-client-deps branch from c885132 to 40d89fa Compare February 4, 2019 21:43

skonto force-pushed the support-client-deps branch from 29ecdeb to ec2d66c Compare May 16, 2019 09:11

srowen reviewed May 17, 2019

View reviewed changes

srowen self-requested a review May 17, 2019 19:06

srowen reviewed May 17, 2019

View reviewed changes

support client deps

3c58f7b

skonto force-pushed the support-client-deps branch from ec2d66c to 3c58f7b Compare May 17, 2019 19:15

srowen approved these changes May 20, 2019

View reviewed changes

erikerlandson closed this in 5e74570 May 22, 2019

dongjoon-hyun added the KUBERNETES label Jul 27, 2019

skonto mentioned this pull request Oct 11, 2019

[SPARK-27936][K8S] Support python deps #25870

Closed

This was referenced Feb 26, 2021

Spark3 diff on top of tag 3.0.1 palantir/spark#735

Closed

Palantir Spark Diff on v3.0.2 palantir/spark#737

Closed

dcoliversun mentioned this pull request Apr 2, 2022

[SPARK-38770][K8S] Remove renameMainAppResource from baseDriverContainer #36044

Closed

[SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System #23546

[SPARK-23153][K8s] Support client dependencies with a Hadoop Compatible File System #23546

Conversation

skonto commented Jan 15, 2019 • edited Loading

What changes were proposed in this pull request?

How was this patch tested?

SparkQA commented Jan 15, 2019

SparkQA commented Jan 15, 2019

SparkQA commented Jan 15, 2019

SparkQA commented Jan 15, 2019

skonto commented Jan 15, 2019 • edited Loading

skonto commented Jan 15, 2019 • edited Loading

SparkQA commented Jan 15, 2019

skonto commented Jan 15, 2019

SparkQA commented Jan 15, 2019

SparkQA commented Jan 15, 2019

skonto commented Jan 15, 2019

SparkQA commented Jan 15, 2019

SparkQA commented Jan 15, 2019

skonto commented Jan 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skonto Jan 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skonto Jan 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skonto Jan 15, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skonto Jan 22, 2019 • edited Loading

Choose a reason for hiding this comment

rvesse left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skonto Jan 22, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

skonto Jan 22, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SparkQA commented Feb 4, 2019

SparkQA commented Feb 4, 2019

SparkQA commented Feb 4, 2019

skonto commented May 16, 2019

skonto commented May 16, 2019 • edited Loading

SparkQA commented May 16, 2019

SparkQA commented May 16, 2019

SparkQA commented May 16, 2019

erikerlandson commented May 17, 2019

srowen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

srowen left a comment

Choose a reason for hiding this comment

skonto commented May 17, 2019 • edited Loading

SparkQA commented May 17, 2019

SparkQA commented May 17, 2019

SparkQA commented May 17, 2019

skonto commented May 20, 2019

skonto commented May 20, 2019 • edited Loading

skonto commented Jan 15, 2019 •

edited

Loading

skonto commented Jan 15, 2019 •

edited

Loading

skonto commented Jan 15, 2019 •

edited

Loading

skonto Jan 15, 2019 •

edited

Loading

skonto Jan 15, 2019 •

edited

Loading

skonto Jan 15, 2019 •

edited

Loading

skonto Jan 22, 2019 •

edited

Loading

rvesse left a comment •

edited

Loading

skonto Jan 22, 2019 •

edited

Loading

skonto Jan 22, 2019 •

edited

Loading

skonto commented May 16, 2019 •

edited

Loading

skonto commented May 17, 2019 •

edited

Loading

skonto commented May 20, 2019 •

edited

Loading

koertkuipers commented May 13, 2020 •

edited

Loading