Add missing dockerfiles#18115

Closed

mccheah wants to merge 491 commits intoapache:masterfrom

apache-spark-on-k8s:add-missing-dockerfiles

This pull request is big! We're only showing the most recent 250 commits

Commits on Nov 30, 2016

[SPARK-18318][ML] ML, Graph 2.1 QA: API: New Scala APIs, docs

yanboliang
authored and
jkbradley
committed
[SPARK-18251][SQL] the type of Dataset can't be Option of non-flat type

cloud-fan
authored and
liancheng
committed
[SPARK-18546][CORE] Fix merging shuffle spills when using encryption.
Marcelo Vanzin
committed

Commits on Dec 1, 2016

Commits on Dec 2, 2016

Commits on Dec 4, 2016

[SPARK-18081][ML][DOCS] Add user guide for Locality Sensitive Hashing(LSH)

Yunni
authored and
jkbradley
committed
[SPARK-18091][SQL] Deep if expressions cause Generated SpecificUnsafeProjection code to exceed JVM code size limit

Kapil Singh
authored and
cloud-fan
committed
[SPARK-18661][SQL] Creating a partitioned datasource table should not scan all files for table

ericl
authored and
cloud-fan
committed

Commits on Dec 5, 2016

Commits on Dec 6, 2016

[SPARK-18634][PYSPARK][SQL] Corruption and Correctness issues with exploding Python UDFs

viirya
authored and
hvanhovell
committed
[SPARK-18729][SS] Move DataFrame.collect out of synchronized block in MemorySink

zsxwing
authored and
tdas
committed
[SPARK-18657][SPARK-18668] Make StreamingQuery.id persists across restart and not auto-generate StreamingQuery.name
tdas
committed
[SPARK-18722][SS] Move no data rate limit from StreamExecution to ProgressReporter

zsxwing
authored and
tdas
committed
[SPARK-18572][SQL] Add a method `listPartitionNames` to `ExternalCatalog`

Michael Allman
authored and
cloud-fan
committed
[SPARK-18721][SS] Fix ForeachSink with watermark + append

zsxwing
authored and
tdas
committed
[SPARK-18634][SQL][TRIVIAL] Touch-up Generate
hvanhovell
committed
[SPARK-18714][SQL] Add a simple time function to SparkSession

rxin
authored and
hvanhovell
committed
[SPARK-18671][SS][TEST] Added tests to ensure stability of that all Structured Streaming log formats

tdas
authored and
zsxwing
committed
[SPARK-18652][PYTHON] Include the example data and third-party licenses in pyspark package.

lins05
authored and
srowen
committed

Commits on Dec 7, 2016

[SPARK-18734][SS] Represent timestamp in StreamingQueryProgress as formatted string instead of millis

tdas
authored and
zsxwing
committed
[SPARK-18671][SS][TEST-MAVEN] Follow up PR to fix test for Maven
tdas
committed
[SPARK-18686][SPARKR][ML] Several cleanup and improvements for spark.logit.
yanboliang
committed
[SPARK-18701][ML] Fix Poisson GLM failure due to wrong initialization

actuaryzhang
authored and
srowen
committed
[SPARK-18678][ML] Skewed reservoir sampling in SamplingUtils
srowen
committed
[SPARK-18208][SHUFFLE] Executor OOM due to a growing LongArray in BytesToBytesMap

Jie Xiong
authored and
hvanhovell
committed
[SPARK-17760][SQL] AnalysisException with dataframe pivot when groupBy column is not attribute

aray
authored and
hvanhovell
committed
[SPARK-18764][CORE] Add a warning log when skipping a corrupted file
zsxwing
committed
[SPARK-18762][WEBUI] Web UI should be http:4040 instead of https:4040

sarutak
authored and
Marcelo Vanzin
committed
[SPARK-18588][TESTS] Fix flaky test: KafkaSourceStressForDontFailOnDataLossSuite

zsxwing
authored and
tdas
committed
[SPARK-18754][SS] Rename recentProgresses to recentProgress

marmbrus
authored and
tdas
committed

Commits on Dec 8, 2016

Commits on Dec 9, 2016

[SPARK-18776][SS] Make Offset for FileStreamSource corrected formatted in json
tdas
committed
[SPARK-18774][CORE][SQL] Ignore non-existing files when ignoreCorruptFiles is enabled (branch 2.1)

zsxwing
authored and
rxin
committed
[SPARKR][PYSPARK] Fix R source package name to match Spark version. Remove pip tar.gz from distribution
shivaram
committed
[SPARK-18349][SPARKR] Update R API documentation on ml model summary

wangmiao1981
authored and
Felix Cheung
committed
Copy the SparkR source package with LFTP
shivaram
committed
Copy pyspark and SparkR packages to latest release dir too

felixcheung
authored and
shivaram
committed
[SPARK-18637][SQL] Stateful UDF should be considered as nondeterministic

Zhan Zhang
authored and
cloud-fan
committed
[MINOR][CORE][SQL][DOCS] Typo fixes

jaceklaskowski
authored and
srowen
committed
[SPARK-17822][R] Make JVMObjectTracker a member variable of RBackend
mengxr
committed
[MINOR][SPARKR] Fix SparkR regex in copy command
shivaram
committed
[SPARK-18745][SQL] Fix signed integer overflow due to toInt cast

kiszk
authored and
hvanhovell
committed

Commits on Dec 10, 2016

[SPARK-18812][MLLIB] explain "Spark ML"
mengxr
committed
[SPARK-18807][SPARKR] Should suppress output print for calls to JVM methods with void return values

felixcheung
authored and
shivaram
committed
[SPARK-18811] StreamSource resolution should happen in stream execution thread

brkyvz
authored and
zsxwing
committed
[SPARK-17460][SQL] Make sure sizeInBytes in Statistics will not overflow

huaxingao
authored and
cloud-fan
committed
[MINOR][DOCS] Remove Apache Spark Wiki address

dongjoon-hyun
authored and
srowen
committed
[SPARK-3359][DOCS] Fix greater-than symbols in Javadoc to allow building with Java 8

michalsenkyr
authored and
srowen
committed

Commits on Dec 11, 2016

[SPARK-18815][SQL] Fix NPE when collecting column stats for string/binary column having only null values

wangzhenhua
authored and
rxin
committed
[SQL][MINOR] simplify a test to fix the maven tests

cloud-fan
authored and
srowen
committed
[SPARK-18628][ML] Update Scala param and Python param to have quotes

krishnakalyan3
authored and
srowen
committed

Commits on Dec 12, 2016

[SPARK-18790][SS] Keep a general offset history of stream batches

Tyson Condie
authored and
zsxwing
committed
[DOCS][MINOR] Clarify Where AccumulatorV2s are Displayed

Bill Chambers
authored and
srowen
committed
[SPARK-18681][SQL] Fix filtering to compatible with partition keys of type int

wangyum
authored and
hvanhovell
committed
[SPARK-18810][SPARKR] SparkR install.spark does not work for RCs, snapshots

felixcheung
authored and
shivaram
committed

Commits on Dec 13, 2016

[SPARK-18796][SS] StreamingQueryManager should not block when starting a query

zsxwing
authored and
tdas
committed
[SPARK-18797][SPARKR] Update spark.logit in sparkr-vignettes

wangmiao1981
authored and
mengxr
committed
[SPARK-18835][SQL] Don't expose Guava types in the JavaTypeInference API.
Marcelo Vanzin
committed
[SPARK-18840][YARN] Avoid throw exception when getting token renewal interval in non HDFS security environment

jerryshao
authored and
Marcelo Vanzin
committed
[SPARK-18816][WEB UI] Executors Logs column only ran visibility check on initial table load

ajbozarth
authored and
srowen
committed
[SPARK-18843][CORE] Fix timeout in awaitResultInForkJoinSafely (branch 2.1, 2.0)
zsxwing
committed
[SPARK-18834][SS] Expose event time stats through StreamingQueryProgress
tdas
committed

Commits on Dec 14, 2016

Commits on Dec 15, 2016

Commits on Mar 8, 2017

[SPARK-18278] Minimal support for submitting to Kubernetes.

mccheah
authored and
ash211
committed
Fix style

mccheah
authored and
ash211
committed
Make naming more consistent

mccheah
authored and
ash211
committed
Fix building assembly with Kubernetes.

mccheah
authored and
ash211
committed
Service account support, use constants from fabric8 library.

mccheah
authored and
ash211
committed
Some small changes

mccheah
authored and
ash211
committed
Use k8s:// formatted URL instead of separate setting.

mccheah
authored and
ash211
committed
Reindent comment to conforn to JavaDoc style

foxish
authored and
ash211
committed
Move kubernetes under resource-managers folder.

mccheah
authored and
ash211
committed
Use tar and gzip to compress+archive shipped jars (#2)

mccheah
authored and
ash211
committed
Use alpine and java 8 for docker images. (#10)

mccheah
authored and
ash211
committed
Copy the Dockerfiles from docker-minimal-bundle into the distribution. (#12)

mccheah
authored and
ash211
committed
inherit IO (#13)

foxish
authored and
ash211
committed
Error messages when the driver container fails to start. (#11)

mccheah
authored and
ash211
committed
Fix linter error to make CI happy (#18)

foxish
authored and
ash211
committed
Documentation for the current state of the world (#16)

mccheah
authored and
ash211
committed
Development workflow documentation for the current state of the world. (#20)

mccheah
authored and
ash211
committed
Added service name as prefix to executor pods (#14)

foxish
authored and
ash211
committed
Add kubernetes profile to travis CI yml file (#21)

kimoonkim
authored and
ash211
committed
Improved the example commands in running-on-k8s document. (#25)

lins05
authored and
ash211
committed
Fix spacing for command highlighting (#31)

foxish
authored and
ash211
committed
Support custom labels on the driver pod. (#27)

mccheah
authored and
ash211
committed
Make pod name unique using the submission timestamp (#32)

foxish
authored and
ash211
committed
A number of small tweaks to the MVP. (#23)

mccheah
authored and
ash211
committed
Correct hadoop profile: hadoop2.7 -> hadoop-2.7 (#41)
ash211
committed
Support setting the driver pod launching timeout. (#36)

lins05
authored and
ash211
committed
Sanitize kubernetesAppId for use in secret, service, and pod names (#45)
ash211
committed
Support spark.driver.extraJavaOptions (#48)

kimoonkim
authored and
ash211
committed
Use "extraScalaTestArgs" to pass extra options to scalatest. (#52)

lins05
authored and
ash211
committed
Use OpenJDK8's official Alpine image. (#51)

mccheah
authored and
ash211
committed
Remove unused driver extra classpath upload code (#54)

mccheah
authored and
ash211
committed
Fix k8s integration tests (#44)

lins05
authored and
ash211
committed
Added GC to components (#56)

foxish
authored and
ash211
committed
Create README to better describe project purpose (#50)
ash211
committed
Access the Driver Launcher Server over NodePort for app launch + submit jars (#30)

mccheah
authored and
ash211
committed
Extract constants and config into separate file. Launch => Submit. (#65)

mccheah
authored and
ash211
committed
Retry the submit-application request to multiple nodes (#69)

mccheah
authored and
ash211
committed
Allow adding arbitrary files (#71)

mccheah
authored and
ash211
committed
Fix NPE around unschedulable pod specs (#79)
ash211
committed
Introduce blocking submit to kubernetes by default (#53)
ash211
committed
Do not wait for pod finishing in integration tests. (#84)

lins05
authored and
ash211
committed
Check for user jars/files existence before creating the driver pod. (#86)

lins05
authored and
ash211
committed
Use readiness probe instead of client-side ping. (#75)

mccheah
authored and
ash211
committed
Note integration tests require Java 8 (#99)
ash211
committed
Bumping up kubernetes-client version to fix GKE and local proxy (#105)

foxish
authored and
ash211
committed
Truncate k8s hostnames to be no longer than 63 characters (#102)
ash211
committed
Fixed loading the executors page through the kubectl proxy. (#95)

lins05
authored and
ash211
committed
Filter nodes to only try and send files to external IPs (#106)

foxish
authored and
ash211
committed
Parse results of minikube status more rigorously (#97)
ash211
committed
Adding legacyHostIP to the list of IPs we look at (#114)

foxish
authored and
ash211
committed
Add -DskipTests to dev docs (#115)
ash211
committed
Shutdown the thread scheduler in LoggingPodStatusWatcher on receiving job finish event notifications (#121)

varunkatta
authored and
ash211
committed
Trigger scalatest plugin in the integration-test phase (#93)

kimoonkim
authored and
ash211
committed
Fix issue with DNS resolution (#118)

foxish
authored and
ash211
committed
Change the API contract for uploading local files (#107)

mccheah
authored and
ash211
committed
Optionally expose the driver UI port as NodePort (#131)

kimoonkim
authored and
ash211
committed
Set the REST service's exit code to the exit code of its driver subprocess (#138)
ash211
committed
Pass the actual iterable from the option to get files (#139)

mccheah
authored and
ash211
committed
Use a separate class to track components that need to be cleaned up (#122)

mccheah
authored and
ash211
committed
Enable unit tests in Travis CI build (#132)

kimoonkim
authored and
ash211
committed
Change driver pod's restart policy from OnFailure to Never (#145)
ash211
committed
Extract SSL configuration handling to a separate class (#123)

mccheah
authored and
ash211
committed
Exclude known flaky tests (#156)

kimoonkim
authored and
ash211
committed
Richer logging and better error handling in driver pod watch (#154)

foxish
authored and
ash211
committed
Document blocking submit calls (#152)
ash211
committed
Allow custom annotations on the driver pod. (#163)

mccheah
authored and
ash211
committed
Update client version & minikube version (#142)

foxish
authored and
ash211
committed
Allow customizing external URI provision + External URI can be set via annotations (#147)

mccheah
authored and
ash211
committed
Remove okhttp from top-level pom (#166)

foxish
authored and
ash211
committed
Allow setting memory on the driver submission server. (#161)

mccheah
authored and
ash211
committed
Add a section for prerequisites (#171)

foxish
authored and
ash211
committed
Add instructions to find master URL (#169)

foxish
authored and
ash211
committed
Propagate exceptions (#172)

mccheah
authored and
ash211
committed
Logging for resource deletion (#170)
ash211
committed

Commits on Mar 14, 2017

Fix pom versions (#178)
foxish
authored

Commits on Mar 15, 2017

Exclude flaky ExternalShuffleServiceSuite from Travis (#185)

kimoonkim
authored and
ash211
committed

Commits on Mar 16, 2017

Commits on Mar 17, 2017

Allow the driver pod's credentials to be shipped from the submission client (squashed) (#192)
ash211
authored

Commits on Mar 20, 2017

Support using PEM files to configure SSL for driver submission (#173)

mccheah
authored and
ash211
committed

Commits on Mar 21, 2017

Update tags on docker images. (#196)

foxish
authored and
ash211
committed

Commits on Mar 22, 2017

Add additional instructions to use release tarball (#198)

foxish
authored and
ash211
committed

Commits on Mar 30, 2017

Support specify CPU cores for driver pod (#207)

hustcat
authored and
ash211
committed

Commits on Apr 5, 2017

Register executors using pod IPs instead of pod host names (#215)

kimoonkim
authored and
foxish
committed

Commits on Apr 10, 2017

Upgrade bouncycastle, force bcprov version (#223)

mccheah
authored and
foxish
committed

Commits on Apr 13, 2017

Stop executors cleanly before deleting their pods (#231)

ash211
authored and
foxish
committed

Commits on Apr 14, 2017

Upgrade Kubernetes client to 2.2.13. (#230)
mccheah
authored

Commits on Apr 17, 2017

Respect JVM http proxy settings when using Feign. (#228)

mccheah
authored and
ash211
committed

Commits on Apr 21, 2017

Staging server for receiving application dependencies. (#212)

mccheah
authored and
ash211
committed
Reorganize packages between v1 work and v2 work (#220)

mccheah
authored and
ash211
committed
Support SSL on the file staging server (#221)

mccheah
authored and
ash211
committed

Commits on Apr 25, 2017

Driver submission with mounting dependencies from the staging server (#227)

mccheah
authored and
foxish
committed

Commits on May 2, 2017

Enable testing against GCE clusters (#243)

foxish
authored and
mccheah
committed
Update running-on-kubernetes.md (#259)

erikerlandson
authored and
foxish
committed

Commits on May 3, 2017

Build with sbt and fix scalastyle checks. (#241)

lins05
authored and
foxish
committed
Updating images in doc (#219)
foxish
authored

Commits on May 5, 2017

Correct readme links (#266)

johscheuer
authored and
foxish
committed

Commits on May 9, 2017

edit readme with a working build example command (#254)

erikerlandson
authored and
foxish
committed

Commits on May 10, 2017

Fix watcher conditional logic (#269)

erikerlandson
authored and
ash211
committed
Dispatch tasks to right executors that have tasks' input HDFS data (#216)

kimoonkim
authored and
ash211
committed

Commits on May 16, 2017

Add parameter for driver pod name (#258)

hustcat
authored and
foxish
committed

Commits on May 17, 2017

Commits on May 18, 2017

Exit properly when the k8s cluster is not available. (#256)

lins05
authored and
ash211
committed
Support driver pod kubernetes credentials mounting in V2 submission (#246)

mccheah
authored and
ash211
committed

Commits on May 19, 2017

Allow client certificate PEM for resource staging server. (#257)

mccheah
authored and
ash211
committed
Differentiate between URI and SSL settings for in-cluster vs. submission (#281)

mccheah
authored and
ash211
committed

Commits on May 22, 2017

Monitor pod status in submission v2. (#283)

mccheah
authored and
ash211
committed

Commits on May 23, 2017

Replace submission v1 with submission v2. (#286)

mccheah
authored and
ash211
committed
Added files should be in the working directories. (#294)

mccheah
authored and
ash211
committed

Commits on May 24, 2017

Add missing license (#296)

mccheah
authored and
ash211
committed
Remove some leftover code and fix a constant. (#297)

mccheah
authored and
ash211
committed

Commits on May 25, 2017

Adding restart policy fix for v2 (#303)

foxish
authored and
mccheah
committed
Add all dockerfiles to distributions.
mccheah
committed