Merge upstream#299

Merged

ash211 merged 358 commits intomasterfrom

rk/merge-upstream

Jan 25, 2018

This pull request is big! We're only showing the most recent 250 commits

Commits on Dec 13, 2017

[SPARK-22763][CORE] SHS: Ignore unknown events and parse through the file

gengliangwang
authored and
gatorsmile
committed
[SPARK-22574][MESOS][SUBMIT] Check submission request parameters

Gschiavon
authored and
Marcelo Vanzin
committed
[SPARK-22772][SQL] Use splitExpressionsWithCurrentInputs to split codes in elt

viirya
authored and
gatorsmile
committed
[SPARK-22764][CORE] Fix flakiness in SparkContextSuite.

Marcelo Vanzin
authored and
squito
committed
[SPARK-22600][SQL][FOLLOW-UP] Fix a compilation error in TPCDS q75/q77

maropu
authored and
gatorsmile
committed

Commits on Dec 14, 2017

Revert "[SPARK-22600][SQL][FOLLOW-UP] Fix a compilation error in TPCDS q75/q77"
cloud-fan
committed
Revert "[SPARK-22600][SQL] Fix 64kb limit for deeply nested expressions under wholestage codegen"
cloud-fan
committed
[SPARK-3181][ML] Implement huber loss for LinearRegression.
yanboliang
committed
[SPARK-22732] Add Structured Streaming APIs to DataSourceV2

jose-torres
authored and
zsxwing
committed
[SPARK-22779][SQL] Resolve default values for fallback configs.

Marcelo Vanzin
authored and
gatorsmile
committed
[SPARK-22775][SQL] move dictionary related APIs from ColumnVector to WritableColumnVector
cloud-fan
committed
[SPARK-22785][SQL] remove ColumnVector.anyNullsSet
cloud-fan
committed
[SPARK-22774][SQL][TEST] Add compilation check into TPCDSQuerySuite

kiszk
authored and
cloud-fan
committed
[SPARK-16496][SQL] Add wholetext as option for reading text in SQL.

ScrapCodes
authored and
gatorsmile
committed
[SPARK-22660][BUILD] Use position() and limit() to fix ambiguity issue in scala-2.12

kellyzly
authored and
srowen
committed
[SPARK-22778][KUBERNETES] Added the missing service metadata for KubernetesClusterManager

liyinan926
authored and
Marcelo Vanzin
committed
[SPARK-22733] Split StreamExecution into MicroBatchExecution and StreamExecution.

jose-torres
authored and
zsxwing
committed
[SPARK-22496][SQL] thrift server adds operation logs

zouchenjun
authored and
gatorsmile
committed

Commits on Dec 16, 2017

[SPARK-22811][PYSPARK][ML] Fix pyspark.ml.tests failure when Hive is not available.

MrBago
authored and
HyukjinKwon
committed

Commits on Dec 17, 2017

[SPARK-22817][R] Use fixed testthat version for SparkR tests in AppVeyor
HyukjinKwon
committed
[MINOR][DOC] Fix the link of 'Getting Started'

mcavdar
authored and
srowen
committed
[SPARK-22816][TEST] Basic tests for PromoteStrings and InConversion

wangyum
authored and
gatorsmile
committed

Commits on Dec 18, 2017

[SPARK-22807][SCHEDULER] Remove config that says docker and replace with container

foxish
authored and
Marcelo Vanzin
committed
[SPARK-20653][CORE] Add cleaning of old elements from the status store.

Marcelo Vanzin
authored and
squito
committed
Revert "Revert "[SPARK-22496][SQL] thrift server adds operation logs""
HyukjinKwon
committed
[SPARK-22813][BUILD] Use lsof or /usr/sbin/lsof in run-tests.py

kiszk
authored and
HyukjinKwon
committed
[SPARK-22777][SCHEDULER] Kubernetes mode dockerfile permission and distribution

foxish
authored and
Marcelo Vanzin
committed

Commits on Dec 19, 2017

[MINOR][SQL] Remove Useless zipWithIndex from ResolveAliases

gatorsmile
authored and
cloud-fan
committed
[SPARK-22673][SQL] InMemoryRelation should utilize existing stats whenever possible

CodingCat
authored and
cloud-fan
committed
[SPARK-21984][SQL] Join estimation based on equi-height histogram

Zhenhua Wang
authored and
cloud-fan
committed
[SPARK-22791][SQL][SS] Redact Output of Explain

gatorsmile
authored and
cloud-fan
committed
[SPARK-22815][SQL] Keep PromotePrecision in Optimized Plans

gatorsmile
authored and
cloud-fan
committed
[SPARK-18016][SQL] Code Generation: Constant Pool Limit - reduce entries for mutable state

kiszk
authored and
cloud-fan
committed
[SPARK-21652][SQL][FOLLOW-UP] Fix rule conflict caused by InferFiltersFromConstraints
gatorsmile
committed
[SPARK-22821][TEST] Basic tests for WidenSetOperationTypes, BooleanEquality, StackCoercion and Division

wangyum
authored and
gatorsmile
committed

Commits on Dec 20, 2017

[SPARK-22827][CORE] Avoid throwing OutOfMemoryError in case of exception in spill

Sital Kedia
authored and
cloud-fan
committed
[SPARK-22829] Add new built-in function date_trunc()

youngbink
authored and
gatorsmile
committed
[SPARK-22649][PYTHON][SQL] Adding localCheckpoint to Dataset API

ferdonline
authored and
gatorsmile
committed
[SPARK-22781][SS] Support creating streaming dataset with ORC files

dongjoon-hyun
authored and
zsxwing
committed
[SPARK-22788][STREAMING] Use correct hadoop config for fs append support.

Marcelo Vanzin
authored and
squito
committed
[SPARK-22824] Restore old offset for binary compatibility

jose-torres
authored and
zsxwing
committed
[SPARK-22832][ML] BisectingKMeans unpersist unused datasets

zhengruifeng
authored and
srowen
committed
[SPARK-22849] ivy.retrieve pattern should also consider `classifier`
gatorsmile
committed
[SPARK-22830] Scala Coding style has been improved in Spark Examples

chetkhatri
authored and
srowen
committed
[SPARK-22847][CORE] Remove redundant code in AppStatusListener while assigning schedulingPool for stage

Ngone51
authored and
Marcelo Vanzin
committed

Commits on Dec 21, 2017

[SPARK-22845][SCHEDULER] Modify spark.kubernetes.allocation.batch.delay to take time instead of int

foxish
authored and
Marcelo Vanzin
committed
[SPARK-22810][ML][PYSPARK] Expose Python API for LinearRegression with huber loss.
yanboliang
committed
[SPARK-22387][SQL] Propagate session configs to data source read/write options

jiangxb1987
authored and
cloud-fan
committed
[SPARK-19634][SQL][ML][FOLLOW-UP] Improve interface of dataframe vectorized summarizer

WeichenXu123
authored and
yanboliang
committed
[SPARK-22848][SQL] Eliminate mutable state from Stack

kiszk
authored and
cloud-fan
committed
[SPARK-22324][SQL][PYTHON] Upgrade Arrow to 0.8.0

BryanCutler
authored and
HyukjinKwon
committed
[SPARK-22852][BUILD] Exclude -Xlint:unchecked from sbt javadoc flags

easel
authored and
srowen
committed
[SPARK-22855][BUILD] Add -no-java-comments to sbt docs/scalacOptions

easel
authored and
srowen
committed
[SPARK-22668][SQL] Ensure no global variables in arguments of method split by CodegenContext.splitExpressions()
cloud-fan
committed
[SPARK-22786][SQL] only use AppStatusPlugin in history server
cloud-fan
committed
[SPARK-22822][TEST] Basic tests for WindowFrameCoercion and DecimalPrecision

wangyum
authored and
gatorsmile
committed
[SPARK-22042][FOLLOW-UP][SQL] ReorderJoinPredicates can break when child's partitioning is not decided

tejasapatil
authored and
gatorsmile
committed
[SPARK-22861][SQL] SQLAppStatusListener handles multi-job executions.

squito
authored and
Marcelo Vanzin
committed

Commits on Dec 22, 2017

[SPARK-22648][K8S] Spark on Kubernetes - Documentation

foxish
authored and
rxin
committed
[SPARK-22854][UI] Read Spark version from event logs.

Marcelo Vanzin
authored and
cloud-fan
committed
[SPARK-22750][SQL] Reuse mutable states when possible

mgaido91
authored and
cloud-fan
committed
[SPARK-22450][CORE][MLLIB][FOLLOWUP] safely register class for mllib - LabeledPoint/VectorWithNorm/TreePoint

zhengruifeng
authored and
srowen
committed
[SPARK-22866][K8S] Fix path issue in Kubernetes dockerfile

foxish
authored and
mridulm
committed
[SPARK-22862] Docs on lazy elimination of columns missing from an encoder

marmbrus
authored and
gatorsmile
committed
[SPARK-22874][PYSPARK][SQL] Modify checking pandas version to use LooseVersion.

ueshin
authored and
HyukjinKwon
committed
[SPARK-22346][ML] VectorSizeHint Transformer for using VectorAssembler in StructuredSteaming

MrBago
authored and
jkbradley
committed

Commits on Dec 23, 2017

[SPARK-22789] Map-only continuous processing execution

jose-torres
authored and
zsxwing
committed
[SPARK-22833][IMPROVEMENT] in SparkHive Scala Examples

chetkhatri
authored and
srowen
committed
[SPARK-20694][EXAMPLES] Update SQLDataSourceExample.scala

CNRui
authored and
srowen
committed
[HOTFIX] Fix Scala style checks
HyukjinKwon
committed
[SPARK-22844][R] Adds date_trunc in R API
HyukjinKwon
committed
[SPARK-22889][SPARKR] Set overwrite=T when install SparkR in tests

shivaram
authored and
Felix Cheung
committed

Commits on Dec 24, 2017

[SPARK-22465][CORE] Add a safety-check to RDD defaultPartitioner

sujithjay
authored and
mridulm
committed

Commits on Dec 25, 2017

Commits on Dec 26, 2017

[SPARK-22893][SQL][HOTFIX] Fix a error message of VersionsSuite

dongjoon-hyun
authored and
gatorsmile
committed
[SPARK-20168][DSTREAM] Add changes to use kinesis fetches from specific timestamp

yashs360
authored and
brkyvz
committed
[SPARK-21552][SQL] Add DecimalType support to ArrowWriter.

ueshin
authored and
HyukjinKwon
committed
[SPARK-22901][PYTHON] Add deterministic flag to pyspark UDF

mgaido91
authored and
gatorsmile
committed
[SPARK-22833][EXAMPLE] Improvement SparkHive Scala Examples

cloud-fan
authored and
gatorsmile
committed
[SPARK-22894][SQL] DateTimeOperations should accept SQL like string type

wangyum
authored and
gatorsmile
committed

Commits on Dec 27, 2017

[SPARK-22846][SQL] Fix table owner is null when creating table through spark sql or thriftserver

xu.wenchun
authored and
cloud-fan
committed
[SPARK-22324][SQL][PYTHON][FOLLOW-UP] Update setup.py file.

ueshin
authored and
HyukjinKwon
committed
[SPARK-22904][SQL] Add tests for decimal operations and string casts

mgaido91
authored and
gatorsmile
committed

Commits on Dec 28, 2017

[SPARK-22899][ML][STREAMING] Fix OneVsRestModel transform on streaming data failed.

WeichenXu123
authored and
jkbradley
committed
[SPARK-18016][SQL][FOLLOW-UP] Code Generation: Constant Pool Limit - reduce entries for mutable state

kiszk
authored and
cloud-fan
committed
[SPARK-22909][SS] Move Structured Streaming v2 APIs to streaming folder

zsxwing
authored and
cloud-fan
committed
[SPARK-22757][KUBERNETES] Enable use of remote dependencies (http, s3, gcs, etc.) in Kubernetes mode

liyinan926
authored and
ueshin
committed
[SPARK-22648][K8S] Add documentation covering init containers and secrets

liyinan926
authored and
ueshin
committed
[SPARK-22843][R] Adds localCheckpoint in R
HyukjinKwon
committed
[SPARK-21208][R] Adds setLocalProperty and getLocalProperty in R
HyukjinKwon
committed
[SPARK-20392][SQL][FOLLOWUP] should not add extra AnalysisBarrier

cloud-fan
authored and
gatorsmile
committed
[SPARK-22917][SQL] Should not try to generate histogram for empty/null columns

Zhenhua Wang
authored and
cloud-fan
committed
[MINOR][BUILD] Fix Java linter errors

dongjoon-hyun
authored and
srowen
committed
[SPARK-22875][BUILD] Assembly build fails for a high user id

gerashegalov
authored and
srowen
committed
[SPARK-22836][UI] Show driver logs in UI when available.

Marcelo Vanzin
authored and
squito
committed
[SPARK-22890][TEST] Basic tests for DateTimeOperations

wangyum
authored and
gatorsmile
committed
[SPARK-11035][CORE] Add in-process Spark app launcher.

Marcelo Vanzin
authored and
squito
committed
[SPARK-22818][SQL] csv escape of quote escape

soonmok-kwon
authored and
gatorsmile
committed

Commits on Dec 29, 2017

[SPARK-22905][MLLIB] Fix ChiSqSelectorModel save implementation

WeichenXu123
authored and
jkbradley
committed
[SPARK-22313][PYTHON][FOLLOWUP] Explicitly import warnings namespace in flume.py

HyukjinKwon
authored and
ueshin
committed
[SPARK-22891][SQL] Make hive client creation thread safe

Feng Liu
authored and
gatorsmile
committed
[SPARK-22834][SQL] Make insertion commands have real children to fix UI issues

gengliangwang
authored and
cloud-fan
committed
[SPARK-22892][SQL] Simplify some estimation logic by using double instead of decimal

Zhenhua Wang
authored and
cloud-fan
committed
[SPARK-22916][SQL] shouldn't bias towards build right if user does not specify

Feng Liu
authored and
gatorsmile
committed
[SPARK-21657][SQL] optimize explode quadratic memory consumpation

uzadude
authored and
cloud-fan
committed
[SPARK-22921][PROJECT-INFRA] Choices for Assigning Jira on Merge

squito
authored and
srowen
committed
[SPARK-22370][SQL][PYSPARK][FOLLOW-UP] Fix a test failure when xmlrunner is installed.

ueshin
authored and
HyukjinKwon
committed
[SPARK-20654][CORE] Add config to limit disk usage of the history server.

Marcelo Vanzin
authored and
squito
committed
[SPARK-22864][CORE] Disable allocation schedule in ExecutorAllocationManagerSuite.

Marcelo Vanzin
authored and
squito
committed
[SPARK-22905][ML][FOLLOWUP] Fix GaussianMixtureModel save

zhengruifeng
authored and
jkbradley
committed
[SPARK-22920][SPARKR] sql functions for current_date, current_timestamp, rtrim/ltrim/trim with trimString

felixcheung
authored and
Felix Cheung
committed
[SPARK-22921][PROJECT-INFRA] Bug fix in jira assigning
squito
committed

Commits on Dec 30, 2017

[SPARK-22922][ML][PYSPARK] Pyspark portion of the fit-multiple API

MrBago
authored and
jkbradley
committed
[SPARK-22734][ML][PYSPARK] Added Python API for VectorSizeHint.

MrBago
authored and
jkbradley
committed
[SPARK-22881][ML][TEST] ML regression package testsuite add StructuredStreaming test

WeichenXu123
authored and
jkbradley
committed
[SPARK-22771][SQL] Concatenate binary inputs into a binary output

maropu
authored and
gatorsmile
committed
[SPARK-21475][Core]Revert "[SPARK-21475][CORE] Use NIO's Files API to replace FileInputStream/FileOutputStream in some critical paths"
zsxwing
committed
[TEST][MINOR] remove redundant `EliminateSubqueryAliases` in test code

wzhfy
authored and
gatorsmile
committed
[SPARK-22919] Bump httpclient versions

Fokko Driesprong
authored and
srowen
committed
[SPARK-22924][SPARKR] R API for sortWithinPartitions

felixcheung
authored and
HyukjinKwon
committed

Commits on Dec 31, 2017

[SPARK-22363][SQL][TEST] Add unit test for Window spilling

gaborgsomogyi
authored and
gatorsmile
committed
[SPARK-22895][SQL] Push down the deterministic predicates that are after the first non-deterministic
gatorsmile
committed
[SPARK-22397][ML] add multiple columns support to QuantileDiscretizer

huaxingao
authored and
Nick Pentreath
committed
[SPARK-22801][ML][PYSPARK] Allow FeatureHasher to treat numeric columns as categorical
Nick Pentreath
committed
[MINOR][DOCS] s/It take/It takes/g

jkremser
authored and
srowen
committed
[SPARK-13030][ML] Create OneHotEncoderEstimator for OneHotEncoder as Estimator

viirya
authored and
jkbradley
committed

Commits on Jan 2, 2018

Commits on Jan 3, 2018

Commits on Jan 4, 2018

[SPARK-22944][SQL] improve FoldablePropagation

cloud-fan
authored and
gatorsmile
committed
[SPARK-22933][SPARKR] R Structured Streaming API for withWatermark, trigger, partitionBy

felixcheung
authored and
Felix Cheung
committed
[SPARK-22950][SQL] Handle ChildFirstURLClassLoader's parent

yaooqinn
authored and
cloud-fan
committed
[SPARK-22945][SQL] add java UDF APIs in the functions object
cloud-fan
committed
[SPARK-22939][PYSPARK] Support Spark UDF in registerFunction
gatorsmile
committed
[SPARK-22771][SQL] Add a missing return statement in Concat.checkInputDataTypes

maropu
authored and
gatorsmile
committed
[SPARK-21475][CORE][2ND ATTEMPT] Change to use NIO's Files API for external shuffle service

jerryshao
authored and
zsxwing
committed
[SPARK-22850][CORE] Ensure queued events are delivered to all event queues.

Marcelo Vanzin
authored and
squito
committed
[SPARK-22948][K8S] Move SparkPodInitContainer to correct package.
Marcelo Vanzin
committed
[SPARK-22953][K8S] Avoids adding duplicated secret volumes when init-container is used

liyinan926
authored and
Marcelo Vanzin
committed

Commits on Jan 5, 2018

[SPARK-22960][K8S] Make build-push-docker-images.sh more dev-friendly.
Marcelo Vanzin
committed
[SPARK-22957] ApproxQuantile breaks if the number of rows exceeds MaxInt

juliuszsompolski
authored and
cloud-fan
committed
[SPARK-22825][SQL] Fix incorrect results of Casting Array to String

maropu
authored and
cloud-fan
committed
[SPARK-22949][ML] Apply CrossValidator approach to Driver/Distributed memory tradeoff for TrainValidationSplit

MrBago
authored and
jkbradley
committed
[SPARK-22757][K8S] Enable spark.jars and spark.files in KUBERNETES mode

liyinan926
authored and
Felix Cheung
committed
[SPARK-22961][REGRESSION] Constant columns should generate QueryPlanConstraints

adrian-ionescu
authored and
gatorsmile
committed
[SPARK-22940][SQL] HiveExternalCatalogVersionsSuite should succeed on platforms that don't have wget

bersprockets
authored and
Marcelo Vanzin
committed
[SPARK-13030][ML] Follow-up cleanups for OneHotEncoderEstimator
jkbradley
committed

Commits on Jan 6, 2018

[SPARK-22914][DEPLOY] Register history.ui.port

gerashegalov
authored and
Marcelo Vanzin
committed
[SPARK-22937][SQL] SQL elt output binary for binary inputs

maropu
authored and
gatorsmile
committed
[SPARK-22960][K8S] Revert use of ARG base_image in images

liyinan926
authored and
Marcelo Vanzin
committed
[SPARK-22930][PYTHON][SQL] Improve the description of Vectorized UDFs for non-deterministic cases

icexelloss
authored and
gatorsmile
committed
[SPARK-22793][SQL] Memory leak in Spark Thrift Server

zuotingbing
authored and
gatorsmile
committed
[SPARK-21786][SQL] When acquiring 'compressionCodecClassName' in 'ParquetOptions', `parquet.compression` needs to be considered.

fjh100456
authored and
gatorsmile
committed
[SPARK-22901][PYTHON][FOLLOWUP] Adds the doc for asNondeterministic for wrapped UDF function

HyukjinKwon
authored and
gatorsmile
committed
[HOTFIX] Fix style checking failure
gatorsmile
committed

Commits on Jan 7, 2018

[SPARK-22973][SQL] Fix incorrect results of Casting Map to String

maropu
authored and
cloud-fan
committed

Commits on Jan 8, 2018

[SPARK-22985] Fix argument escaping bug in from_utc_timestamp / to_utc_timestamp codegen

JoshRosen
authored and
gatorsmile
committed
[SPARK-22566][PYTHON] Better error message for `_merge_type` in Pandas to Spark DF conversion

gberger-palantir
authored and
ueshin
committed
[SPARK-22979][PYTHON][SQL] Avoid per-record type dispatch in Python data conversion (EvaluatePython.fromJava)

HyukjinKwon
authored and
cloud-fan
committed
[SPARK-22983] Don't push filters beneath aggregates with empty grouping expressions

JoshRosen
authored and
gatorsmile
committed
[SPARK-21865][SQL] simplify the distribution semantic of Spark SQL
cloud-fan
committed
[SPARK-22952][CORE] Deprecate stageAttemptId in favour of stageAttemptNumber

advancedxy
authored and
cloud-fan
committed
[SPARK-22992][K8S] Remove assumption of the DNS domain

foxish
authored and
Marcelo Vanzin
committed
[SPARK-22912] v2 data source support in MicroBatchExecution

jose-torres
authored and
tdas
committed

Commits on Jan 9, 2018

[SPARK-22972] Couldn't find corresponding Hive SerDe for data source provider org.apache.spark.sql.hive.orc

xubo245
authored and
gatorsmile
committed
[SPARK-22990][CORE] Fix method isFairScheduler in JobsTab and StagesTab

gengliangwang
authored and
cloud-fan
committed
[SPARK-22984] Fix incorrect bitmap copying and offset adjustment in GenerateUnsafeRowJoiner

JoshRosen
authored and
cloud-fan
committed
[SPARK-21292][DOCS] refreshtable example

felixcheung
authored and
Felix Cheung
committed
[SPARK-21293][SPARKR][DOCS] structured streaming doc update

felixcheung
authored and
Felix Cheung
committed
[SPARK-23000] Fix Flaky test suite DataSourceWithHiveMetastoreCatalogSuite in Spark 2.3

gatorsmile
authored and
cloud-fan
committed
[SPARK-22998][K8S] Set missing value for SPARK_MOUNTED_CLASSPATH in the executors

liyinan926
authored and
Felix Cheung
committed
[SPARK-16060][SQL] Support Vectorized ORC Reader

dongjoon-hyun
authored and
cloud-fan
committed
[SPARK-22981][SQL] Fix incorrect results of Casting Struct to String

maropu
authored and
cloud-fan
committed

Commits on Jan 10, 2018

[SPARK-23005][CORE] Improve RDD.take on small number of partitions

gengliangwang
authored and
cloud-fan
committed
[MINOR] fix a typo in BroadcastJoinSuite
cloud-fan
committed
[SPARK-23018][PYTHON] Fix createDataFrame from Pandas timestamp series assignment

BryanCutler
authored and
ueshin
committed
[SPARK-23009][PYTHON] Fix for non-str col names to createDataFrame from Pandas

BryanCutler
authored and
HyukjinKwon
committed
[SPARK-22982] Remove unsafe asynchronous close() call from FileDownloadChannel

JoshRosen
authored and
cloud-fan
committed
[SPARK-16060][SQL][FOLLOW-UP] add a wrapper solution for vectorized orc reader
cloud-fan
committed
[SPARK-22993][ML] Clarify HasCheckpointInterval param doc

sethah
authored and
Felix Cheung
committed
[SPARK-22997] Add additional defenses against use of freed MemoryBlocks
JoshRosen
committed
[SPARK-23019][CORE] Wait until SparkContext.stop() finished in SparkLauncherSuite

gengliangwang
authored and
Marcelo Vanzin
committed
[SPARK-22951][SQL] fix aggregation after dropDuplicates on empty data frames

Feng Liu
authored and
liancheng
committed

Commits on Jan 11, 2018

[SPARK-22587] Spark job fails if fs.defaultFS and application jar are different url

Mingjie Tang
authored and
jerryshao
committed
[SPARK-23001][SQL] Fix NullPointerException when DESC a database with NULL description

gatorsmile
authored and
cloud-fan
committed
[SPARK-20657][CORE] Speed up rendering of the stages page.

Marcelo Vanzin
authored and
cloud-fan
committed
[SPARK-22967][TESTS] Fix VersionSuite's unit tests by change Windows path into URI path

Ngone51
authored and
HyukjinKwon
committed
[SPARK-23000][TEST-HADOOP2.6] Fix Flaky test suite DataSourceWithHiveMetastoreCatalogSuite
gatorsmile
committed
[SPARK-19732][FOLLOW-UP] Document behavior changes made in na.fill and fillna

gatorsmile
authored and
HyukjinKwon
committed
Update PageRank.scala

ddna1021
authored and
srowen
committed
[SPARK-22994][K8S] Use a single image for all Spark containers.
Marcelo Vanzin
committed
[SPARK-22908] Add kafka source and sink for continuous processing.

jose-torres
authored and
tdas
committed
[SPARK-23046][ML][SPARKR] Have RFormula include VectorSizeHint in pipeline

MrBago
authored and
jkbradley
committed

Commits on Jan 12, 2018

[SPARK-23008][ML] OnehotEncoderEstimator python API

WeichenXu123
authored and
jkbradley
committed
[SPARK-22986][CORE] Use a cache to avoid instantiating multiple instances of broadcast variable values

ho3rexqj
authored and
cloud-fan
committed
[SPARK-23008][ML][FOLLOW-UP] mark OneHotEncoder python API deprecated

WeichenXu123
authored and
Nick Pentreath
committed
[SPARK-23025][SQL] Support Null type in scala reflection

mgaido91
authored and
gatorsmile
committed
Update rdd-programming-guide.md

Matthias Beaupère
authored and
srowen
committed
[SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT

gatorsmile
authored and
cloud-fan
committed
[MINOR][BUILD] Fix Java linter errors

dongjoon-hyun
authored and
sameeragarwal
committed
Merge branch 'master' into rk/merge-upstream
Robert Kruszewski
committed
[SPARK-22975][SS] MetricsReporter should not throw exception when there was no progress reported

mgaido91
authored and
zsxwing
committed
Resolve conflicts keeping our k8s code
Robert Kruszewski
committed
Revert "[SPARK-22908] Add kafka source and sink for continuous processing."
sameeragarwal
committed

Commits on Jan 13, 2018

[SPARK-22980][PYTHON][SQL] Clarify the length of each series is of each batch within scalar Pandas UDF
HyukjinKwon
committed
[SPARK-23043][BUILD] Upgrade json4s to 3.5.3

shimamoto
authored and
srowen
committed
[SPARK-22870][CORE] Dynamic allocation should allow 0 idle time

wangyum
authored and
srowen
committed
[SPARK-23036][SQL][TEST] Add withGlobalTempView for testing

xubo245
authored and
gatorsmile
committed
[SPARK-22790][SQL] add a configurable factor to describe HadoopFsRelation's size

CodingCat
authored and
gatorsmile
committed
[SPARK-21213][SQL][FOLLOWUP] Use compatible types for comparisons in compareAndGetNewStats

maropu
authored and
gatorsmile
committed

Commits on Jan 14, 2018

[SPARK-22959][PYTHON] Configuration to select the modules for daemon and worker in PySpark
HyukjinKwon
committed
[SPARK-23063][K8S] K8s changes for publishing scripts (and a couple of other misses)

foxish
authored and
Felix Cheung
committed
[SPARK-23038][TEST] Update docker/spark-test (JDK/OS)

dongjoon-hyun
authored and
Felix Cheung
committed
[SPARK-23069][DOCS][SPARKR] fix R doc for describe missing text

felixcheung
authored and
HyukjinKwon
committed
[SPARK-23021][SQL] AnalysisBarrier should override innerChildren to print correct explain output

maropu
authored and
gatorsmile
committed
[SPARK-23051][CORE] Fix for broken job description in Spark UI

smurakozi
authored and
srowen
committed
[SPARK-22999][SQL] show databases like command' can remove the like keyword

guoxiaolong
authored and
gatorsmile
committed

Commits on Jan 15, 2018

[SPARK-23054][SQL] Fix incorrect results of casting UserDefinedType to String

maropu
authored and
cloud-fan
committed
[SPARK-23049][SQL] `spark.sql.files.ignoreCorruptFiles` should work for ORC files

dongjoon-hyun
authored and
cloud-fan
committed
Consistent versions
Robert Kruszewski
committed
one bouncycastle
Robert Kruszewski
committed
versions
Robert Kruszewski
committed
[SPARK-23023][SQL] Cast field data to strings in showString

maropu
authored and
cloud-fan
committed
[SPARK-19550][BUILD][FOLLOW-UP] Remove MaxPermSize for sql module

wangyum
authored and
srowen
committed
[SPARK-23070] Bump previousSparkVersion in MimaBuild.scala to be 2.2.0
gatorsmile
committed
[SPARK-23035][SQL] Fix improper information of TempTableAlreadyExistsException

xubo245
authored and
gatorsmile
committed
Merge branch 'master' into rk/merge-upstream
Robert Kruszewski
committed

Commits on Jan 22, 2018

Fixes
Robert Kruszewski
committed

Commits on Jan 23, 2018

need to write as int96
Robert Kruszewski
committed
we're smaller?
Robert Kruszewski
committed
no mesos and hive thrift
Robert Kruszewski
committed

Commits on Jan 24, 2018

change test
Robert Kruszewski
committed
correct assert
Robert Kruszewski
committed

Commits on Jan 25, 2018

Update ParquetInteroperabilitySuite.scala
robert3005
authored