Prototype: Data Source V2#9

Closed

cloud-fan wants to merge 149 commits intomasterfrom

data-source-v2-prototype

Commits on Jul 24, 2017

[SPARK-17528][SQL][FOLLOWUP] remove unnecessary data copy in object hash aggregate

cloud-fan
authored and
liancheng
committed
[SPARK-21502][MESOS] fix --supervise for mesos in cluster mode

skonto
authored and
Marcelo Vanzin
committed

Commits on Jul 25, 2017

[SPARK-21516][SQL][TEST] Overriding afterEach() in DatasetCacheSuite must call super.afterEach()

kiszk
authored and
cloud-fan
committed
[SPARK-20855][Docs][DStream] Update the Spark kinesis docs to use the KinesisInputDStream builder instead of deprecated KinesisUtils

yashs360
authored and
srowen
committed
[SPARK-21498][EXAMPLES] quick start -> one py demo have some bug in code

lizhaoch
authored and
srowen
committed
[SPARK-21175] Reject OpenBlocks when memory shortage on shuffle service.

jinxing
authored and
cloud-fan
committed
[SPARK-21383][YARN] Fix the YarnAllocator allocates more Resource

DjvuLee
authored and
Marcelo Vanzin
committed
[SPARK-21447][WEB UI] Spark history server fails to render compressed

Eric Vandenberg
authored and
Marcelo Vanzin
committed
[SPARK-21491][GRAPHX] Enhance GraphX performance: breakOut instead of .toMap

SereneAnt
authored and
srowen
committed

Commits on Jul 26, 2017

[SPARK-20586][SQL] Add deterministic to ScalaUDF
gatorsmile
committed
[SPARK-21494][NETWORK] Use correct app id when authenticating to external service.
Marcelo Vanzin
committed
[SPARK-21517][CORE] Avoid copying memory when transfer chunks remotely

caneMi
authored and
zsxwing
committed
[SPARK-21524][ML] unit test fix: ValidatorParamsSuiteHelpers generates wrong temp files

YY-OnCall
authored and
srowen
committed
[SPARK-20988][ML] Logistic regression uses aggregator hierarchy

sethah
authored and
Nick Pentreath
committed
[SPARK-21485][SQL][DOCS] Spark SQL documentation generation for built-in functions

HyukjinKwon
authored and
rxin
committed

Commits on Jul 27, 2017

[SPARK-21530] Update description of spark.shuffle.maxChunksBeingTransferred.

jinxing
authored and
cloud-fan
committed
[SPARK-21271][SQL] Ensure Unsafe.sizeInBytes is a multiple of 8

kiszk
authored and
cloud-fan
committed
[SPARK-21440][SQL][PYSPARK] Refactor ArrowConverters and add ArrayType and StructType support.

ueshin
authored and
cloud-fan
committed
[SPARK-19270][ML] Add summary table to GLM summary

actuaryzhang
authored and
yanboliang
committed
[SPARK-21319][SQL] Fix memory leak in sorter
cloud-fan
committed
[SPARK-21538][SQL] Attribute resolution inconsistency in the Dataset API

aokolnychyi
authored and
gatorsmile
committed

Commits on Jul 28, 2017

[SPARK-21306][ML] OneVsRest should support setWeightCol

facaiy
authored and
yanboliang
committed
[MINOR][BUILD] Fix current lint-java failures
srowen
committed
[SPARK-21553][SPARK SHELL] Add the description of the default value of master parameter in the spark-shell

davidxdh
authored and
srowen
committed
[SPARK-21541][YARN] Spark Logs show incorrect job status for a job that does not create SparkContext

pgandhi
authored and
Tom Graves
committed
Typo in comment

nahoj
authored and
srowen
committed

Commits on Jul 29, 2017

[SPARK-20090][PYTHON] Add StructType.fieldNames in PySpark

HyukjinKwon
authored and
holdenk
committed
[SPARK-21508][DOC] Fix example code provided in Spark Streaming Documentation

Remis Haroon
authored and
srowen
committed
[SPARK-21357][DSTREAMS] FileInputDStream not remove out of date RDD

shaofei007
authored and
srowen
committed
[SPARK-21555][SQL] RuntimeReplaceable should be compared semantically by its canonicalized child

viirya
authored and
gatorsmile
committed
[SPARK-19451][SQL] rangeBetween method should accept Long value as boundary

jiangxb1987
authored and
gatorsmile
committed
[SPARK-20962][SQL] Support subquery column aliases in FROM clause

maropu
authored and
gatorsmile
committed

Commits on Jul 30, 2017

[SQL] Fix typo in DataframeWriter doc

GuoChenzhao
authored and
HyukjinKwon
committed
[SPARK-21297][WEB-UI] Add count in 'JDBC/ODBC Server' page.

guoxiaolong
authored and
srowen
committed
[MINOR][DOC] Replace numTasks with numPartitions in programming guide

polarker
authored and
srowen
committed

Commits on Jul 31, 2017

[MINOR] Minor comment fixes in merge_spark_pr.py script
HyukjinKwon
committed
[SPARK-19839][CORE] release longArray in BytesToBytesMap

Zhan Zhang
authored and
gatorsmile
committed
[SPARK-21575][SPARKR] Eliminate needless synchronization in java-R serialization

SereneAnt
authored and
yanboliang
committed

Commits on Aug 1, 2017

[SPARK-18950][SQL] Report conflicting fields when merging two StructTypes

jiayue-zhang
authored and
gatorsmile
committed
[SPARK-21381][SPARKR] SparkR: pass on setHandleInvalid for classification algorithms

wangmiao1981
authored and
Felix Cheung
committed
[SPARK-21589][SQL][DOC] Add documents about Hive UDF/UDTF/UDAF

maropu
authored and
gatorsmile
committed
[SPARK-21475][CORE] Use NIO's Files API to replace FileInputStream/FileOutputStream in some critical paths

jerryshao
authored and
srowen
committed
[SPARK-21388][ML][PYSPARK] GBTs inherit from HasStepSize & LInearSVC from HasThreshold

zhengruifeng
authored and
yanboliang
committed
[SPARK-21585] Application Master marking application status as Failed for Client Mode

pgandhi
authored and
Tom Graves
committed
[SPARK-21522][CORE] Fix flakiness in LauncherServerSuite.
Marcelo Vanzin
committed
[SPARK-20079][YARN] Fix client AM not allocating executors after restart.
Marcelo Vanzin
committed
[SPARK-21592][BUILD] Skip maven-compiler-plugin main and test compilations in Maven build

gslowikowski
authored and
srowen
committed
[SPARK-21593][DOCS] Fix 2 rendering errors on configuration page
srowen
committed
[SPARK-21339][CORE] spark-shell --packages option does not add jars to classpath on windows

Devaraj K
authored and
Marcelo Vanzin
committed
[SPARK-12717][PYTHON] Adding thread-safe broadcast pickle registry

BryanCutler
authored and
HyukjinKwon
committed

Commits on Aug 2, 2017

Commits on Aug 3, 2017

[SPARK-21615][ML][MLLIB][DOCS] Fix broken redirect in collaborative filtering docs to databricks training repo

Ayush Singh
authored and
srowen
committed
[SPARK-21604][SQL] if the object extends Logging, i suggest to remove the var LOG which is useless.

zuotingbing
authored and
srowen
committed
[SPARK-21611][SQL] Error class name for log in several classes.

zuotingbing
authored and
srowen
committed
[SPARK-21605][BUILD] Let IntelliJ IDEA correctly detect Language level and Target byte code version

baibaichen
authored and
srowen
committed
[SPARK-21602][R] Add map_keys and map_values functions to R
HyukjinKwon
committed
[SPARK-21599][SQL] Collecting column statistics for datasource tables may fail with java.util.NoSuchElementException

dilipbiswal
authored and
gatorsmile
committed
[SPARK-20713][SPARK CORE] Convert CommitDenied to TaskKilled.

louis lyu
authored and
Tom Graves
committed
Fix Java SimpleApp spark application

christiam
authored and
srowen
committed

Commits on Aug 4, 2017

Commits on Aug 5, 2017

[SPARK-21374][CORE] Fix reading globbed paths from S3 into DF with disabled FS cache

zsxwing
authored and
gatorsmile
committed
[SPARK-21580][SQL] Integers in aggregation expressions are wrongly taken as group-by ordinal

10110346
authored and
gatorsmile
committed
[INFRA] Close stale PRs
HyukjinKwon
committed
[SPARK-21485][FOLLOWUP][SQL][DOCS] Describes examples and arguments separately, and note/since in SQL built-in function documentation

HyukjinKwon
authored and
gatorsmile
committed
[SPARK-21640] Add errorifexists as a valid string for ErrorIfExists save mode

arodriguez
authored and
gatorsmile
committed

Commits on Aug 6, 2017

[SPARK-21637][SPARK-21451][SQL] get `spark.hadoop.*` properties from sysProps to hiveconf

yaooqinn
authored and
gatorsmile
committed
[SPARK-20963][SQL] Support column aliases for join relations in FROM clause

maropu
authored and
gatorsmile
committed
[SPARK-21588][SQL] SQLContext.getConf(key, null) should return null

vinodkc
authored and
gatorsmile
committed
[SPARKR][BUILD] AppVeyor change to latest R version

felixcheung
authored and
HyukjinKwon
committed
[SPARK-21574][SQL] Point out user to set hive config before SparkSession is initialized

wangyum
authored and
gatorsmile
committed
[SPARK-20963][SQL][FOLLOW-UP] Use UnresolvedSubqueryColumnAliases for visitTableName

maropu
authored and
gatorsmile
committed
[SPARK-21622][ML][SPARKR] Support offset in SparkR GLM

actuaryzhang
authored and
Felix Cheung
committed
Add "full_outer" name to join types

BartekH
authored and
gatorsmile
committed
[MINOR][BUILD] Remove duplicate test-jar:test spark-sql dependency from Hive module

srowen
authored and
gatorsmile
committed

Commits on Aug 7, 2017

[SPARK-21621][CORE] Reset numRecordsWritten after DiskBlockObjectWriter.commitAndGet called

ConeyLiu
authored and
cloud-fan
committed
[SPARK-13041][MESOS] Adds sandbox uri to spark dispatcher ui

skonto
authored and
srowen
committed
[SPARK-21623][ML] fix RF doc

Peng Meng
authored and
srowen
committed
[SPARK-21544][DEPLOY][TEST-MAVEN] Tests jar of some module should not upload twice

caneGuy
authored and
srowen
committed
[SPARK-21647][SQL] Fix SortMergeJoin when using CROSS

gatorsmile
authored and
cloud-fan
committed
[SPARK][DOCS] Added note on meaning of position to substring function

maclockard
authored and
srowen
committed
[SPARK-21565][SS] Propagate metadata in attribute replacement.

Jose Torres
authored and
zsxwing
committed
[SPARK-21648][SQL] Fix confusing assert failure in JDBC source when parallel fetching parameters are not properly provided.
gatorsmile
committed

Commits on Aug 8, 2017

[SPARK-21542][ML][PYTHON] Python persistence helper functions

ajaysaini725
authored and
jkbradley
committed
[SPARK-19270][FOLLOW-UP][ML] PySpark GLR model.summary should return a printable representation.
yanboliang
committed
[SPARK-21640][FOLLOW-UP][SQL] added errorifexists on IllegalArgumentException message

Marcos P. Sanchez
authored and
gatorsmile
committed
[SPARK-21567][SQL] Dataset should work with type alias

viirya
authored and
cloud-fan
committed
[MINOR][R][BUILD] More reliable detection of R version for Windows in AppVeyor
HyukjinKwon
committed
[SPARK-20655][CORE] In-memory KVStore implementation.
Marcelo Vanzin
committed
[SPARK-21671][CORE] Move kvstore to "util" sub-package, add private annotation.
Marcelo Vanzin
committed

Commits on Aug 9, 2017

[SPARK-20433][BUILD] Bump jackson from 2.6.5 to 2.6.7.1

srowen
authored and
Marcelo Vanzin
committed
[SPARK-21596][SS] Ensure places calling HDFSMetadataLog.get check the return value

zsxwing
authored and
tdas
committed
[SPARK-21608][SPARK-9221][SQL] Window rangeBetween() API should allow literal boundary

jiangxb1987
authored and
cloud-fan
committed
[SPARK-21503][UI] Spark UI shows incorrect task status for a killed Executor Process

pgandhi
authored and
cloud-fan
committed
[SPARK-21176][WEB UI] Use a single ProxyServlet to proxy all workers and applications

aosagie
authored and
cloud-fan
committed
[SPARK-21523][ML] update breeze to 0.13.2 for an emergency bugfix in strong wolfe line search

WeichenXu123
authored and
yanboliang
committed
[SPARK-21663][TESTS] test("remote fetch below max RPC message size") should call masterTracker.stop() in MapOutputTrackerSuite

wangjiaochun
authored and
cloud-fan
committed
[SPARK-21665][CORE] Need to close resources after use

vinodkc
authored and
srowen
committed
[SPARK-21276][CORE] Update lz4-java to the latest (v1.4.0)

maropu
authored and
srowen
committed
[SPARK-21504][SQL] Add spark version info into table metadata
gatorsmile
committed
[SPARK-21587][SS] Added filter pushdown through watermarks.

Jose Torres
authored and
tdas
committed
[SPARK-21551][PYTHON] Increase timeout for PythonRDD.serveIterator

peay
authored and
rxin
committed

Commits on Aug 10, 2017

[SPARK-14932][SQL] Allow DataFrame.replace() to replace values with None

jiayue-zhang
authored and
gatorsmile
committed
[SPARK-21669] Internal API for collecting metrics/stats during FileFormatWriter jobs

adrian-ionescu
authored and
rxin
committed
[SPARK-21638][ML] Fix RF/GBT Warning message error

Peng Meng
authored and
srowen
committed

Commits on Aug 11, 2017

[SPARK-21699][SQL] Remove unused getTableOption in ExternalCatalog
rxin
committed
[SPARK-21675][WEBUI] Add a navigation bar at the bottom of the Details for Stage Page

yaooqinn
authored and
srowen
committed
[SPARK-21519][SQL] Add an option to the JDBC data source to initialize the target DB environment

LucaCanali
authored and
gatorsmile
committed
[SPARK-21595] Separate thresholds for buffering and spilling in ExternalAppendOnlyUnsafeRowArray

tejasapatil
authored and
hvanhovell
committed
[SPARK-19122][SQL] Unnecessary shuffle+sort added if join predicates ordering differ from bucketing and sorting order

tejasapatil
authored and
gatorsmile
committed
[SPARK-12559][SPARK SUBMIT] fix --packages for stand-alone cluster mode

skonto
authored and
Marcelo Vanzin
committed

Commits on Aug 12, 2017

[MINOR][BUILD] Download RAT and R version info over HTTPS; use RAT 0.12

srowen
authored and
HyukjinKwon
committed
[SPARK-17025][ML][PYTHON] Persistence for Pipelines with Python-only Stages

ajaysaini725
authored and
jkbradley
committed
[SPARK-21709][BUILD] sbt 0.13.16 and some plugin updates

pj.fanning
authored and
srowen
committed

Commits on Aug 14, 2017

[MINOR][SQL] Additional test case for CheckCartesianProducts rule

aokolnychyi
authored and
gatorsmile
committed
[SPARK-21176][WEB UI] Format worker page links to work with proxy

aosagie
authored and
srowen
committed
[SPARK-21563][CORE] Fix race condition when serializing TaskDescriptions and adding jars

ash211
authored and
cloud-fan
committed
[SPARK-21658][SQL][PYSPARK] Add default None for value in na.replace in PySpark

chihhanyu
authored and
HyukjinKwon
committed
[MINOR][SQL][TEST] no uncache table in joinsuite test

heary-cao
authored and
gatorsmile
committed
[SPARK-19471][SQL] AggregationIterator does not initialize the generated result projection before using it

DonnyZone
authored and
gatorsmile
committed
[SPARK-21696][SS] Fix a potential issue that may generate partial snapshot files

zsxwing
authored and
tdas
committed

Commits on Aug 15, 2017

[SPARK-21721][SQL] Clear FileSystem deleteOnExit cache when paths are successfully removed

viirya
authored and
gatorsmile
committed
[SPARK-21724][SQL][DOC] Adds since information in the documentation of date functions

HyukjinKwon
authored and
gatorsmile
committed
[SPARK-21732][SQL] Lazily init hive metastore client

zsxwing
authored and
gatorsmile
committed
[SPARK-19471][SQL] AggregationIterator does not initialize the generated result projection before using it

DonnyZone
authored and
gatorsmile
committed
[SPARK-18464][SQL][FOLLOWUP] support old table which doesn't store schema in table properties

cloud-fan
authored and
gatorsmile
committed
[SPARK-17742][CORE] Handle child process exit in SparkLauncher.
Marcelo Vanzin
committed
[SPARK-21731][BUILD] Upgrade scalastyle to 0.9.
Marcelo Vanzin
committed
[MINOR] Fix a typo in the method name `UserDefinedFunction.asNonNullabe`

jiangxb1987
authored and
gatorsmile
committed

Commits on Aug 16, 2017

[SPARK-21712][PYSPARK] Clarify type error for Column.substr()

nchammas
authored and
HyukjinKwon
committed
[SPARK-19634][ML] Multivariate summarizer - dataframes API

WeichenXu123
authored and
yanboliang
committed
[SPARK-21422][BUILD] Depend on Apache ORC 1.4.0

dongjoon-hyun
authored and
gatorsmile
committed
[SPARK-21723][ML] Fix writing LibSVM (key not found: numFeatures)

Jan Vrsovsky
authored and
srowen
committed
[SPARK-13969][ML] Add FeatureHasher transformer
Nick Pentreath
committed
[SPARK-21656][CORE] spark dynamic allocation should not idle timeout executors when tasks still to run

John Lee
authored and
Tom Graves
committed
[SPARK-21603][SQL] The wholestage codegen will be much slower then that is closed when the function is too long

eatoncys
authored and
gatorsmile
committed
[SPARK-21738] Thriftserver doesn't cancel jobs when session is closed

mgaido91
authored and
gatorsmile
committed
[SPARK-21680][ML][MLLIB] optimize Vector compress

Peng Meng
authored and
srowen
committed

Commits on Aug 17, 2017

[SPARK-3151][BLOCK MANAGER] DiskStore.getBytes fails for files larger than 2GB

Eyal Farago
authored and
cloud-fan
committed
[SPARK-21743][SQL] top-most limit should not cause memory leak

cloud-fan
authored and
gatorsmile
committed
working prototype
cloud-fan
committed