MapGroupsWithState, and FlatMapGroupsWithState#34

Closed

tdas wants to merge 124 commits intomasterfrom

Commits on Jan 17, 2017

[SPARK-18905][STREAMING] Fix the issue of removing a failed jobset from JobScheduler.jobSets

CodingCat
authored and
zsxwing
committed
[SPARK-19148][SQL] do not expose the external table concept in Catalog
cloud-fan
committed
[MINOR][YARN] Move YarnSchedulerBackendSuite to resource-managers/yarn directory.
yanboliang
committed
[SPARK-19240][SQL][TEST] add test for setting location for managed table
cloud-fan
committed
[SPARK-19219][SQL] Fix Parquet log output defaults

nicklavers
authored and
srowen
committed
[SPARK-3249][DOC] Fix links in ScalaDoc that cause warning messages in `sbt/sbt unidoc`

HyukjinKwon
authored and
srowen
committed
[SPARK-19179][YARN] Change spark.yarn.access.namenodes config and update docs

jerryshao
authored and
Tom Graves
committed
[SPARK-19019] [PYTHON] Fix hijacked `collections.namedtuple` and port cloudpickle changes for PySpark to work with Python 3.6.0

HyukjinKwon
authored and
davies
committed
[SPARK-19065][SQL] Don't inherit expression id in dropDuplicates

zsxwing
authored and
cloud-fan
committed
[SPARK-19129][SQL] SessionCatalog: Disallow empty part col values in partition spec

gatorsmile
authored and
cloud-fan
committed
[SPARK-19239][PYSPARK] Check parameters whether equals None when specify the column in jdbc API

DjvuLee
authored and
gatorsmile
committed
[MINOR][SQL] Remove duplicate call of reset() function in CurrentOrigin.withOrigin()

jiangxb1987
authored and
rxin
committed
[SPARK-18917][SQL] Remove schema check in appending data
rxin
committed
[SPARK-13721][SQL] Support outer generators in DataFrame API

Bogdan Raducanu
authored and
hvanhovell
committed
[SPARK-18206][ML] Add instrumentation for MLP,NB,LDA,AFT,GLM,Isotonic,LiR

zhengruifeng
authored and
jkbradley
committed

Commits on Jan 18, 2017

Prototype - almost working
tdas
committed
Renamed to mapGroupsWithState
tdas
committed
Fixed bugs
tdas
committed
[SPARK-18243][SQL] Port Hive writing to use FileFormat interface

cloud-fan
authored and
gatorsmile
committed
[SPARK-19227][SPARK-19251] remove unused imports and outdated comments

uncleGen
authored and
srowen
committed
[SPARK-18782][BUILD] Bump Hadoop 2.6 version to use Hadoop 2.6.5

a-roberts
authored and
srowen
committed
Removed prints
tdas
committed
Test state remove
tdas
committed
[SPARK-19024][SQL] Implement new approach to write a permanent view

jiangxb1987
authored and
cloud-fan
committed
[SPARK-19223][SQL][PYSPARK] Fix InputFileBlockHolder for datasources which are based on HadoopRDD or NewHadoopRDD

viirya
authored and
cloud-fan
committed
[SPARK-19231][SPARKR] add error handling for download and untar for Spark release

felixcheung
authored and
Felix Cheung
committed
[SPARK-18113] Use ask to replace askWithRetry in canCommit and make receiver idempotent.

jinxing
authored and
Marcelo Vanzin
committed
[SPARK-19113][SS][TESTS] Ignore StreamingQueryException thrown from awaitInitialization to avoid breaking tests
zsxwing
committed
[SPARK-19168][STRUCTURED STREAMING] StateStore should be aborted upon error

lw-lin
authored and
zsxwing
committed
[SPARK-19182][DSTREAM] Optimize the lock in StreamingJobProgressListener to not block UI when generating Streaming jobs

uncleGen
authored and
zsxwing
committed
Test restart, and test with metrics
tdas
committed
[SPARK-14975][ML] Fixed GBTClassifier to predict probability per training instance and fixed interfaces

imatiach-msft
authored and
jkbradley
committed

Commits on Jan 19, 2017

Update known_translations for contributor names
yhuai
committed
[SPARK-19265][SQL] make table relation cache general and does not depend on hive

cloud-fan
authored and
gatorsmile
committed
[SPARK-14272][ML] Add Loglikelihood in GaussianMixtureSummary

zhengruifeng
authored and
yanboliang
committed
[SPARK-19059][SQL] Unable to retrieve data from parquet table whose name startswith underscore

jayadevanmurali
authored and
cloud-fan
committed
[SPARK-16654][CORE] Add UI coverage for Application Level Blacklisting

José Hiram Soltren
authored and
squito
committed
[SPARK-19295][SQL] IsolatedClientLoader's downloadVersion should log the location of downloaded metastore client jars
yhuai
committed
[SPARK-17912] [SQL] Refactor code generation to get data for ColumnVector/ColumnarBatch

kiszk
authored and
davies
committed

Commits on Jan 20, 2017

[SPARK-19292][SQL] filter with partition columns should be case-insensitive on Hive tables

cloud-fan
authored and
gatorsmile
committed
[SPARK-19271][SQL] Change non-cbo estimation of aggregate

wangzhenhua
authored and
gatorsmile
committed
[SPARK-19302][DOC][MINOR] Fix the wrong item format in security.md

sarutak
authored and
srowen
committed
[SPARK-19069][CORE] Expose task 'status' and 'duration' in spark history server REST API.

Parag Chaudhari
authored and
squito
committed
[SPARK-19314][SS][CATALYST] Do not allow sort before aggregation in Structured Streaming plan
tdas
committed

Commits on Jan 21, 2017

[SPARK-18589][SQL] Fix Python UDF accessing attributes from both side of join

Davies Liu
authored and
hvanhovell
committed
[SPARK-19267][SS] Fix a race condition when stopping StateStore

zsxwing
authored and
tdas
committed
Fixed everything
tdas
committed
[SPARK-16101][SQL] Refactoring CSV write path to be consistent with JSON data source

HyukjinKwon
authored and
cloud-fan
committed
[SPARK-14536][SQL] fix to handle null value in array type column for postgres.

sureshthalamati
authored and
gatorsmile
committed
[SPARK-19305][SQL] partitioned table should always put partition columns at the end of table schema
cloud-fan
committed
[SPARK-17724][STREAMING][WEBUI] Unevaluated new lines in tooltip in DAG Visualization of a job

keypointt
authored and
srowen
committed
[SPARK-19117][SPARK-18922][TESTS] Fix the rest of flaky, newly introduced and missed test failures on Windows

HyukjinKwon
authored and
srowen
committed

Commits on Jan 22, 2017

Commits on Jan 23, 2017

[SPARK-16101][SQL] Refactoring CSV read path to be consistent with JSON data source

HyukjinKwon
authored and
cloud-fan
committed
[SPARK-19229][SQL] Disallow Creating Hive Source Tables when Hive Support is Not Enabled
gatorsmile
committed
[SPARK-19309][SQL] disable common subexpression elimination for conditional expressions
cloud-fan
committed
[SPARK-19155][ML] Make family case insensitive in GLM

actuaryzhang
authored and
yanboliang
committed
[SPARK-19218][SQL] Fix SET command to show a result correctly and in a sorted order

dongjoon-hyun
authored and
gatorsmile
committed
[SPARK-19146][CORE] Drop more elements when stageData.taskData.size > retainedTasks

wangyum
authored and
srowen
committed
[SPARK-19284][SQL] append to partitioned datasource table should without custom partition location

windpiger
authored and
cloud-fan
committed
[SPARK-14709][ML] spark.ml API for linear SVM

YY-OnCall
authored and
jkbradley
committed
[SPARK-17455][MLLIB] Improve PAVA implementation in IsotonicRegression

neggert
authored and
jkbradley
committed
[SPARK-16473][MLLIB] Fix BisectingKMeans Algorithm failing in edge case

imatiach-msft
authored and
jkbradley
committed
[SPARK-19306][CORE] Fix inconsistent state in DiskBlockObject when expection occurred

jerryshao
authored and
Marcelo Vanzin
committed

Commits on Jan 24, 2017

[SPARK-17747][ML] WeightCol support non-double numeric datatypes

zhengruifeng
authored and
jkbradley
committed
[SPARK-19290][SQL] add a new extending interface in Analyzer for post-hoc resolution

cloud-fan
authored and
gatorsmile
committed
[SPARK-19272][SQL] Remove the param `viewOriginalText` from `CatalogTable`

jiangxb1987
authored and
cloud-fan
committed
[SPARK-9435][SQL] Reuse function in Java UDF to correctly support expressions that require equality comparison between ScalaUDF

HyukjinKwon
authored and
gatorsmile
committed
[SPARK-19268][SS] Disallow adaptive query execution for streaming queries
zsxwing
committed
[SPARK-16101][HOTFIX] Fix the build with Scala 2.10 by explicit typed argument

HyukjinKwon
authored and
tdas
committed
[SPARK-18823][SPARKR] add support for assigning to column

felixcheung
authored and
Felix Cheung
committed
[DOCS] Fix typo in docs

uncleGen
authored and
srowen
committed
delete useless var “j”

xiaoyesoso
authored and
srowen
committed
[SPARK-19152][SQL] DataFrameWriter.saveAsTable support hive append

windpiger
authored and
cloud-fan
committed
[SPARK-19246][SQL] CataLogTable's partitionSchema order and exist check

windpiger
authored and
cloud-fan
committed
[SPARK-14049][CORE] Add functionality in spark history sever API to query applications by end time

Parag Chaudhari
authored and
squito
committed
[SPARK-17913][SQL] compare atomic and string type column may return confusing result

cloud-fan
authored and
gatorsmile
committed
[SPARK-18036][ML][MLLIB] Fixing decision trees handling edge cases

imatiach-msft
authored and
jkbradley
committed
[SPARK-19139][CORE] New auth mechanism for transport library.

Marcelo Vanzin
authored and
zsxwing
committed
[SPARK-19017][SQL] NOT IN subquery with more than one column may return incorrect results

nsyca
authored and
hvanhovell
committed
[SPARK-19334][SQL] Fix the code injection vulnerability related to Generator functions.

sarutak
authored and
hvanhovell
committed

Commits on Jan 25, 2017

[SPARK-19330][DSTREAMS] Also show tooltip for successful batches

lw-lin
authored and
zsxwing
committed
[SPARK-16046][DOCS] Aggregations in the Spark SQL programming guide

aokolnychyi
authored and
gatorsmile
committed
[SPARK-18750][YARN] Avoid using "mapValues" when allocating containers.

Marcelo Vanzin
authored and
Tom Graves
committed
[SPARK-19313][ML][MLLIB] GaussianMixture should limit the number of features

sethah
authored and
yanboliang
committed
[SPARK-18863][SQL] Output non-aggregate expressions without GROUP BY in a subquery does not yield an error

nsyca
authored and
hvanhovell
committed
[SPARK-19311][SQL] fix UDT hierarchy issue

gmoehler
authored and
gatorsmile
committed
[SPARK-19307][PYSPARK] Make sure user conf is propagated to SparkContext.
Marcelo Vanzin
committed
[SPARK-19064][PYSPARK] Fix pip installing of sub components
holdenk
committed

Commits on Jan 26, 2017

[SPARK-14804][SPARK][GRAPHX] Fix checkpointing of VertexRDD/EdgeRDD
tdas
committed
[SPARK-18495][UI] Document meaning of green dot in DAG visualization

uncleGen
authored and
rxin
committed
[SPARK-18020][STREAMING][KINESIS] Checkpoint SHARD_END to finish reading closed shards

maropu
authored and
brkyvz
committed
[TESTS][SQL] Setup testdata at the beginning for tests to run independently

dilipbiswal
authored and
gatorsmile
committed
[SPARK-19220][UI] Make redirection to HTTPS apply to all URIs.

Marcelo Vanzin
authored and
sarutak
committed
[CORE][DOCS] Update a help message for --files in spark-submit

maropu
authored and
srowen
committed
[SPARK-18936][SQL] Infrastructure for session local timezone support.

ueshin
authored and
hvanhovell
committed
[SPARK-19338][SQL] Add UDF names in explain

maropu
authored and
gatorsmile
committed

Commits on Jan 27, 2017

[SPARK-18218][ML][MLLIB] Reduce shuffled data size of BlockMatrix multiplication and solve potential OOM and low parallelism usage problem By split middle dimension in matrix multiplication

WeichenXu123
authored and
brkyvz
committed
[SPARK-18821][SPARKR] Bisecting k-means wrapper in SparkR

wangmiao1981
authored and
Felix Cheung
committed
[SPARK-18788][SPARKR] Add API for getNumPartitions

felixcheung
authored and
Felix Cheung
committed
[SPARK-18929][ML] Add Tweedie distribution in GLM

actuaryzhang
authored and
yanboliang
committed
[SPARK-12970][DOCS] Fix the example in SturctType APIs for Scala and Java

HyukjinKwon
authored and
srowen
committed
[SPARK-19333][SPARKR] Add Apache License headers to R files

felixcheung
authored and
Felix Cheung
committed
[SPARK-19324][SPARKR] Spark VJM stdout output is getting dropped in SparkR

felixcheung
authored and
shivaram
committed
[SPARK-19365][CORE] Optimize RequestMessage serialization
zsxwing
committed

Commits on Jan 29, 2017

[SPARK-19152][SQL][FOLLOWUP] simplify CreateHiveTableAsSelectCommand

cloud-fan
authored and
gatorsmile
committed
[SPARK-18872][SQL][TESTS] New test cases for EXISTS subquery

dilipbiswal
authored and
gatorsmile
committed
[SPARK-19385][SQL] During canonicalization, `NOT(...(l, r))` should not expect such cases that l.hashcode > r.hashcode

lw-lin
authored and
gatorsmile
committed

Commits on Jan 30, 2017

Commits on Jan 31, 2017