[SPARK-9487] Use the same num. worker threads in Scala/Python unit tests#15689

Closed

skanjila wants to merge 133 commits intoapache:masterfrom

skanjila:spark-9487

Commits on Oct 30, 2016

initial set of changes for local[4] in core

Saikat Kanjilal
authored and
Saikat Kanjilal
committed
[SPARK-18103][SQL] Rename *FileCatalog to *FileIndex

ericl
authored and
rxin
committed
added mllib changes to local[4]

Saikat Kanjilal
authored and
Saikat Kanjilal
committed
[SPARK-18106][SQL] ANALYZE TABLE should raise a ParseException for invalid option

dongjoon-hyun
authored and
hvanhovell
committed
[SPARK-17919] Make timeout to RBackend configurable in SparkR

falaki
authored and
Felix Cheung
committed
[SPARK-16137][SPARKR] randomForest for R

felixcheung
authored and
Felix Cheung
committed
[SPARK-18110][PYTHON][ML] add missing parameter in Python for RandomForest regression and classification

felixcheung
authored and
Felix Cheung
committed

Commits on Nov 1, 2016

[SPARK-18087][SQL] Optimize insert to not require REPAIR TABLE

ericl
authored and
rxin
committed
[SPARK-18167][SQL] Retry when the SQLQuerySuite test flakes

ericl
authored and
rxin
committed
[SPARK-18024][SQL] Introduce an internal commit protocol API
rxin
committed
[SPARK-18107][SQL] Insert overwrite statement runs much slower in spark-sql than it does in hive-client

viirya
authored and
rxin
committed
[MINOR][DOC] Remove spaces following slashs

dongjoon-hyun
authored and
srowen
committed
[SPARK-18111][SQL] Wrong ApproximatePercentile answer when multiple records have the minimum value

wangzhenhua
authored and
srowen
committed
[SPARK-15994][MESOS] Allow enabling Mesos fetch cache in coarse executor backend

drcrallen
authored and
srowen
committed
[SPARK-16881][MESOS] Migrate Mesos configs to use ConfigEntry

techaddict
authored and
srowen
committed
[SPARK-18114][MESOS] Fix mesos cluster scheduler generage command option error

Wang Lei
authored and
srowen
committed
[SPARK-17996][SQL] Fix unqualified catalog.getFunction(...)
hvanhovell
committed
[SPARK-16839][SQL] redundant aliases after cleanupAliases

eyal farago
authored and
hvanhovell
committed
Revert "[SPARK-16839][SQL] redundant aliases after cleanupAliases"
hvanhovell
committed
[SPARK-17848][ML] Move LabelCol datatype cast into Predictor.fit

zhengruifeng
authored and
jkbradley
committed
[SPARK-18103][FOLLOW-UP][SQL][MINOR] Rename `MetadataLogFileCatalog` to `MetadataLogFileIndex`

lw-lin
authored and
rxin
committed
[SPARK-18189][SQL] Fix serialization issue in KeyValueGroupedDataset

seyfe
authored and
rxin
committed
[SPARK-18148][SQL] Misleading Error Message for Aggregation Without Window/GroupBy

jiangxb1987
authored and
rxin
committed
[SPARK-18167] Disable flaky SQLQuerySuite test

ericl
authored and
rxin
committed
[SPARK-17764][SQL] Add `to_json` supporting to convert nested struct column to JSON string

HyukjinKwon
authored and
marmbrus
committed
[SPARK-17350][SQL] Disable default use of KryoSerializer in Thrift Server

JoshRosen
authored and
rxin
committed
[SPARK-18182] Expose ReplayListenerBus.read() overload which takes string iterator

JoshRosen
authored and
rxin
committed

Commits on Nov 2, 2016

[SPARK-18088][ML] Various ChiSqSelector cleanups
jkbradley
committed
[SPARK-18025] Use commit protocol API in structured streaming

rxin
authored and
marmbrus
committed
[SPARK-18216][SQL] Make Column.expr public
rxin
committed
[SPARK-17838][SPARKR] Check named arguments for options and use formatted R friendly message from JVM exception message

HyukjinKwon
authored and
Felix Cheung
committed
[SPARK-17992][SQL] Return all partitions from HiveShim when Hive throws a metastore exception when attempting to fetch partitions by filter

Michael Allman
authored and
rxin
committed
[SPARK-17475][STREAMING] Delete CRC files if the filesystem doesn't use checksum files

frreiss
authored and
rxin
committed
[SPARK-18183][SPARK-18184] Fix INSERT [INTO|OVERWRITE] TABLE ... PARTITION for Datasource tables

ericl
authored and
cloud-fan
committed
[SPARK-18192] Support all file formats in structured streaming
rxin
committed
[SPARK-18144][SQL] logging StreamingQueryListener$QueryStartedEvent

CodingCat
authored and
zsxwing
committed
[SPARK-17532] Add lock debugging info to thread dumps.

rdblue
authored and
rxin
committed
[MINOR] Use <= for clarity in Pi examples' Monte Carlo process

mrydzy
authored and
srowen
committed
[SPARK-18198][DOC][STREAMING] Highlight code snippets

lw-lin
authored and
srowen
committed
[SPARK-18204][WEBUI] Remove SparkUI.appUIAddress

jaceklaskowski
authored and
srowen
committed
[SPARK-18076][CORE][SQL] Fix default Locale used in DateFormat, NumberFormat to Locale.US
srowen
committed
[SPARK-16839][SQL] Simplify Struct creation code path

eyal farago
authored and
hvanhovell
committed
[SPARK-17683][SQL] Support ArrayType in Literal.apply

maropu
authored and
rxin
committed
[SPARK-17895] Improve doc for rangeBetween and rowsBetween

david-weiluo-ren
authored and
rxin
committed
[SPARK-14393][SQL] values generated by non-deterministic functions shouldn't change after coalesce or union

mengxr
authored and
rxin
committed
[SPARK-18160][CORE][YARN] spark.files & spark.jars should not be passed to driver in yarn mode

zjffdu
authored and
Marcelo Vanzin
committed
[SPARK-17058][BUILD] Add maven snapshots-and-staging profile to build/test against staging artifacts

steveloughran
authored and
rxin
committed
[SPARK-18214][SQL] Simplify RuntimeReplaceable type coercion
rxin
committed

Commits on Nov 3, 2016

[SPARK-17470][SQL] unify path for data source table and locationUri for hive serde table

cloud-fan
authored and
yhuai
committed
[SPARK-17963][SQL][DOCUMENTATION] Add examples (extend) in each expression and improve documentation

HyukjinKwon
authored and
gatorsmile
committed
[SPARK-18175][SQL] Improve the test case coverage of implicit type casting
gatorsmile
committed
[SPARK-18200][GRAPHX] Support zero as an initial capacity in OpenHashSet

dongjoon-hyun
authored and
rxin
committed
[SPARK-17122][SQL] support drop current database

adrian-wang
authored and
gatorsmile
committed
[SPARK-18219] Move commit protocol API (internal) from sql/core to core module
rxin
committed
[SQL] minor - internal doc improvement for InsertIntoTable.
rxin
committed
[SPARK-18177][ML][PYSPARK] Add missing 'subsamplingRate' of pyspark GBTClassifier

zhengruifeng
authored and
yanboliang
committed
[SPARK-17981][SPARK-17957][SQL] Fix Incorrect Nullability Setting to False in FilterExec

gatorsmile
authored and
hvanhovell
committed
[SPARK-17949][SQL] A JVM object based aggregate operator

liancheng
authored and
yhuai
committed
[SPARK-18244][SQL] Rename partitionProviderIsHive -> tracksPartitionsInCatalog
rxin
committed
[SPARK-18237][HIVE] hive.exec.stagingdir have no effect

福星
authored and
rxin
committed
[SPARK-18099][YARN] Fail if same files added to distributed cache for --files and --archives

kishorvpatil
authored and
Tom Graves
committed
added local[4] to repl,sparksql,streaming, all tests pass
Ubuntu
committed
[SPARK-18212][SS][KAFKA] increase executor poll timeout

koeninger
authored and
marmbrus
committed
[SPARKR][TEST] remove unnecessary suppressWarnings

wangmiao1981
authored and
Felix Cheung
committed
[SPARK-18257][SS] Improve error reporting for FileStressSuite
rxin
committed

Commits on Nov 4, 2016

[SPARK-18138][DOCS] Document that Java 7, Python 2.6, Scala 2.10, Hadoop < 2.6 are deprecated in Spark 2.1.0

srowen
authored and
rxin
committed
[SPARK-18259][SQL] Do not capture Throwable in QueryExecution

hvanhovell
authored and
rxin
committed
[SPARK-14393][SQL][DOC] update doc for python and R

felixcheung
authored and
Felix Cheung
committed
[SPARK-18200][GRAPHX][FOLLOW-UP] Support zero as an initial capacity in OpenHashSet

dongjoon-hyun
authored and
rxin
committed
Closing some stale/invalid pull requests
rxin
committed
[SPARK-18197][CORE] Optimise AppendOnlyMap implementation

a-roberts
authored and
rxin
committed
[SPARK-17337][SQL] Do not pushdown predicates through filters with predicate subqueries
hvanhovell
committed
[SPARK-18167] Re-enable the non-flaky parts of SQLQuerySuite

ericl
authored and
yhuai
committed

Commits on Nov 5, 2016

[SPARK-18256] Improve the performance of event log replay in HistoryServer

JoshRosen
authored and
yhuai
committed
[SPARK-18189] [SQL] [Followup] Move test from ReplSuite to prevent java.lang.ClassCircularityError
rxin
committed
[SPARK-17710][FOLLOW UP] Add comments to state why 'Utils.classForName' is not used

weiqingy
authored and
rxin
committed
[SPARK-18260] Make from_json null safe

brkyvz
authored and
rxin
committed
[SPARK-17183][SPARK-17983][SPARK-18101][SQL] put hive serde table schema to table properties like data source table

cloud-fan
authored and
rxin
committed
[SPARK-18287][SQL] Move hash expressions from misc.scala into hash.scala

rxin
authored and
hvanhovell
committed
[SPARK-18192][MINOR][FOLLOWUP] Missed json test in FileStreamSinkSuite

HyukjinKwon
authored and
hvanhovell
committed
[SPARK-17849][SQL] Fix NPE problem when using grouping sets

wangyang
authored and
hvanhovell
committed
[SPARK-17964][SPARKR] Enable SparkR with Mesos client mode and cluster mode

susanxhuynh
authored and
srowen
committed

Commits on Nov 6, 2016

[MINOR][DOCUMENTATION] Fix some minor descriptions in functions consistently with expressions

HyukjinKwon
authored and
Felix Cheung
committed
[SPARK-18276][ML] ML models should copy the training summary and set parent

sethah
authored and
yanboliang
committed
[SPARK-17854][SQL] rand/randn allows null/long as input seed

HyukjinKwon
authored and
srowen
committed
[SPARK-18210][ML] Pipeline.copy does not create an instance with the same UID

wojtek-szymanski
authored and
yanboliang
committed

Commits on Nov 7, 2016

Commits on Nov 8, 2016

[SPARK-18236] Reduce duplicate objects in Spark UI and HistoryServer
JoshRosen
committed
[SPARK-18295][SQL] Make to_json function null safe (matching it to from_json)

HyukjinKwon
authored and
marmbrus
committed
[SPARK-18086] Add support for Hive session vars.

rdblue
authored and
rxin
committed
[SPARK-18261][STRUCTURED STREAMING] Add statistics to MemorySink for joining

lw-lin
authored and
rxin
committed
[SPARK-18217][SQL] Disallow creating permanent views based on temporary views or UDFs

gatorsmile
authored and
rxin
committed
[SPARK-16575][CORE] partition calculation mismatch with sc.binaryFiles

fidato13
authored and
rxin
committed
[SPARK-18207][SQL] Fix a compilation error due to HashExpression.doGenCode

kiszk
authored and
hvanhovell
committed
[SPARK-18137][SQL] Fix RewriteDistinctAggregates UnresolvedException when a UDAF has a foldable TypeCheck

root
authored and
hvanhovell
committed
[SPARK-13770][DOCUMENTATION][ML] Document the ML feature Interaction

chie8842
authored and
srowen
committed
[MINOR][DOC] Unify example marks

zhengruifeng
authored and
srowen
committed
[SPARK-17868][SQL] Do not use bitmasks during parsing and analysis of CUBE/ROLLUP/GROUPING SETS

jiangxb1987
authored and
hvanhovell
committed
[SPARK-18346][SQL] TRUNCATE TABLE should fail if no partition is matched for the given non-partial partition spec
cloud-fan
committed
[SPARK-18191][CORE] Port RDD API to use commit protocol

jiangxb1987
authored and
rxin
committed
[SPARK-18357] Fix yarn files/archive broken issue andd unit tests

kishorvpatil
authored and
Tom Graves
committed
[SPARK-17748][ML] Minor cleanups to one-pass linear regression with elastic net
jkbradley
committed
[SPARK-18280][CORE] Fix potential deadlock in `StandaloneSchedulerBackend.dead`
zsxwing
committed
[SPARK-18342] Make rename failures fatal in HDFSBackedStateStore

brkyvz
authored and
tdas
committed

Commits on Nov 9, 2016

[SPARK-18239][SPARKR] Gradient Boosted Tree for R

felixcheung
authored and
Felix Cheung
committed
[SPARK-18333][SQL] Revert hacks in parquet and orc reader to support case insensitive resolution

ericl
authored and
cloud-fan
committed
[SPARK-18368] Fix regexp_replace with task serialization.

rdblue
authored and
rxin
committed
[SPARK-17659][SQL] Partitioned View is Not Supported By SHOW CREATE TABLE
gatorsmile
committed
[SPARK-18292][SQL] LogicalPlanToSQLSuite should not use resource dependent path for golden file generation

dongjoon-hyun
authored and
srowen
committed
[SPARK-18338][SQL][TEST-MAVEN] Fix test case initialization order under Maven builds

liancheng
authored and
yhuai
committed
[SPARK-16808][CORE] History Server main page does not honor APPLICATION_WEB_PROXY_BASE

vijoshi
authored and
Marcelo Vanzin
committed
Revert "[SPARK-18368] Fix regexp_replace with task serialization."
yhuai
committed
[SPARK-18368][SQL] Fix regexp replace when serialized

rdblue
authored and
yhuai
committed
added streaming, repl to unit test runs, changed to local[4]

Saikat Kanjilal
authored and
Saikat Kanjilal
committed
[SPARK-18370][SQL] Add table information to InsertIntoHadoopFsRelationCommand

hvanhovell
authored and
rxin
committed
[SPARK-18191][CORE][FOLLOWUP] Call `setConf` if `OutputFormat` is `Configurable`.

jiangxb1987
authored and
rxin
committed
[SPARK-17829][SQL] Stable format for offset log

Tyson Condie
authored and
zsxwing
committed

Commits on Nov 10, 2016

fixed merge conflicts

Saikat Kanjilal
authored and
Saikat Kanjilal
committed