Sr/new master#414

Merged

robert3005 merged 218 commits intomasterfrom

Oct 5, 2018

Commits on Sep 6, 2018

[SPARK-25313][SQL] Fix regression in FileFormatWriter output names

gengliangwang
authored and
cloud-fan
committed
[SPARK-25337][SQL][TEST] runSparkSubmit` should provide non-testing mode

dongjoon-hyun
authored and
srowen
committed
[SPARK-25252][SQL] Support arrays of any types by to_json

MaxGekk
authored and
HyukjinKwon
committed
[SPARK-25317][CORE] Avoid perf regression in Murmur3 Hash on UTF8String

mgaido91
authored and
cloud-fan
committed
[SPARK-25108][SQL] Fix the show method to display the wide character alignment problem

xuejianbest
authored and
srowen
committed
[SPARK-25328][PYTHON] Add an example for having two columns as the grouping key in group aggregate pandas UDF

HyukjinKwon
authored and
BryanCutler
committed
[SPARK-25268][GRAPHX] run Parallel Personalized PageRank throws serialization Exception

shahidki31
authored and
jkbradley
committed
[SPARK-25072][PYSPARK] Forbid extra value for custom Row

authored and
BryanCutler
committed
[SPARK-25222][K8S] Improve container status logging

rvesse
authored and
mccheah
committed
[SPARK-25262][K8S] Allow SPARK_LOCAL_DIRS to be tmpfs backed on K8S

rvesse
authored and
mccheah
committed

Commits on Sep 7, 2018

[SPARK-25208][SQL][FOLLOW-UP] Reduce code size.

ueshin
authored and
cloud-fan
committed
[SPARK-25330][BUILD][BRANCH-2.3] Revert Hadoop 2.7 to 2.7.3

wangyum
authored and
srowen
committed
[SPARK-22357][CORE][FOLLOWUP] SparkContext.binaryFiles ignore minPartitions parameter
srowen
committed
[SPARK-25237][SQL] Remove updateBytesReadWithFileSize in FileScanRDD

authored and
srowen
committed
[SPARK-25267][SQL][TEST] Disable ConvertToLocalRelation in the test cases of sql/core and sql/hive

dilipbiswal
authored and
gatorsmile
committed
[SPARK-12321][SQL][FOLLOW-UP] Add tests for fromString
gatorsmile
committed
[SPARK-21786][SQL][FOLLOWUP] Add compressionCodec test for CTAS

fjh100456
authored and
dongjoon-hyun
committed
[SPARK-25270] lint-python: Add flake8 to find syntax errors and undefined names

cclauss
authored and
holdenk
committed
[MINOR][SS] Fix kafka-0-10-sql trivials

dongjinleekr
authored and
srowen
committed
[SPARK-23429][CORE] Add executor memory metrics to heartbeat and expose in executors REST API

edwinalu
authored and
mccheah
committed

Commits on Sep 9, 2018

Commits on Sep 10, 2018

Commits on Sep 11, 2018

[SPARK-25371][SQL] struct() should allow being called with 0 args

mgaido91
authored and
cloud-fan
committed
[SPARK-25278][SQL][FOLLOWUP] remove the hack in ProgressReporter
cloud-fan
committed
[SPARK-17916][SPARK-25241][SQL][FOLLOW-UP] Fix empty string being parsed as null when nullValue is set.

authored and
HyukjinKwon
committed
[SPARK-25389][SQL] INSERT OVERWRITE DIRECTORY STORED AS should prevent duplicate fields
dongjoon-hyun
committed
[SPARK-25221][DEPLOY] Consistent trailing whitespace treatment of conf values

gerashegalov
authored and
Marcelo Vanzin
committed
[SPARK-24889][CORE] Update block info when unpersist rdds

viirya
authored and
Marcelo Vanzin
committed
[INFRA] Close stale PRs.
Marcelo Vanzin
committed
[SPARK-25398] Minor bugs from comparing unrelated types
srowen
committed
Revert "[SPARK-23820][CORE] Enable use of long form of callsite in logs"
srowen
committed
[SPARK-25399][SS] Continuous processing state should not affect microbatch execution jobs

mukulmurthy
authored and
tdas
committed

Commits on Sep 12, 2018

[SPARK-25402][SQL] Null handling in BooleanSimplification

gatorsmile
authored and
cloud-fan
committed
[SPARK-25352][SQL] Perform ordered global limit when limit number is bigger than topKSortFallbackThreshold

viirya
authored and
cloud-fan
committed
[SPARK-25363][SQL] Fix schema pruning in where clause by ignoring unnecessary root fields

viirya
authored and
dbtsai
committed

Commits on Sep 13, 2018

[SPARK-23820][CORE] Enable use of long form of callsite in logs

michaelmior
authored and
cloud-fan
committed
[SPARK-25387][SQL] Fix for NPE caused by bad CSV input

MaxGekk
authored and
cloud-fan
committed
[SPARK-25357][SQL] Add metadata to SparkPlanInfo to dump more information like file path to event log

LantaoJin
authored and
cloud-fan
committed
[SPARK-25238][PYTHON] lint-python: Fix W605 warnings for pycodestyle 2.4

srowen
authored and
HyukjinKwon
committed
[SPARK-25415][SQL] Make plan change log in RuleExecutor configurable by SQLConf

maryannxue
authored and
gatorsmile
committed
[SPARK-25295][K8S] Fix executor names collision

Stavros Kontopoulos
authored and
liyinan926
committed
[SPARK-25352][SQL][FOLLOWUP] Add helper method and address style issue

viirya
authored and
hvanhovell
committed
[SPARK-25170][DOC] Add list and short description of Spark Executor Task Metrics to the documentation.

LucaCanali
authored and
srowen
committed
[SPARK-25406][SQL] For ParquetSchemaPruningSuite.scala, move calls to `withSQLConf` inside calls to `test`

mallman
authored and
dbtsai
committed
[SPARK-25338][TEST] Ensure to call super.beforeAll() and super.afterAll() in test cases

kiszk
authored and
dongjoon-hyun
committed
[SPARK-25400][CORE][TEST] Increase test timeouts

squito
authored and
srowen
committed

Commits on Sep 14, 2018

Commits on Sep 15, 2018

Commits on Sep 16, 2018

[SPARK-25425][SQL] Extra options should override session options in DataSource V2

MaxGekk
and
dongjoon-hyun
committed
[SPARK-25438][SQL][TEST] Fix FilterPushdownBenchmark to use the same memory assumption
dongjoon-hyun
committed
[SPARK-25439][TESTS][SQL] Fixes TPCHQuerySuite datatype of customer.c_nationkey to BIGINT according to spec

npoggi
authored and
gatorsmile
committed
[SPARK-24418][FOLLOWUP][DOC] Update docs to show Scala 2.11.12

dongjoon-hyun
authored and
dbtsai
committed
[MINOR][DOCS] Axe deprecated doc refs

Michael Chirico
authored and
Felix Cheung
committed

Commits on Sep 17, 2018

Commits on Sep 18, 2018

Commits on Sep 19, 2018

Commits on Sep 20, 2018

[SPARK-25471][PYTHON][TEST] Fix pyspark-sql test error when using Python 3.6 and Pandas 0.23

BryanCutler
authored and
HyukjinKwon
committed
[SPARK-24157][SS][FOLLOWUP] Rename to spark.sql.streaming.noDataMicroBatches.enabled
rxin
committed
[SPARK-24341][FOLLOWUP][DOCS] Add migration note for IN subqueries behavior

mgaido91
authored and
cloud-fan
committed
[SPARK-25457][SQL] IntegralDivide returns data type of the operands

mgaido91
authored and
cloud-fan
committed
[SPARK-4502][SQL] Rename to spark.sql.optimizer.nestedSchemaPruning.enabled

rxin
authored and
gatorsmile
committed
[SPARK-23648][R][SQL] Adds more types for hint in SparkR

huaxingao
authored and
Felix Cheung
committed
[SPARK-25339][TEST] Refactor FilterPushdownBenchmark

wangyum
authored and
cloud-fan
committed
[MINOR][PYTHON][TEST] Use collect() instead of show() to make the output silent
HyukjinKwon
committed
Revert [SPARK-19355][SPARK-25352]

viirya
authored and
cloud-fan
committed
[SPARK-25460][SS] DataSourceV2: SS sources do not respect SessionConfigSupport

HyukjinKwon
authored and
cloud-fan
committed
[SPARK-25417][SQL] ArrayContains function may return incorrect result when right expression is implicitly down casted

dilipbiswal
authored and
cloud-fan
committed
[MINOR][PYTHON] Use a helper in `PythonUtils` instead of direct accessing Scala package
HyukjinKwon
committed
[SPARK-25450][SQL] PushProjectThroughUnion rule uses the same exprId for project expressions in each Union child, causing mistakes in constant propagation

maryannxue
authored and
gatorsmile
committed
[SPARK-25381][SQL] Stratified sampling by Column argument

MaxGekk
authored and
HyukjinKwon
committed
[SPARK-24918][CORE] Executor Plugin API

NiharS
authored and
Marcelo Vanzin
committed
[SPARK-25366][SQL] Zstd and brotli CompressionCodec are not supported for parquet files

10110346
authored and
srowen
committed
[SPARK-25472][SS] Don't have legitimate stops of streams cause stream exceptions
brkyvz
committed

Commits on Sep 21, 2018

[SPARK-24777][SQL] Add write benchmark for AVRO

gengliangwang
authored and
gatorsmile
committed
Revert "[SPARK-23715][SQL] the input of to/from_utc_timestamp can not have timezone

gatorsmile
authored and
cloud-fan
committed
[SPARK-25494][SQL] Upgrade Spark's use of Janino to 3.0.10

rednaxelafx
authored and
gatorsmile
committed
[SPARK-25487][SQL][TEST] Refactor PrimitiveArrayBenchmark

seancxmao
authored and
kiszk
committed
[SPARK-25384][SQL] Clarify fromJsonForceNullableSchema will be removed in Spark 3.0

rxin
authored and
cloud-fan
committed
[SPARK-23549][SQL] Rename config spark.sql.legacy.compareDateTimestampInTimestamp

rxin
authored and
cloud-fan
committed
[SPARK-25469][SQL] Eval methods of Concat, Reverse and ElementAt should use pattern matching only once

mn-mikke
authored and
maropu
committed
[SPARK-24355] Spark external shuffle server improvement to better handle block fetch requests.

Sanket Chintapalli
authored and
tgravescs
committed
[SPARK-25499][TEST] Refactor BenchmarkBase and Benchmark

gengliangwang
authored and
cloud-fan
committed
[SPARK-19724][SQL] allowCreatingManagedTableUsingNonemptyLocation should have legacy prefix

rxin
authored and
gatorsmile
committed
[SPARK-25321][ML] Fix local LDA model constructor

WeichenXu123
authored and
mengxr
committed

Commits on Sep 22, 2018

[SPARK-25465][TEST] Refactor Parquet test suites in project Hive

gengliangwang
authored and
gatorsmile
committed

Commits on Sep 23, 2018

Commits on Sep 24, 2018

Commits on Sep 25, 2018

Commits on Sep 26, 2018

[SPARK-25422][CORE] Don't memory map blocks streamed to disk.

squito
authored and
cloud-fan
committed
[SPARK-24324][PYTHON][FOLLOW-UP] Rename the Conf to spark.sql.legacy.execution.pandas.groupedMap.assignColumnsByName

gatorsmile
and
HyukjinKwon
committed
[SPARK-21291][R] add R partitionBy API in DataFrame

huaxingao
authored and
HyukjinKwon
committed
[SPARK-25514][SQL] Generating pretty JSON by to_json

MaxGekk
authored and
HyukjinKwon
committed
[SPARK-25534][SQL] Make `SQLHelper` trait
dongjoon-hyun
committed
[SPARK-25541][SQL] CaseInsensitiveMap should be serializable after '-' or 'filterKeys'

gengliangwang
authored and
cloud-fan
committed
[SPARK-25379][SQL] Improve AttributeSet and ColumnPruning performance

mgaido91
authored and
cloud-fan
committed
[SPARK-20937][DOCS] Describe spark.sql.parquet.writeLegacyFormat property in Spark SQL, DataFrames and Datasets Guide

seancxmao
authored and
HyukjinKwon
committed
[SPARK-25509][CORE] Windows doesn't support POSIX permissions

Rong Tang
authored and
srowen
committed
[SPARK-25318] Add exception handling when wrapping the input stream during the the fetch or stage retry in response to a corrupted block

Reza Safi
authored and
Marcelo Vanzin
committed
[SPARK-24519][CORE] Compute SHUFFLE_MIN_NUM_PARTS_TO_HIGHLY_COMPRESS only once

rxin
authored and
dongjoon-hyun
committed
[SPARK-25533][CORE][WEBUI] AppSummary should hold the information about succeeded Jobs and completed stages only

shahidki31
authored and
Marcelo Vanzin
committed

Commits on Sep 27, 2018

[SPARK-25372][YARN][K8S] Deprecate and generalize keytab / principal config

ifilonenko
authored and
Marcelo Vanzin
committed
[SPARK-25454][SQL] add a new config for picking minimum precision for integral literals

cloud-fan
authored and
gatorsmile
committed
[SPARK-25540][SQL][PYSPARK] Make HiveContext in PySpark behave as the same as Scala.

ueshin
authored and
cloud-fan
committed
[SPARK-25481][SQL][TEST] Refactor ColumnarBatchBenchmark to use main method

committed
[SPARK-25536][CORE] metric value for METRIC_OUTPUT_RECORDS_WRITTEN is incorrect

shahidki31
authored and
dongjoon-hyun
committed
[SPARK-25525][SQL][PYSPARK] Do not update conf for existing SparkContext in SparkSession.getOrCreate.

ueshin
authored and
HyukjinKwon
committed
[SPARK-25468][WEBUI] Highlight current page index in the spark UI

authored and
srowen
committed
[SPARK-25485][SQL][TEST] Refactor UnsafeProjectionBenchmark to use main method

committed
[SPARK-23715][SQL][DOC] improve document for from/to_utc_timestamp
cloud-fan
committed
[SPARK-25522][SQL] Improve type promotion for input arguments of elementAt function

dilipbiswal
authored and
cloud-fan
committed
[SPARK-25314][SQL] Fix Python UDF accessing attributes from both side of join in join conditions

xuanyuanking
authored and
cloud-fan
committed
[SPARK-25551][SQL] Remove unused InSubquery expression

mgaido91
authored and
cloud-fan
committed
[SPARK-25541][SQL][FOLLOWUP] Remove overriding filterKeys in CaseInsensitiveMap

gengliangwang
authored and
cloud-fan
committed
[SPARK-21436][CORE] Take advantage of known partitioner for distinct on RDDs to avoid a shuffle

holdenk
authored and
cloud-fan
committed
[SPARK-24341][SQL][FOLLOWUP] remove duplicated error checking
cloud-fan
committed
[SPARK-25546][CORE] Don't cache value of EVENT_LOG_CALLSITE_LONG_FORM.

Marcelo Vanzin
authored and
dongjoon-hyun
committed

Commits on Sep 28, 2018

[SPARK-25459][SQL] Add viewOriginalText back to CatalogTable

Chris Zhao
and
dongjoon-hyun
committed
[SPARK-25505][SQL] The output order of grouping columns in Pivot is different from the input order

maryannxue
authored and
gatorsmile
committed
[SPARK-23285][DOC][FOLLOWUP] Fix missing markup tag
dongjoon-hyun
committed
[SPARK-25542][CORE][TEST] Move flaky test in OpenHashMapSuite to OpenHashSetSuite and make it against OpenHashSet

viirya
authored and
dongjoon-hyun
committed
[SPARK-25458][SQL] Support FOR ALL COLUMNS in ANALYZE TABLE

dilipbiswal
authored and
gatorsmile
committed
[SPARK-25429][SQL] Use Set instead of Array to improve lookup performance

wangyum
authored and
gatorsmile
committed
[SPARK-25449][CORE] Heartbeat shouldn't include accumulators for zero metrics

mukulmurthy
authored and
zsxwing
committed

Commits on Sep 29, 2018

[SPARK-25559][SQL] Remove the unsupported predicates in Parquet when possible

dbtsai
and
dongjoon-hyun
committed
[SPARK-25262][DOC][FOLLOWUP] Fix missing markup tag

dongjoon-hyun
authored and
HyukjinKwon
committed
[SPARK-25570][SQL][TEST] Replace 2.3.1 with 2.3.2 in HiveExternalCatalogVersionsSuite

dongjoon-hyun
authored and
HyukjinKwon
committed
[SPARK-25447][SQL] Support JSON options by schema_of_json()

MaxGekk
authored and
HyukjinKwon
committed
[SPARK-25262][DOC][FOLLOWUP] Fix link tags in html table

viirya
authored and
HyukjinKwon
committed
[SPARK-25048][SQL] Pivoting by multiple columns in Scala/Java

MaxGekk
authored and
HyukjinKwon
committed
[SPARK-25508][SQL][TEST] Refactor OrcReadBenchmark to use main method

committed
[SPARK-25572][SPARKR] test only if not cran

felixcheung
authored and
Felix Cheung
committed

Commits on Oct 1, 2018

Commits on Oct 2, 2018

Commits on Oct 3, 2018

Commits on Oct 5, 2018

[SPARK-17159][STREAM] Significant speed up for running spark streaming against Object store.

authored and
srowen
committed
[SPARK-25609][TESTS] Reduce time of test for SPARK-22226

mgaido91
authored and
gatorsmile
committed
[SPARK-25606][TEST] Reduce DateExpressionsSuite test time costs in Jenkins

wangyum
authored and
gatorsmile
committed
[SPARK-25605][TESTS] Run cast string to timestamp tests for a subset of timezones

mgaido91
authored and
gatorsmile
committed
[SPARK-25408] Move to mode ideomatic Java8

Fokko Driesprong
authored and
srowen
committed
Revert "[SPARK-25408] Move to mode ideomatic Java8"
cloud-fan
committed
[SPARK-25521][SQL] Job id showing null in the logs when insert into command Job is finished.

sujith71955
authored and
cloud-fan
committed
resolve conflicts
sjrand
committed
[SPARK-24601] Update Jackson to 2.9.6

Fokko Driesprong
authored and
HyukjinKwon
committed
merge in master again to get jackson fix
sjrand
committed
resolve conflict in pom after second master merge
sjrand
committed