[SPARK-23429][CORE] Add executor memory metrics to heartbeat and expose in executors REST API#20940

Closed

edwinalu wants to merge 145 commits intoapache:masterfrom

edwinalu:SPARK-23429

Commits on Mar 28, 2018

SPARK-23429: Add executor memory metrics to heartbeat and expose in executors REST API
edwinalu
committed

Commits on Mar 30, 2018

[SPARK-23808][SQL] Set default Spark session in test-only spark sessions.

jose-torres
authored and
gatorsmile
committed
[SPARK-23743][SQL] Changed a comparison logic from containing 'slf4j' to starting with 'org.slf4j'

jongyoul
authored and
jerryshao
committed
[SPARK-23727][SQL] Support for pushing down filters for DateType in parquet

yucai
authored and
cloud-fan
committed
Roll forward "[SPARK-23096][SS] Migrate rate source to V2"

jose-torres
authored and
cloud-fan
committed
[SPARK-23500][SQL][FOLLOWUP] Fix complex type simplification rules to apply to entire plan

gatorsmile
authored and
cloud-fan
committed
[SPARK-23640][CORE] Fix hadoop config may override spark config

wangyum
authored and
Marcelo Vanzin
committed
[SPARK-23827][SS] StreamingJoinExec should ensure that input data is partitioned into specific number of partitions
tdas
committed

Commits on Mar 31, 2018

[SPARK-23040][CORE][FOLLOW-UP] Avoid double wrap result Iterator.

jiangxb1987
authored and
cloud-fan
committed

Commits on Apr 2, 2018

modify MimaExcludes.scala to filter changes to SparkListenerExecutorMetricsUpdate
edwinalu
committed
[SPARK-15009][PYTHON][FOLLOWUP] Add default param checks for CountVectorizerModel
BryanCutler
committed
[SPARK-23825][K8S] Requesting memory + memory overhead for pod memory

dvogelbacher
authored and
mccheah
committed
[SPARK-23285][K8S] Add a config property for specifying physical executor cores

liyinan926
authored and
foxish
committed
[SPARK-23713][SQL] Cleanup UnsafeWriter and BufferHolder classes

kiszk
authored and
hvanhovell
committed
[SPARK-23834][TEST] Wait for connection before disconnect in LauncherServer test.
Marcelo Vanzin
committed
[SPARK-23690][ML] Add handleinvalid to VectorAssembler

Yogesh Garg
authored and
jkbradley
committed

Commits on Apr 3, 2018

[SPARK-19964][CORE] Avoid reading from remote repos in SparkSubmitSuite.

Marcelo Vanzin
authored and
HyukjinKwon
committed
[MINOR][DOC] Fix a few markdown typos

lemonjing
authored and
HyukjinKwon
committed
[MINOR][CORE] Show block manager id when remove RDD/Broadcast fails.

jiangxb1987
authored and
HyukjinKwon
committed
[SPARK-23099][SS] Migrate foreach sink to DataSourceV2

jose-torres
authored and
tdas
committed
[SPARK-23587][SQL] Add interpreted execution for MapObjects expression

viirya
authored and
hvanhovell
committed

Commits on Apr 4, 2018

[SPARK-23809][SQL] Active SparkSession should be set by getOrCreate

ericl
authored and
gatorsmile
committed
[SPARK-23802][SQL] PropagateEmptyRelation can leave query plan in unresolved state

Robert Kruszewski
authored and
gatorsmile
committed
[SPARK-23826][TEST] TestHiveSparkSession should set default session

gatorsmile
authored and
cloud-fan
committed
[SPARK-21351][SQL] Update nullability based on children's output

maropu
authored and
cloud-fan
committed
[SPARK-23583][SQL] Invoke should support interpreted execution

kiszk
authored and
hvanhovell
committed
[SPARK-23668][K8S] Add config option for passing through k8s Pod.spec.imagePullSecrets

Andrew Korzhuev
authored and
foxish
committed
[SPARK-23838][WEBUI] Running SQL query is displayed as "completed" in SQL tab

gengliangwang
authored and
Marcelo Vanzin
committed
[SPARK-23637][YARN] Yarn might allocate more resource if a same executor is killed multiple times.

jinxing
authored and
Marcelo Vanzin
committed

Commits on Apr 6, 2018

[SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks to choose several types of memory block

kiszk
authored and
cloud-fan
committed
[SPARK-23870][ML] Forward RFormula handleInvalid Param to VectorAssembler to handle invalid values in non-string columns

Yogesh Garg
authored and
jkbradley
committed
[SPARK-23823][SQL] Keep origin in transformExpression

JiahuiJiang
authored and
gatorsmile
committed
[SPARK-19724][SQL] create a managed table with an existed default table should throw an exception

gengliangwang
authored and
gatorsmile
committed
[MINOR][DOC] Fix some typos and grammar issues

dsakuma
authored and
HyukjinKwon
committed
[SPARK-23822][SQL] Improve error message for Parquet schema mismatches

yuchenhuo
authored and
gatorsmile
committed
[SPARK-23861][SQL][DOC] Clarify default window frame with and without orderBy clause

icexelloss
authored and
HyukjinKwon
committed
[SPARK-23882][CORE] UTF8StringSuite.writeToOutputStreamUnderflow() is not expected to be supported

kiszk
authored and
hvanhovell
committed
[SPARK-23859][ML] Initial PR for Instrumentation improvements: UUID and logging levels

MrBago
authored and
jkbradley
committed
[SPARK-10399][SPARK-23879][HOTFIX] Fix Java lint errors

kiszk
authored and
ueshin
committed
[SPARK-23828][ML][PYTHON] PySpark StringIndexerModel should have constructor from labels

huaxingao
authored and
BryanCutler
committed
[SPARK-21898][ML][FOLLOWUP] Fix Scala 2.12 build.
ueshin
committed

Commits on Apr 8, 2018

Commits on Apr 9, 2018

Fix typo in Python docstring kinesis example

Nolan Emirot
authored and
srowen
committed
[SPARK-23881][CORE][TEST] Fix flaky test JobCancellationSuite."interruptible iterator of shuffle reader"

jiangxb1987
authored and
gatorsmile
committed
[SPARK-23816][CORE] Killed tasks should ignore FetchFailures.

squito
authored and
Marcelo Vanzin
committed
[SPARK-22856][SQL] Add wrappers for codegen output and nullability

viirya
authored and
gatorsmile
committed
[SPARK-14681][ML] Provide label/impurity stats for spark.ml decision tree nodes

WeichenXu123
authored and
jkbradley
committed
[INFRA] Close stale PRs.
Marcelo Vanzin
committed

Commits on Apr 10, 2018

[SPARK-23947][SQL] Add hashUTF8String convenience method to hasher classes

rednaxelafx
authored and
gatorsmile
committed
[SPARK-23898][SQL] Simplify add & subtract code generation

hvanhovell
authored and
gatorsmile
committed
[SPARK-23841][ML] NodeIdCache should unpersist the last cached nodeIdsForInstances

zhengruifeng
authored and
srowen
committed
[SPARK-23864][SQL] Add unsafe object writing to UnsafeWriter
hvanhovell
committed
[SPARK-19724][SQL][FOLLOW-UP] Check location of managed table when ignoreIfExists is true

gengliangwang
authored and
gatorsmile
committed
[SPARK-23751][ML][PYSPARK] Kolmogorov-Smirnoff test Python API in pyspark.ml

WeichenXu123
authored and
jkbradley
committed
[SPARK-23871][ML][PYTHON] add python api for VectorAssembler handleInvalid

huaxingao
authored and
jkbradley
committed

Commits on Apr 11, 2018

[SPARK-23944][ML] Add the set method for the two LSHModel

lu-wang-dl
authored and
jkbradley
committed
[SPARK-23847][FOLLOWUP][PYTHON][SQL] Actually test [desc|acs]_nulls_[first|last] functions in PySpark
HyukjinKwon
committed
[MINOR][DOCS] Fix R documentation generation instruction for roxygen2
HyukjinKwon
committed
[SPARK-23951][SQL] Use actual java class instead of string representation.

hvanhovell
authored and
cloud-fan
committed
[SPARK-23960][SQL][MINOR] Mark HashAggregateExec.bufVars as transient

rednaxelafx
authored and
cloud-fan
committed
[SPARK-6951][CORE] Speed up parsing of event logs during listing.

Marcelo Vanzin
authored and
squito
committed
[SPARK-22941][CORE] Do not exit JVM when submit fails with in-process launcher.

Marcelo Vanzin
authored and
squito
committed
[SPARK-22883] ML test for StructuredStreaming: spark.ml.feature, I-M
jkbradley
committed
typo rawPredicition changed to rawPrediction

JBauerKogentix
authored and
jkbradley
committed

Commits on Apr 12, 2018

Commits on Apr 13, 2018

[SPARK-23867][SCHEDULER] use droppedCount in logWarning

Patrick Pisciuneri
authored and
cloud-fan
committed
[SPARK-23748][SS] Fix SS continuous process doesn't support SubqueryAlias issue

jerryshao
authored and
tdas
committed
[SPARK-23942][PYTHON][SQL] Makes collect in PySpark as action for a query executor listener
HyukjinKwon
committed
[SPARK-23971] Should not leak Spark sessions across test suites

ericl
authored and
gatorsmile
committed
[SPARK-23815][CORE] Spark writer dynamic partition overwrite mode may fail to write output on multi level partition

Fangshi Li
authored and
cloud-fan
committed
[SPARK-23905][SQL] Add UDF weekday

yucai
authored and
gatorsmile
committed
[SPARK-22839][K8S] Refactor to unify driver and executor pod builder APIs

mccheah
authored and
foxish
committed
[SPARK-23896][SQL] Improve PartitioningAwareFileIndex

gengliangwang
authored and
cloud-fan
committed
[SPARK-23375][SQL] Eliminate unneeded Sort in Optimizer

mgaido91
authored and
cloud-fan
committed
[SPARK-23963][SQL] Properly handle large number of columns in query on text-based Hive table

bersprockets
authored and
gatorsmile
committed
[SPARK-23966][SS] Refactoring all checkpoint file writing logic in a common CheckpointFileManager interface
tdas
committed

Commits on Apr 14, 2018

[SPARK-23979][SQL] MultiAlias should not be a CodegenFallback

viirya
authored and
cloud-fan
committed

Commits on Apr 16, 2018

[SPARK-23956][YARN] Use effective RPC port in AM registration

gerashegalov
authored and
jerryshao
committed
[SPARK-23917][SQL] Add array_max function

mgaido91
authored and
gatorsmile
committed
[SPARK-21033][CORE][FOLLOW-UP] Update Spillable

wangyum
authored and
cloud-fan
committed
[SPARK-9312][ML] Add RawPrediction, numClasses, and numFeatures for OneVsRestModel

lu-wang-dl
authored and
jkbradley
committed
[SPARK-21088][ML] CrossValidator, TrainValidationSplit support collect all models when fitting: Python API

WeichenXu123
authored and
jkbradley
committed
[SPARK-23873][SQL] Use accessors in interpreted LambdaVariable

viirya
authored and
hvanhovell
committed

Commits on Apr 17, 2018

[SPARK-23918][SQL] Add array_min function

mgaido91
authored and
ueshin
committed
[SPARK-23687][SS] Add a memory source for continuous processing.

jose-torres
authored and
tdas
committed
[SPARK-23747][STRUCTURED STREAMING] Add EpochCoordinator unit tests

Efim Poberezkin
authored and
tdas
committed
[SPARK-23875][SQL] Add IndexedSeq wrapper for ArrayData

viirya
authored and
hvanhovell
committed
[SPARK-23835][SQL] Add not-null check to Tuples' arguments deserialization

mgaido91
authored and
cloud-fan
committed
[SPARK-22676] Avoid iterating all partition paths when spark.sql.hive.verifyPartitionPath=true

jinxing
authored and
cloud-fan
committed
[SPARK-23948] Trigger mapstage's job listener in submitMissingTasks

jinxing
authored and
squito
committed
[SPARK-23986][SQL] freshName can generate non-unique names

mgaido91
authored and
cloud-fan
committed
[SPARK-21741][ML][PYSPARK] Python API for DataFrame-based multivariate summarizer

WeichenXu123
authored and
jkbradley
committed

Commits on Apr 18, 2018

[SPARK-22968][DSTREAM] Throw an exception on partition revoking issue

jerryshao
authored and
koeninger
committed
[SPARK-21479][SQL] Outer join filter pushdown in null supplying table when condition is on one of the joined columns

maryannxue
authored and
cloud-fan
committed
[SPARK-23341][SQL] define some standard options for data source v2
cloud-fan
committed
[SPARK-24002][SQL] Task not serializable caused by org.apache.parquet.io.api.Binary$ByteBufferBackedBinary.getBytes
gatorsmile
committed
[SPARK-23926][SQL] Extending reverse function to support ArrayType arguments

mn-mikke
authored and
ueshin
committed
[SPARK-24007][SQL] EqualNullSafe for FloatType and DoubleType might generate a wrong result by codegen.

ueshin
authored and
gatorsmile
committed
[SPARK-23875][SQL][FOLLOWUP] Add IndexedSeq wrapper for ArrayData

viirya
authored and
hvanhovell
committed
[SPARK-23775][TEST] Make DataFrameRangeSuite not flaky

gaborgsomogyi
authored and
Marcelo Vanzin
committed

Commits on Apr 19, 2018

[SPARK-24014][PYSPARK] Add onStreamingStarted method to StreamingListener

viirya
authored and
jerryshao
committed
[SPARK-23919][SQL] Add array_position function

kiszk
authored and
ueshin
committed
[SPARK-23924][SQL] Add element_at function

kiszk
authored and
ueshin
committed
[SPARK-23584][SQL] NewInstance should support interpreted execution

maropu
authored and
hvanhovell
committed
[SPARK-23588][SQL] CatalystToExternalMap should support interpreted execution

maropu
authored and
hvanhovell
committed
[SPARK-22676][FOLLOW-UP] fix code style for test.

jinxing
authored and
cloud-fan
committed
[SPARK-21811][SQL] Fix the inconsistency behavior when finding the widest common type

jiangxb1987
authored and
HyukjinKwon
committed
[SPARK-24021][CORE] fix bug in BlacklistTracker's updateBlacklistForFetchFailure

Ngone51
authored and
squito
committed
[SPARK-23989][SQL] exchange should copy data before non-serialized shuffle

cloud-fan
authored and
hvanhovell
committed
[SPARK-24026][ML] Add Power Iteration Clustering to spark.ml

wangmiao1981
authored and
jkbradley
committed
[SPARK-22362][SQL] Add unit test for Window Aggregate Functions

attilapiros
authored and
hvanhovell
committed
[SPARK-24022][TEST] Make SparkContextSuite not flaky

gaborgsomogyi
authored and
Marcelo Vanzin
committed

Commits on Apr 20, 2018

[SPARK-23877][SQL] Use filter predicates to prune partitions in metadata-only queries

rdblue
authored and
cloud-fan
committed
[SPARK-23736][SQL] Extending the concat function to support array columns

mn-mikke
authored and
ueshin
committed
[SPARK-23588][SQL][FOLLOW-UP] Resolve a map builder method per execution in CatalystToExternalMap

maropu
authored and
hvanhovell
committed
[SPARK-23595][SQL] ValidateExternalType should support interpreted execution

maropu
authored and
hvanhovell
committed
Revert "[SPARK-23775][TEST] Make DataFrameRangeSuite not flaky"
Marcelo Vanzin
committed

Commits on Apr 21, 2018

[SPARK-24029][CORE] Set SO_REUSEADDR on listen sockets.

Marcelo Vanzin
authored and
HyukjinKwon
committed
[SPARK-24033][SQL] Fix Mismatched of Window Frame specifiedwindowframe(RowFrame, -1, -1)
gatorsmile
committed

Commits on Apr 22, 2018

Address code review comments, change event logging to stage end.
edwinalu
committed
[SPARK-23799][SQL] FilterEstimation.evaluateInSet produces devision by zero in a case of empty table with analyzed statistics

Mykhailo Shtelma
authored and
gatorsmile
committed

Commits on Apr 23, 2018