[SPARK-11569] [ML] Fix StringIndexer to handle null value properly#9709

Closed

jliwork wants to merge 174 commits intoapache:masterfrom

jliwork:SPARK-11569

Commits on Nov 12, 2015

Fix for SPARK-11569
jliwork
committed

Commits on Nov 14, 2015

[SPARK-7970] Skip closure cleaning for SQL operations

nitindexter
authored and
Andrew Or
committed
[SPARK-11694][SQL] Parquet logical types are not being tested properly

HyukjinKwon
authored and
liancheng
committed
[MINOR][DOCS] typo in docs/configuration.md

vectorijk
authored and
srowen
committed
[SPARK-11573] Correct 'reflective access of structural type member meth…

gliptak
authored and
srowen
committed
Typo in comment: use 2 seconds instead of 1

Rohan Bhanderi
authored and
srowen
committed

Commits on Nov 15, 2015

Commits on Nov 16, 2015

[SPARK-11086][SPARKR] Use dropFactors column-wise instead of nested loop when createDataFrame

zero323
authored and
shivaram
committed
[SPARK-10500][SPARKR] sparkr.zip cannot be created if /R/lib is unwritable

Sun Rui
authored and
shivaram
committed
[SPARK-9928][SQL] Removal of LogicalLocalTable

gatorsmile
authored and
yhuai
committed
Revert "[SPARK-11572] Exit AsynchronousListenerBus thread when stop() is called"
JoshRosen
committed
[SPARK-11745][SQL] Enable more JSON parsing options
rxin
committed
[SPARK-11044][SQL] Parquet writer version fixed as version1

HyukjinKwon
authored and
liancheng
committed
[SPARK-11692][SQL] Support for Parquet logical types, JSON and BSON (embedded types)

HyukjinKwon
authored and
liancheng
committed
[SPARK-11522][SQL] input_file_name() returns "" for external tables

xwu0226
authored and
yhuai
committed
[SPARK-11752] [SQL] fix timezone problem for DateTimeUtils.getSeconds

cloud-fan
authored and
davies
committed
[SPARK-11743] [SQL] Add UserDefinedType support to RowEncoder

viirya
authored and
davies
committed
[SPARK-11731][STREAMING] Enable batching on Driver WriteAheadLog by default

brkyvz
authored and
tdas
committed
[SPARK-6328][PYTHON] Python API for StreamingListener

djalova
authored and
tdas
committed
[SPARK-11718][YARN][CORE] Fix explicitly killed executor dies silently issue

jerryshao
authored and
Marcelo Vanzin
committed
[SPARK-11754][SQL] consolidate `ExpressionEncoder.tuple` and `Encoders.tuple`

cloud-fan
authored and
marmbrus
committed
[SPARK-11390][SQL] Query plan with/without filterPushdown indistinguishable

Zee Chen
authored and
marmbrus
committed
Revert "[SPARK-11271][SPARK-11016][CORE] Use Spark BitSet instead of RoaringBitmap to reduce memory usage"
davies
committed
[SPARK-11742][STREAMING] Add the failure info to the batch lists

zsxwing
authored and
tdas
committed
[SPARK-11553][SQL] Primitive Row accessors should not convert null to default value

Bartlomiej Alberski
authored and
marmbrus
committed
[SPARK-8658][SQL] AttributeReference's equals method compares all the members

gatorsmile
authored and
marmbrus
committed
[SPARK-11625][SQL] add java test for typed aggregate

cloud-fan
authored and
marmbrus
committed

Commits on Nov 17, 2015

[SPARKR][HOTFIX] Disable flaky SparkR package build test

shivaram
authored and
Andrew Or
committed
[SPARK-11480][CORE][WEBUI] Wrong callsite is displayed when using AsyncRDDActions#takeAsync

sarutak
authored and
Andrew Or
committed
[SPARK-11710] Document new memory management model
Andrew Or
committed
[EXAMPLE][MINOR] Add missing awaitTermination in click stream example

jerryshao
authored and
Andrew Or
committed
[SPARK-11612][ML] Pipeline and PipelineModel persistence
jkbradley
committed
[SPARK-11617][NETWORK] Fix leak in TransportFrameDecoder.
Marcelo Vanzin
committed
[SPARK-11768][SPARK-9196][SQL] Support now function in SQL (alias for current_timestamp).
rxin
committed
[SPARK-11694][FOLLOW-UP] Clean up imports, use a common function for metadata and add a test for FIXED_LEN_BYTE_ARRAY

HyukjinKwon
authored and
liancheng
committed
[SPARK-11447][SQL] change NullType to StringType during binaryComparison between NullType and StringType

kevinyu98
authored and
yhuai
committed
[MINOR] [SQL] Fix randomly generated ArrayData in RowEncoderSuite

viirya
authored and
davies
committed
[SPARK-11191][SQL][FOLLOW-UP] Cleans up unnecessary anonymous HiveFunctionRegistry
liancheng
committed
[SPARK-11751] Doc describe error in the "Spark Streaming Programming Guide" page

wypb
authored and
srowen
committed
[SPARK-11779][DOCS] Fix reference to deprecated MESOS_NATIVE_LIBRARY

philipphoffmann
authored and
srowen
committed
[SPARK-11744][LAUNCHER] Fix print version throw exception when using pyspark shell

jerryshao
authored and
Marcelo Vanzin
committed
[SPARK-11695][CORE] Set s3a credentials

Chris Bannister
authored and
Marcelo Vanzin
committed
[SPARK-11766][MLLIB] add toJson/fromJson to Vector/Vectors

mengxr
authored and
jkbradley
committed
[SPARK-11679][SQL] Invoking method " apply(fields: java.util.List[StructField])" in "StructType" gets ClassCastException

jackieMaKing
authored and
marmbrus
committed
[SPARK-11089][SQL] Adds option for disabling multi-session in Thrift server

liancheng
authored and
marmbrus
committed
[SPARK-8658][SQL][FOLLOW-UP] AttributeReference's equals method compares all the members

gatorsmile
authored and
marmbrus
committed
[SPARK-10186][SQL] support postgre array type in JDBCRDD

cloud-fan
authored and
marmbrus
committed
[SPARK-11769][ML] Add save, load to all basic Transformers

jkbradley
authored and
mengxr
committed
[SPARK-11767] [SQL] limit the size of caced batch

Davies Liu
authored and
davies
committed
[SPARK-11732] Removes some MiMa false positives

thunterdb
authored and
srowen
committed
[SPARK-11729] Replace example code in ml-linear-methods.md using include_example

yinxusen
authored and
mengxr
committed
[SPARK-11763][ML] Add save,load to LogisticRegression Estimator

jkbradley
authored and
mengxr
committed
[SPARK-11764][ML] make Param.jsonEncode/jsonDecode support Vector
mengxr
committed
[SPARK-11786][CORE] Tone down messages from akka error monitor.
Marcelo Vanzin
committed
[SPARK-11740][STREAMING] Fix the race condition of two checkpoints in a batch

zsxwing
authored and
tdas
committed
[SPARK-9552] Add force control for killExecutors to avoid false killing for those busy executors

GraceH
authored and
Andrew Or
committed
[MINOR] Correct comments in JavaDirectKafkaWordCount

Rohan Bhanderi
authored and
Andrew Or
committed
[SPARK-11790][STREAMING][TESTS] Increase the connection timeout

zsxwing
authored and
Andrew Or
committed
[SPARK-11771][YARN][TRIVIAL] maximum memory in yarn is controlled by two params have both in error msg

holdenk
authored and
Andrew Or
committed

Commits on Nov 18, 2015

[SPARK-11726] Throw exception on timeout when waiting for REST server response

jacek-lewandowski
authored and
Andrew Or
committed
[SPARK-9065][STREAMING][PYSPARK] Add MessageHandler for Kafka Python API

jerryshao
authored and
tdas
committed
[SPARK-11793][SQL] Dataset should set the resolved encoders internally for maps.
rxin
committed
[SPARK-11016] Move RoaringBitmap to explicit Kryo serializer

Davies Liu
authored and
davies
committed
[SPARK-11583] [CORE] MapStatus Using RoaringBitmap More Properly

yaooqinn
authored and
davies
committed
[SPARK-11737] [SQL] Fix serialization of UTF8String with Kyro

Davies Liu
authored and
davies
committed
[SPARK-11797][SQL] collect, first, and take should use encoders for serialization
rxin
committed
[SPARK-11755][R] SparkR should export "predict"

yanboliang
authored and
shivaram
committed
[SPARK-11761] Prevent the call to StreamingContext#stop() in the listener bus's thread

tedyu
authored and
tdas
committed
[SPARK-7013][ML][TEST] Add unit test for spark.ml StandardScaler

RoyGao
authored and
mengxr
committed
[SPARK-11643] [SQL] parse year with leading zero

Davies Liu
authored and
davies
committed
[SPARK-11728] Replace example code in ml-ensembles.md using include_example

yinxusen
authored and
mengxr
committed
[SPARK-10186][SQL][FOLLOW-UP] simplify test

cloud-fan
authored and
rxin
committed
[SPARK-11802][SQL] Kryo-based encoder for opaque types in Datasets
rxin
committed
[SPARK-11792][SQL] SizeEstimator cannot provide a good size estimation of UnsafeHashedRelations

yhuai
authored and
rxin
committed
[SPARK-10946][SQL] JDBC - Use Statement.executeUpdate instead of PreparedStatement.executeUpdate for DDLs

somideshmukh
authored and
srowen
committed
[SPARK-6541] Sort executors by ID (numeric)

jbonofre
authored and
srowen
committed
[SPARK-11652][CORE] Remote code execution with InvokerTransformer
srowen
committed
rmse was wrongly calculated

Viveka Kulharia
authored and
srowen
committed
[SPARK-11804] [PYSPARK] Exception raise when using Jdbc predicates opt…

zjffdu
authored and
davies
committed
[SPARK-11281][SPARKR] Add tests covering the issue.

zero323
authored and
shivaram
committed
[SPARK-11773][SPARKR] Implement collection functions in SparkR.

Sun Rui
authored and
shivaram
committed
[SPARK-11195][CORE] Use correct classloader for TaskResultGetter

Hurshal Patel
authored and
yhuai
committed
[SPARK-11803][SQL] fix Dataset self-join

cloud-fan
authored and
marmbrus
committed
[SPARK-11725][SQL] correctly handle null inputs for UDF

cloud-fan
authored and
marmbrus
committed
[SPARK-11795][SQL] combine grouping attributes into a single NamedExpression

cloud-fan
authored and
marmbrus
committed
[MINOR][BUILD] Ignore ensime cache

jodersky
authored and
JoshRosen
committed
[SPARK-11792] [SQL] [FOLLOW-UP] Change SizeEstimation to KnownSizeEstimation and make estimatedSize return Long instead of Option[Long]
yhuai
committed
[SPARK-11739][SQL] clear the instantiated SQLContext

Davies Liu
authored and
yhuai
committed
[SPARK-4557][STREAMING] Spark Streaming foreachRDD Java API method should accept a VoidFunction<...>

BryanCutler
authored and
tdas
committed
[SPARK-11809] Switch the default Mesos mode to coarse-grained mode
rxin
committed
[SPARK-10745][CORE] Separate configs between shuffle and RPC

zsxwing
authored and
Marcelo Vanzin
committed
[SPARK-11720][SQL][ML] Handle edge cases when count = 0 or 1 for Stats function

JihongMA
authored and
mengxr
committed
[SPARK-6790][ML] Add spark.ml LinearRegression import/export

fayeshine
authored and
mengxr
committed
[SPARK-6789][ML] Add Readable, Writable support for spark.ml ALS, ALSModel

jkbradley
authored and
mengxr
committed
[SPARK-11813][MLLIB] Avoid serialization of vocab in Word2Vec

hhbyyh
authored and
mengxr
committed
[SPARK-11684][R][ML][DOC] Update SparkR glm API doc, user guide and example codes

yanboliang
authored and
mengxr
committed
[SPARK-11820][ML][PYSPARK] PySpark LiR & LoR should support weightCol

yanboliang
authored and
mengxr
committed
[SPARK-11544][SQL] sqlContext doesn't use PathFilter

dilipbiswal
authored and
yhuai
committed
[SPARK-11810][SQL] Java-based encoder for opaque types in Datasets.
rxin
committed
[SPARK-6787][ML] add read/write to estimators under ml.feature (1)
mengxr
committed
[SPARK-11649] Properly set Akka frame size in SparkListenerSuite test

JoshRosen
authored and
Andrew Or
committed
[SPARK-10930] History "Stages" page "duration" can be confusing

Derek Dagit
authored and
Andrew Or
committed

Commits on Nov 19, 2015

[SPARK-11495] Fix potential socket / file handle leaks that were found via static analysis

JoshRosen
authored and
rxin
committed
[SPARK-11814][STREAMING] Add better default checkpoint duration
tdas
committed
[SPARK-11791] Fix flaky test in BatchedWriteAheadLogSuite

brkyvz
authored and
tdas
committed
[SPARK-11636][SQL] Support classes defined in the REPL with Encoders
marmbrus
committed
[SPARK-11839][ML] refactor save/write traits

mengxr
authored and
jkbradley
committed
[SPARK-11833][SQL] Add Java tests for Kryo/Java Dataset encoders
rxin
committed
[SPARK-11787][SQL] Improve Parquet scan performance when using flat schemas.

nongli
authored and
rxin
committed
Revert "[SPARK-11544][SQL] sqlContext doesn't use PathFilter"
yhuai
committed
[SPARK-11816][ML] fix some style issue in ML/MLlib examples

hhbyyh
authored and
mengxr
committed
[SPARK-11614][SQL] serde parameters should be set only when all params are ready

navis
authored and
yhuai
committed
[SPARK-11842][ML] Small cleanups to existing Readers and Writers

jkbradley
authored and
mengxr
committed
[SPARK-11339][SPARKR] Document the list of functions in R base package that are masked by functions with same name in SparkR

felixcheung
authored and
shivaram
committed
[SPARK-11569] [ML] fix merge conflict
jliwork
committed
revised fix based on comments
jliwork
committed
fix style error
jliwork
committed
[SPARK-11849][SQL] Analyzer should replace current_date and current_timestamp with literals
rxin
committed
[SPARK-11840][SQL] Restore the 1.5's behavior of planning a single distinct aggregation.
yhuai
committed
[SPARK-11830][CORE] Make NettyRpcEnv bind to the specified host

zsxwing
authored and
rxin
committed
[SPARK-11633][SQL] LogicalRDD throws TreeNode Exception : Failed to Copy Node

gatorsmile
authored and
marmbrus
committed
[SPARK-11848][SQL] Support EXPLAIN in DataSet APIs

gatorsmile
authored and
marmbrus
committed
[SPARK-11750][SQL] revert SPARK-11727 and code clean up

cloud-fan
authored and
marmbrus
committed
[SPARK-11778][SQL] parse table name before it is passed to lookupRelation

Huaxin Gao
authored and
marmbrus
committed
[SPARK-11812][PYSPARK] invFunc=None works properly with python's reduceByKeyAndWindow

dtolpin
authored and
tdas
committed
[SPARK-11858][SQL] Move sql.columnar into sql.execution.
rxin
committed
[SPARK-11831][CORE][TESTS] Use port 0 to avoid port conflicts in tests

zsxwing
authored and
Andrew Or
committed
[SPARK-11799][CORE] Make it explicit in executor logs that uncaught e…

Srinivasa Reddy Vundela
authored and
Andrew Or
committed
[SPARK-11828][CORE] Register DAGScheduler metrics source after app id is known.

Marcelo Vanzin
authored and
Andrew Or
committed
[SPARK-11746][CORE] Use cache-aware method dependencies

suyanNone
authored and
Andrew Or
committed
[SPARK-11275][SQL] Incorrect results when using rollup/cube

aray
authored and
yhuai
committed
[SPARK-11569] [ML] resolve merge conflict
jliwork
committed

Commits on Nov 20, 2015

[SPARK-4134][CORE] Lower severity of some executor loss logs.

Marcelo Vanzin
authored and
Andrew Or
committed
[SPARK-11845][STREAMING][TEST] Added unit test to verify TrackStateRDD is correctly checkpointed

tdas
authored and
Andrew Or
committed
[SPARK-11864][SQL] Improve performance of max/min

Davies Liu
authored and
rxin
committed
[SPARK-11544][SQL][TEST-HADOOP1.0] sqlContext doesn't use PathFilter

dilipbiswal
authored and
yhuai
committed
[SPARK-11846] Add save/load for AFTSurvivalRegression and IsotonicRegression

yinxusen
authored and
mengxr
committed
[SPARK-11829][ML] Add read/write to estimators under ml.feature (II)

yanboliang
authored and
mengxr
committed
[SPARK-11875][ML][PYSPARK] Update doc for PySpark HasCheckpointInterval

yanboliang
authored and
mengxr
committed
[SPARK-11869][ML] Clean up TempDirectory properly in ML tests

jkbradley
authored and
mengxr
committed
[SPARK-11867] Add save/load for kmeans and naive bayes

yinxusen
authored and
mengxr
committed
[SPARK-11877] Prevent agg. fallback conf. from leaking across test suites
JoshRosen
committed
[SPARK-11852][ML] StandardScaler minor refactor

yanboliang
authored and
mengxr
committed
[SPARK-11689][ML] Add user guide and example code for LDA under spark.ml

hhbyyh
authored and
mengxr
committed
[SPARK-11876][SQL] Support printSchema in DataSet API

gatorsmile
authored and
marmbrus
committed
[SPARK-11817][SQL] Truncating the fractional seconds to prevent inserting a NULL

viirya
authored and
yhuai
committed
[SPARK-11819][SQL] nice error message for missing encoder

cloud-fan
authored and
marmbrus
committed
[SPARK-11650] Reduce RPC timeouts to speed up slow AkkaUtilsSuite test
JoshRosen
committed
[SPARK-11724][SQL] Change casting between int and timestamp to consistently treat int in seconds.

nongli
authored and
yhuai
committed
[SPARK-11870][STREAMING][PYSPARK] Rethrow the exceptions in TransformFunction and TransformFunctionSerializer

zsxwing
authored and
tdas
committed
[SPARK-11887] Close PersistenceEngine at the end of PersistenceEngineSuite tests
JoshRosen
committed
[SPARK-11716][SQL] UDFRegistration just drops the input type when re-creating the UserDefinedFunction

jbonofre
authored and
marmbrus
committed
[SPARK-11756][SPARKR] Fix use of aliases - SparkR can not output help information for SparkR:::summary correctly

felixcheung
authored and
shivaram
committed
[SPARK-11636][SQL] Support classes defined in the REPL with Encoders
marmbrus
committed
[SPARK-11549][DOCS] Replace example code in mllib-evaluation-metrics.md using include_example

Vikas Nelamangala
authored and
mengxr
committed
[SPARK-11787][SPARK-11883][SQL][FOLLOW-UP] Cleanup for this patch.

nongli
authored and
rxin
committed
[SPARK-11889][SQL] Fix type inference for GroupedDataset.agg in REPL
marmbrus
committed
[SPARK-11890][SQL] Fix compilation for Scala 2.11
marmbrus
committed

Commits on Nov 22, 2015

[SPARK-11908][SQL] Add NullType support to RowEncoder

viirya
authored and
marmbrus
committed

Commits on Nov 23, 2015