Spark diditaxi#1

Open

cenyuhai wants to merge 450 commits intoguowei2:masterfrom

cenyuhai:spark-diditaxi

This pull request is big! We're only showing the most recent 250 commits

Commits on Feb 12, 2015

[SPARK-5762] Fix shuffle write time for sort-based shuffle

kayousterhout
authored and
Andrew Or
committed
[SPARK-5760][SPARK-5761] Fix standalone rest protocol corner cases + revamp tests
Andrew Or
committed
[SPARK-5759][Yarn]ExecutorRunnable should catch YarnException while NMClient start contain...

lianhuiwang
authored and
Andrew Or
committed
SPARK-5747: Fix wordsplitting bugs in make-distribution.sh

dyross
authored and
Andrew Or
committed
[SPARK-5780] [PySpark] Mute the logging during unit tests

Davies Liu
authored and
Andrew Or
committed
[SPARK-5758][SQL] Use LongType as the default type for integers in JSON schema inference.

yhuai
authored and
marmbrus
committed
[SPARK-5573][SQL] Add explode to dataframes
marmbrus
committed
[SPARK-5755] [SQL] remove unnecessary Add

adrian-wang
authored and
marmbrus
committed
[SPARK-5335] Fix deletion of security groups within a VPC

Vladimir Grigor
authored and
srowen
committed
[SQL] Move SaveMode to SQL package.

yhuai
authored and
marmbrus
committed

Commits on Feb 13, 2015

[SPARK-3299][SQL]Public API in SQLContext to list tables

yhuai
authored and
marmbrus
committed
[SPARK-3365][SQL]Wrong schema generated for List type

tianyi
authored and
liancheng
committed
[SQL] Fix docs of SQLContext.tables

yhuai
authored and
liancheng
committed
[SPARK-4832][Deploy]some other processes might take the daemon pid

WangTaoTheTonic
authored and
srowen
committed
[SPARK-5732][CORE]:Add an option to print the spark version in spark script.

uncleGen
authored and
Andrew Or
committed
[SPARK-5503][MLLIB] Example code for Power Iteration Clustering

sboeschhuawei
authored and
mengxr
committed
[SPARK-5783] Better eventlog-parsing error messages

ryan-williams
authored and
Andrew Or
committed
[SPARK-5735] Replace uses of EasyMock with Mockito

JoshRosen
authored and
Andrew Or
committed
SPARK-5805 Fixed the type error in documentation.

emres
authored and
mengxr
committed
[HOTFIX] Fix build break in MesosSchedulerBackendSuite
Andrew Or
committed
[SPARK-5642] [SQL] Apply column pruning on unused aggregation fields

adrian-wang
authored and
marmbrus
committed
[SPARK-5789][SQL]Throw a better error message if JsonRDD.parseJson encounters unrecoverable parsing errors.

yhuai
authored and
marmbrus
committed
[SPARK-5806] re-organize sections in mllib-clustering.md
mengxr
committed

Commits on Feb 14, 2015

Commits on Feb 15, 2015

Commits on Feb 16, 2015

[SPARK-5769] Set params in constructors and in setParams in Python ML pipelines
mengxr
committed
SPARK-5815 [MLLIB] Deprecate SVDPlusPlus APIs that expose DoubleMatrix from JBLAS

srowen
authored and
mengxr
committed
[Ml] SPARK-5796 Don't transform data on a last estimator in Pipeline

petro-rudenko
authored and
mengxr
committed
[Ml] SPARK-5804 Explicitly manage cache in Crossvalidator k-fold loop

petro-rudenko
authored and
mengxr
committed
[Minor] [SQL] Renames stringRddToDataFrame to stringRddToDataFrameHolder for consistency
liancheng
committed
[SPARK-4553] [SPARK-5767] [SQL] Wires Parquet data source with the newly introduced write support for data source API
liancheng
committed
[SPARK-5799][SQL] Compute aggregation function on specified numeric columns

viirya
authored and
rxin
committed
SPARK-5795 [STREAMING] api.java.JavaPairDStream.saveAsNewAPIHadoopFiles may not friendly to java
srowen
committed
[SQL] [Minor] Update the SpecificMutableRow.copy

chenghao-intel
authored and
marmbrus
committed
[SPARK-5824] [SQL] add null format in ctas and set default col comment to null

adrian-wang
authored and
marmbrus
committed
[SQL] Initial support for reporting location of error in sql string
marmbrus
committed
[SQL] Add fetched row count in SparkSQLCLIDriver

OopsOutOfMemory
authored and
marmbrus
committed
[SPARK-5296] [SQL] Add more filter types for data sources API

liancheng
authored and
marmbrus
committed
[SPARK-5833] [SQL] Adds REFRESH TABLE command

liancheng
authored and
marmbrus
committed
SPARK-5841: remove DiskBlockManager shutdown hook on stop

Matt Whelan
authored and
srowen
committed
SPARK-5357: Update commons-codec version to 1.10 (current)

Matt Whelan
authored and
srowen
committed
[SPARK-2313] Use socket to communicate GatewayServer port back to Python driver
JoshRosen
committed
HOTFIX: Break in Jekyll build from #4589
pwendell
committed
[SPARK-5746][SQL] Check invalid cases for the write path of data source API

yhuai
authored and
marmbrus
committed
[SPARK-5839][SQL]HiveMetastoreCatalog does not recognize table names and aliases of data source tables.

yhuai
authored and
marmbrus
committed
[SQL] Optimize arithmetic and predicate operators

kai
authored and
marmbrus
committed
[SPARK-4865][SQL]Include temporary tables in SHOW TABLES

yhuai
authored and
marmbrus
committed

Commits on Feb 17, 2015

SPARK-5848: tear down the ConsoleProgressBar timer

Matt Whelan
authored and
srowen
committed
[SPARK-5788] [PySpark] capture the exception in python write thread

Davies Liu
authored and
JoshRosen
committed
[SPARK-3340] Deprecate ADD_JARS and ADD_FILES

azagrebin
authored and
Andrew Or
committed
[SPARK-5849] Handle more types of invalid JSON requests in SubmitRestProtocolMessage.parseAction

JoshRosen
authored and
Andrew Or
committed
[SQL] Various DataFrame doc changes.
rxin
committed
[SPARK-5363] [PySpark] check ending mark in non-block way

Davies Liu
authored and
JoshRosen
committed
SPARK-5850: Remove experimental label for Scala 2.11 and FlumePollingStream
pwendell
committed
[SPARK-5853][SQL] Schema support in Row.
rxin
committed
[SPARK-5802][MLLIB] cache transformed data in glm
mengxr
committed
HOTFIX: Style issue causing build break
pwendell
committed
[SPARK-5826][Streaming] Fix Configuration not serializable problem

jerryshao
authored and
srowen
committed
Revert "[SPARK-5363] [PySpark] check ending mark in non-block way"
JoshRosen
committed
SPARK-5856: In Maven build script, launch Zinc with more memory
pwendell
committed
[SPARK-5858][MLLIB] Remove unnecessary first() call in GLM
mengxr
committed
[SPARK-5166][SPARK-5247][SPARK-5258][SQL] API Cleanup / Documentation
marmbrus
committed
[SPARK-5859] [PySpark] [SQL] fix DataFrame Python API

Davies Liu
authored and
marmbrus
committed
[SPARK-5778] throw if nonexistent metrics config file provided

ryan-williams
authored and
pwendell
committed
[SPARK-5661]function hasShutdownDeleteTachyonDir should use shutdownDeleteTachyonPaths to determine whether contains file

viper-kun
authored and
srowen
committed
SPARK-5841 [CORE] [HOTFIX] Memory leak in DiskBlockManager
srowen
committed
[SPARK-5864] [PySpark] support .jar as python package

Davies Liu
authored and
pwendell
committed
[Minor] fix typo in SQL document

CodingCat
authored and
marmbrus
committed
[SPARK-5862][SQL] Only transformUp the given plan once in HiveMetastoreCatalog

viirya
authored and
marmbrus
committed
[Minor][SQL] Use same function to check path parameter in JSONRelation

viirya
authored and
marmbrus
committed
[SQL] [Minor] Update the HiveContext Unittest

chenghao-intel
authored and
marmbrus
committed
[SPARK-5868][SQL] Fix python UDFs in HiveContext and checks in SQLContext
marmbrus
committed
[SPARK-4172] [PySpark] Progress API in Python

Davies Liu
authored and
JoshRosen
committed
[SPARK-5871] output explain in Python

Davies Liu
authored and
marmbrus
committed
[SPARK-5872] [SQL] create a sqlCtx in pyspark shell

Davies Liu
authored and
marmbrus
committed
[SPARK-5852][SQL]Fail to convert a newly created empty metastore parquet table to a data source parquet table.

yhuai
authored and
marmbrus
committed

Commits on Feb 18, 2015

[SPARK-5785] [PySpark] narrow dependency for cogroup/join in PySpark

Davies Liu
authored and
JoshRosen
committed
[SPARK-5811] Added documentation for maven coordinates and added Spark Packages support

brkyvz
authored and
pwendell
committed
[SPARK-4454] Properly synchronize accesses to DAGScheduler cacheLocs map

JoshRosen
authored and
pwendell
committed
[SPARK-4454] Revert getOrElse() cleanup in DAGScheduler.getCacheLocs()
JoshRosen
committed
Revert "Preparing development version 1.3.1-SNAPSHOT"
pwendell
committed
Revert "Preparing Spark release v1.3.0-snapshot1"
pwendell
committed
[SPARK-5875][SQL]logical.Project should not be resolved if it contains aggregates or generators

yhuai
authored and
marmbrus
committed
Preparing Spark release v1.3.0-rc1
pwendell
committed
Preparing development version 1.3.1-SNAPSHOT
pwendell
committed
[SPARK-5723][SQL]Change the default file format to Parquet for CTAS statements.

yhuai
authored and
marmbrus
committed
[SPARK-5731][Streaming][Test] Fix incorrect test in DirectKafkaStreamSuite
tdas
committed
[Minor] [SQL] Cleans up DataFrame variable names and toDF() calls

liancheng
authored and
rxin
committed
Avoid deprecation warnings in JDBCSuite.

tmyklebu
authored and
rxin
committed
[SPARK-5878] fix DataFrame.repartition() in Python

Davies Liu
authored and
rxin
committed
SPARK-4610 addendum: [Minor] [MLlib] Minor doc fix in GBT classification example

MechCoder
authored and
srowen
committed
SPARK-5669 [BUILD] [HOTFIX] Spark assembly includes incompatibly licensed libgfortran, libgcc code via JBLAS
srowen
committed
[SPARK-5519][MLLIB] add user guide with example code for fp-growth
mengxr
committed
[SPARK-5507] Added documentation for BlockMatrix

brkyvz
authored and
mengxr
committed
[SPARK-5840][SQL] HiveContext cannot be serialized due to tuple extraction

rxin
authored and
marmbrus
committed
[SPARK-5722] [SQL] [PySpark] infer int as LongType

Davies Liu
authored and
marmbrus
committed

Commits on Feb 19, 2015

[SPARK-5879][MLLIB] update PIC user guide and add a Java example
mengxr
committed
[SPARK-5846] Correctly set job description and pool for SQL jobs

kayousterhout
authored and
liancheng
committed
SPARK-5548: Fix for AkkaUtilsSuite failure - attempt 2

jacek-lewandowski
authored and
Andrew Or
committed
[SPARK-5816] Add huge compatibility warning in DriverWrapper
Andrew Or
committed
[SPARK-5423][Core] Cleanup resources in DiskMapIterator.finalize to ensure deleting the temp file

zsxwing
authored and
Ubuntu
committed
[SPARK-5825] [Spark Submit] Remove the double checking instance name when stopping the service

chenghao-intel
authored and
Andrew Or
committed
[SPARK-5904][SQL] DataFrame API fixes.

rxin
authored and
marmbrus
committed
[SPARK-5902] [ml] Made PipelineStage.transformSchema public instead of private to ml

jkbradley
authored and
mengxr
committed
[Spark-5889] Remove pid file after stopping service.

zhzhan
authored and
srowen
committed
SPARK-4682 [CORE] Consolidate various 'Clock' classes

srowen
authored and
Andrew Or
committed
SPARK-5570: No docs stating that `new SparkConf().set("spark.driver.memory", ...) will not work

Ilya Ganelin
authored and
Andrew Or
committed

Commits on Feb 20, 2015

Commits on Feb 21, 2015

SPARK-5841 [CORE] [HOTFIX 2] Memory leak in DiskBlockManager

nishkamravi2
authored and
Andrew Or
committed
[SPARK-5937][YARN] Fix ClientSuite to set YARN mode, so that the correct class is used in t...

harishreedharan
authored and
Andrew Or
committed

Commits on Feb 22, 2015

[DOCS] Fix typo in API for custom InputFormats based on the “new” MapReduce API

Alexander
authored and
srowen
committed
[DataFrame] [Typo] Fix the typo

chenghao-intel
authored and
srowen
committed
SPARK-5669 [BUILD] Reverse exclusion of JBLAS libs for 1.3
srowen
committed
Revert "[SPARK-4808] Removing minimum number of elements read before spill check"
Andrew Or
committed

Commits on Feb 23, 2015

[EXAMPLES] fix typo.

fukuo33
authored and
srowen
committed
[SPARK-5943][Streaming] Update the test to use new API to reduce the warning

jerryshao
authored and
srowen
committed
[SPARK-5939][MLLib] make FPGrowth example app take parameters

Jacky Li
authored and
mengxr
committed
[MLLIB] SPARK-5912 Programming guide for feature selection

avulanov
authored and
jkbradley
committed

Commits on Feb 24, 2015

Commits on Feb 25, 2015

[SPARK-5751] [SQL] [WIP] Revamped HiveThriftServer2Suite for robustness
liancheng
committed
[SPARK-5904][SQL] DataFrame Java API test suites.

rxin
authored and
marmbrus
committed
[SPARK-5985][SQL] DataFrame sortBy -> orderBy in Python.

rxin
authored and
marmbrus
committed
[SPARK-5993][Streaming][Build] Fix assembly jar location of kafka-assembly
tdas
committed
[SPARK-5286][SQL] SPARK-5286 followup

yhuai
authored and
marmbrus
committed
[SPARK-5994] [SQL] Python DataFrame documentation fixes

Davies Liu
authored and
marmbrus
committed
[SPARK-5996][SQL] Fix specialized outbound conversions

marmbrus
authored and
mengxr
committed
SPARK-5930 [DOCS] Documented default of spark.shuffle.io.retryWait is confusing

srowen
authored and
rxin
committed
[SPARK-1955][GraphX]: VertexRDD can incorrectly assume index sharing

Brennon York
authored and
ankurdave
committed
[SPARK-5982] Remove incorrect Local Read Time Metric
kayousterhout
committed
[SPARK-5944] [PySpark] fix version in Python API docs

Davies Liu
authored and
marmbrus
committed
[SPARK-6010] [SQL] Merging compatible Parquet schemas before computing splits

liancheng
authored and
marmbrus
committed
[SPARK-5999][SQL] Remove duplicate Literal matching block

viirya
authored and
marmbrus
committed
[SPARK-5926] [SQL] make DataFrame.explain leverage queryExecution.logical

yanboliang
authored and
marmbrus
committed

Commits on Feb 26, 2015

[SPARK-5974] [SPARK-5980] [mllib] [python] [docs] Update ML guide with save/load, Python GBT

jkbradley
authored and
mengxr
committed
[SPARK-5724] fix the misconfiguration in AkkaUtils

CodingCat
authored and
rxin
committed
[SPARK-1182][Docs] Sort the configuration parameters in configuration.md

Brennon York
authored and
rxin
committed
[SPARK-5976][MLLIB] Add partitioner to factors returned by ALS
mengxr
committed
[SPARK-6023][SQL] ParquetConversions fails to replace the destination MetastoreRelation of an InsertIntoTable node to ParquetRelation2

yhuai
authored and
liancheng
committed
[SPARK-6016][SQL] Cannot read the parquet table after overwriting the existing table when spark.sql.parquet.cacheMetadata=true

yhuai
authored and
liancheng
committed
[SPARK-6007][SQL] Add numRows param in DataFrame.show()

Jacky Li
authored and
rxin
committed
[SPARK-6015] fix links to source code in Python API docs

Davies Liu
authored and
rxin
committed
[SPARK-5363] Fix bug in PythonRDD: remove() inside iterator is not safe

Davies Liu
authored and
JoshRosen
committed
Modify default value description for spark.scheduler.minRegisteredResourcesRatio on docs.

li-zhihui
authored and
Andrew Or
committed
[SPARK-6027][SPARK-5546] Fixed --jar and --packages not working for KafkaUtils and improved error message

tdas
authored and
Andrew Or
committed
[SPARK-6018] [YARN] NoSuchMethodError in Spark app is swallowed by YARN AM

Cheolsoo Park
authored and
Andrew Or
committed
Add a note for context termination for History server on Yarn

moutai
authored and
Andrew Or
committed
[SPARK-5951][YARN] Remove unreachable driver memory properties in yarn client mode

mohitgoyal557
authored and
Andrew Or
committed

Commits on Feb 27, 2015

SPARK-4579 [WEBUI] Scheduling Delay appears negative

srowen
authored and
Andrew Or
committed
[SPARK-6037][SQL] Avoiding duplicate Parquet schema merging

viirya
authored and
liancheng
committed
[SPARK-6024][SQL] When a data source table has too many columns, it's schema cannot be stored in metastore.

yhuai
authored and
rxin
committed
SPARK-2168 [Spark core] Use relative URIs for the app links in the History Server.

elyast
authored and
Andrew Or
committed
fix spark-6033, clarify the spark.worker.cleanup behavior in standalone mode

许鹏
authored and
Andrew Or
committed
[SPARK-6058][Yarn] Log the user class exception in ApplicationMaster

zsxwing
authored and
srowen
committed
[SPARK-4587] [mllib] [docs] Fixed save,load calls in ML guide examples

jkbradley
authored and
mengxr
committed
[Streaming][Minor] Remove useless type signature of Java Kafka direct stream API

jerryshao
authored and
tdas
committed

Commits on Feb 28, 2015

Commits on Mar 1, 2015

SPARK-5984: Fix TimSort bug causes ArrayOutOfBoundsException

hotou
authored and
rxin
committed
[SPARK-6074] [sql] Package pyspark sql bindings.

Marcelo Vanzin
authored and
srowen
committed

Commits on Mar 2, 2015

[SPARK-6053][MLLIB] support save/load in PySpark's ALS
mengxr
committed
[SPARK-6083] [MLLib] [DOC] Make Python API example consistent in NaiveBayes

MechCoder
authored and
mengxr
committed
[Streaming][Minor]Fix some error docs in streaming examples

jerryshao
authored and
srowen
committed
[SPARK-6073][SQL] Need to refresh metastore cache after append data in CreateMetastoreDataSourceAsSelect

yhuai
authored and
liancheng
committed
[SPARK-6052][SQL]In JSON schema inference, we should always set containsNull of an ArrayType to true

yhuai
authored and
liancheng
committed
[SPARK-6111] Fixed usage string in documentation.

kennethmyers-ibm
authored and
srowen
committed
[SPARK-5741][SQL] Support the path contains comma in HiveContext

watermen
authored and
marmbrus
committed
[SPARK-6080] [PySpark] correct LogisticRegressionWithLBFGS regType parameter for pyspark

yanboliang
authored and
mengxr
committed
[DOCS] Refactored Dataframe join comment to use correct parameter ordering

Paul Power
authored and
marmbrus
committed
SPARK-5390 [DOCS] Encourage users to post on Stack Overflow in Community Docs
srowen
committed
[Minor] Fix doc typo for describing primitiveTerm effectiveness condition

viirya
authored and
marmbrus
committed
[SPARK-6040][SQL] Fix the percent bug in tablesample

watermen
authored and
marmbrus
committed
[SPARK-6050] [yarn] Relax matching of vcore count in received containers.

Marcelo Vanzin
authored and
tgravescs
committed

Commits on Mar 3, 2015

Commits on Mar 4, 2015

[SPARK-6141][MLlib] Upgrade Breeze from 0.10 to 0.11 to fix convergence bug
mengxr
committed
[SPARK-6136] [SQL] Removed JDBC integration tests which depends on docker-client
liancheng
committed
[SPARK-6134][SQL] Fix wrong datatype for casting FloatType and default LongType value in defaultPrimitive

viirya
authored and
liancheng
committed
[SPARK-6144] [core] Fix addFile when source files are on "hdfs:"

Marcelo Vanzin
authored and
Andrew Or
committed

Commits on Mar 5, 2015

Commits on Mar 21, 2015