[SPARK-27299][GRAPH][WIP] Spark Graph API design proposal #24297

s1ck · 2019-04-04T14:06:37Z

What changes were proposed in this pull request?

This PR demonstrates a prototypical implementation of the new Spark Graph API. The PR should mainly be used to discuss the API proposed in this GoogleDoc. This PR is not intended to be merged.

The PR introduces two modules:

spark-graph-api (containing the API to be discussed)
spark-cypher (a prototypical implementation of spark-graph-api)

Please use the PR and/or the GoogleDoc to comment the content of spark-graph-api. There will be follow-up PRs for spark-cypher.

How was this patch tested?

spark-cypher has been tested using the openCypher Technology Compatibility Kit

Contributors

Design, documentation and implementation have been a collaborative effort:

Co-Authored-By: Xiangrui Meng meng@databricks.com
Co-Authored-By: Max Kießling max.kiessling@neotechnology.com
Co-Authored-By: Mats Rydberg mats@neotechnology.com
Co-Authored-By: Philip Stutz philip.stutz@gmail.com
Co-Authored-By: Sören Reichardt soren.reichardt@neotechnology.com
Co-Authored-By: Jonatan Jäderberg jonatan.jaderberg@gmail.com
Co-Authored-By: Tobias Johansson tobias.johansson@neotechnology.com
Co-Authored-By: Alastair Green alastair.green@neo4j.com

dongjoon-hyun · 2019-04-04T19:16:03Z

ok to test

SparkQA · 2019-04-04T19:23:34Z

Test build #104302 has finished for PR 24297 at commit d678814.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-04-04T21:00:45Z

Although this is WIP, could you add LICENSE?

========================================================================
Running Apache RAT checks
========================================================================
Attempting to fetch rat
Could not find Apache license headers in the following files:
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/api/pom.xml
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/api/src/main/scala/org/apache/spark/graph/api/CypherResult.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/api/src/main/scala/org/apache/spark/graph/api/CypherSession.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/api/src/main/scala/org/apache/spark/graph/api/GraphElementFrame.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/api/src/main/scala/org/apache/spark/graph/api/PropertyGraph.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/api/src/main/scala/org/apache/spark/graph/api/PropertyGraphType.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/pom.xml
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/SparkCypherEntity.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/SparkCypherFunctions.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/SparkCypherRecords.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/SparkCypherResult.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/SparkCypherSession.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/SparkEntityTable.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/SparkGraphDirectoryStructure.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/SparkTable.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/adapters/MappingAdapter.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/adapters/RelationalGraphAdapter.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/adapters/SchemaAdapter.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/conversions/CypherValueEncoders.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/conversions/ExprConversions.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/conversions/RowConversion.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/conversions/StringEncodingUtilities.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/conversions/TemporalConversions.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/conversions/TypeConversions.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/io/ReadWriteGraph.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/udfs/TemporalUdfs.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/main/scala/org/apache/spark/cypher/util/HadoopFSUtils.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/test/resources/tck/failing_blacklist
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/test/resources/tck/failure_reporting_blacklist
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/test/resources/tck/temporal_blacklist
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/test/resources/tck/wont_fix_blacklist
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/test/scala/org/apache/spark/cypher/GraphExamplesSuite.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/test/scala/org/apache/spark/cypher/PropertyGraphReadWrite.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/test/scala/org/apache/spark/cypher/SharedCypherContext.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/test/scala/org/apache/spark/cypher/construction/ScanGraphFactory.scala
 !????? /home/jenkins/workspace/SparkPullRequestBuilder@2/graph/cypher/src/test/scala/org/apache/spark/cypher/tck/SparkCypherTckSuite.scala
[error] running /home/jenkins/workspace/SparkPullRequestBuilder@2/dev/check-license ; received return code 1

felixcheung · 2019-04-06T05:34:34Z

so this is adding a graph directory next to graphx?

s1ck · 2019-04-06T08:06:47Z

@felixcheung The SPIP involves Cypher querying and graph algorithms. The graphx directory will eventually be removed.

SparkQA · 2019-04-07T09:33:05Z

Test build #104353 has finished for PR 24297 at commit a779702.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-04-08T02:33:47Z

@s1ck . How do you build this PR in your environment? Jenkins seems to complain about Cyclic reference issue.

[error] Cyclic reference involving 
[error]    Project(id cypher, base: /home/jenkins/workspace/SparkPullRequestBuilder/graph/cypher, dependencies: List(ResolvedClasspathDependency(ProjectRef(file:/home/jenkins/workspace/SparkPullRequestBuilder/,core),None),

s1ck · 2019-04-08T07:14:03Z

@dongjoon-hyun I'm using Maven/Intellij to build and run. I used the same sbt.project.name for the two modules .. I guess this lead to the cyclic reference error.

SparkQA · 2019-04-08T07:17:56Z

Test build #104374 has finished for PR 24297 at commit 451e279.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

s1ck · 2019-04-08T07:33:35Z

@dongjoon-hyun The cyclic reference is fixed. I won't dive into fixing Scala style tests for now since this is WIP and open for discussion. Btw, do you know if there is a Spark code style xml for IntelliJ?

dongjoon-hyun · 2019-04-08T17:10:01Z

IntelliJ already understands checkstyle.xml. If you open the project with IntelliJ in Apache Spark project, you must see the red line warnings.

https://github.com/apache/spark/blob/master/dev/checkstyle.xml

The current warnings look like two types mostly. You can fix them easily.

Max line length (100).
Import ordering and grouping (java -> scala -> 3rd party -> org.apache.spark)

SparkQA · 2019-04-15T15:41:04Z

Test build #104592 has finished for PR 24297 at commit b6f26aa.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
implicit class RichExpression(expr: Expr)
implicit class TemporalExpression(val expr: Expr) extends AnyVal

dongjoon-hyun · 2019-04-15T15:43:31Z

Hi, @mengxr . Could you help this PR to pass the Jenkins, please?

mengxr · 2019-04-15T15:59:40Z

@dongjoon-hyun This PR is a prototype for API and design discussions. We should break it down into smaller ones after we reach an agreement on the API and design. I don't think we can merge this one directly.

dongjoon-hyun · 2019-04-15T16:52:39Z

Thank you, @mengxr . I see.

SparkQA · 2019-04-18T19:11:39Z

Test build #104719 has finished for PR 24297 at commit bfe66f8.

This patch fails Scala style tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-05-21T21:52:58Z

Test build #105643 has finished for PR 24297 at commit 9dde5f9.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
case class SchemaAdapter(schema: PropertyGraphSchema) extends PropertyGraphType

SparkQA · 2019-06-12T11:43:46Z

Test build #106419 has finished for PR 24297 at commit 94b01fd.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-06-12T11:51:51Z

Test build #106420 has finished for PR 24297 at commit d74df52.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds no public classes.

SparkQA · 2019-06-12T12:15:49Z

Test build #106421 has finished for PR 24297 at commit 4b6935b.

This patch fails Scala style tests.
This patch does not merge cleanly.
This patch adds the following public classes (experimental):
abstract class GraphElementFrame

## What changes were proposed in this pull request? This PR introduces the necessary Maven modules for the new [Spark Graph](https://issues.apache.org/jira/browse/SPARK-25994) feature for Spark 3.0. * `spark-graph` is a parent module that users depend on to get all graph functionalities (Cypher and Graph Algorithms) * `spark-graph-api` defines the [Property Graph API](https://docs.google.com/document/d/1Wxzghj0PvpOVu7XD1iA8uonRYhexwn18utdcTxtkxlI) that is being shared between Cypher and Algorithms * `spark-cypher` contains a Cypher query engine implementation Both, `spark-graph-api` and `spark-cypher` depend on Spark SQL. Note, that the Maven module for Graph Algorithms is not part of this PR and will be introduced in https://issues.apache.org/jira/browse/SPARK-27302 A PoC for a running Cypher implementation can be found in this WIP PR apache#24297 ## How was this patch tested? Pass the Jenkins with all profiles and manually build and check the followings. ``` $ ls assembly/target/scala-2.12/jars/spark-cypher* assembly/target/scala-2.12/jars/spark-cypher_2.12-3.0.0-SNAPSHOT.jar $ ls assembly/target/scala-2.12/jars/spark-graph* | grep -v graphx assembly/target/scala-2.12/jars/spark-graph-api_2.12-3.0.0-SNAPSHOT.jar assembly/target/scala-2.12/jars/spark-graph_2.12-3.0.0-SNAPSHOT.jar ``` Closes apache#24490 from s1ck/SPARK-27300. Lead-authored-by: Martin Junghanns <martin.junghanns@neotechnology.com> Co-authored-by: Max Kießling <max@kopfueber.org> Co-authored-by: Martin Junghanns <martin.junghanns@neo4j.com> Co-authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

kiszk · 2019-06-17T02:41:39Z

graph/cypher/src/main/scala/org/apache/spark/cypher/SparkCypherRecords.scala

+
+  override def toString: String = {
+    if (header.isEmpty) {
+      s"CAPSRecords.empty"


nit: s is not necessary

kiszk · 2019-06-17T02:46:46Z

graph/cypher/src/main/scala/org/apache/spark/cypher/SparkGraphDirectoryStructure.scala

+      noLabelNodeDirectoryName
+    } else {
+      // TODO: Find more elegant solution for encoding underline characters
+      seq.map(_.replace("_", "--UNDERLINE--")).mkString("_").encodeSpecialCharacters


nit: Is it better to define a variable to hold "--UNDERLINE-- and use it at all of the references for consistency?

kiszk · 2019-06-17T02:56:51Z

graph/cypher/src/main/scala/org/apache/spark/cypher/adapters/MappingAdapter.scala

+
+  implicit class RichRelationshipDataFrame(val relDf: RelationshipFrame) extends AnyVal {
+    def toRelationshipMapping: ElementMapping = RelationshipMappingBuilder
+        .on(relDf.idColumn)


kiszk · 2019-06-17T02:59:12Z

graph/cypher/src/main/scala/org/apache/spark/cypher/conversions/ExprConversions.scala

+      * values from the evaluated children.
+      */
+    def nullSafeConversion(expr: Expr)(withConvertedChildren: Seq[Column] => Column)
+      (implicit header: RecordHeader, df: DataFrame, parameters: CypherMap): Column = {


SparkQA · 2019-10-14T17:28:13Z

Test build #112050 has finished for PR 24297 at commit 9114585.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-14T22:12:29Z

Test build #112072 has finished for PR 24297 at commit 4669b16.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-15T08:20:43Z

Test build #112098 has finished for PR 24297 at commit c337f6c.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

Make example work when started from the IDE

Co-Authored-By: Sören Reichardt <soren.reichardt@neotechnology.com>

Graph examples

SparkQA · 2019-10-15T12:40:24Z

Test build #112105 has finished for PR 24297 at commit f656a08.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

Co-authored-by: Max Kießling <max.kiessling@neotechnology.com>

SparkQA · 2019-10-15T12:48:00Z

Test build #112106 has finished for PR 24297 at commit 31ce35d.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-10-15T15:42:54Z

Test build #112116 has finished for PR 24297 at commit 314ea23.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

Co-authored-by: Max Kießling <max.kiessling@neotechnology.com>

SparkQA · 2019-10-16T09:21:06Z

Test build #112157 has finished for PR 24297 at commit 3420b6c.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

Co-authored-by: Max Kießling <max.kiessling@neotechnology.com>

SparkQA · 2019-10-17T08:15:51Z

Test build #112211 has finished for PR 24297 at commit 3786c73.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds no public classes.

This was referenced Apr 18, 2019

Add draft for binary switch based on separate module opencypher/morpheus#896

Closed

Add draft for binary switch based on extended CAPSSession opencypher/morpheus#897

Closed

s1ck mentioned this pull request Apr 29, 2019

[SPARK-27300][GRAPH] Add Spark Graph modules and dependencies #24490

Closed

s1ck mentioned this pull request Jun 12, 2019

[SPARK-27303][GRAPH] Add Spark Graph API #24851

Closed

dongjoon-hyun added the GRAPH label Jun 14, 2019

kiszk reviewed Jun 17, 2019

View reviewed changes

Address review comments

c337f6c

s1ck force-pushed the 3.0-spark-graph-poc branch from 4669b16 to c337f6c Compare October 15, 2019 08:15

FlorentinD and others added 8 commits October 15, 2019 14:35

Add a simple spark-graph example

47948cd

Add CompleteGraphAPIExample

17b6214

Merge pull request #1 from knutwalker/spark_graph_examples

a7be704

Make example work when started from the IDE

Add pauses in complete graph API example

b8018bb

Rework complete graph API example to movies dataset

ae4d457

Add example with more advanced queries

793086b

Get examples to work from the IDE

5f59d28

Co-Authored-By: Sören Reichardt <soren.reichardt@neotechnology.com>

Merge pull request #2 from FlorentinD/graph-examples

f656a08

Graph examples

Adapt examples post rebase

31ce35d

Co-authored-by: Max Kießling <max.kiessling@neotechnology.com>

s1ck added 3 commits October 15, 2019 17:01

Rename and document type column validation

80f9ce6

Inline TODO for relationship validation

cb8ccdc

Refactor GraphElementDataset builders to element objects

314ea23

Beautify example code

3420b6c

Co-authored-by: Max Kießling <max.kiessling@neotechnology.com>

Beautify example code

3786c73

Co-authored-by: Max Kießling <max.kiessling@neotechnology.com>

s1ck closed this Dec 18, 2019

s1ck deleted the 3.0-spark-graph-poc branch December 18, 2019 09:16

s1ck restored the 3.0-spark-graph-poc branch December 18, 2019 09:16

s1ck deleted the 3.0-spark-graph-poc branch December 18, 2019 09:16

s1ck restored the 3.0-spark-graph-poc branch December 18, 2019 09:16

[SPARK-27299][GRAPH][WIP] Spark Graph API design proposal #24297

[SPARK-27299][GRAPH][WIP] Spark Graph API design proposal #24297

Uh oh!

Conversation

s1ck commented Apr 4, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

How was this patch tested?

Contributors

Uh oh!

dongjoon-hyun commented Apr 4, 2019

Uh oh!

SparkQA commented Apr 4, 2019

Uh oh!

dongjoon-hyun commented Apr 4, 2019

Uh oh!

felixcheung commented Apr 6, 2019

Uh oh!

s1ck commented Apr 6, 2019

Uh oh!

SparkQA commented Apr 7, 2019

Uh oh!

dongjoon-hyun commented Apr 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

s1ck commented Apr 8, 2019

Uh oh!

SparkQA commented Apr 8, 2019

Uh oh!

s1ck commented Apr 8, 2019

Uh oh!

dongjoon-hyun commented Apr 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SparkQA commented Apr 15, 2019

Uh oh!

dongjoon-hyun commented Apr 15, 2019

Uh oh!

mengxr commented Apr 15, 2019

Uh oh!

dongjoon-hyun commented Apr 15, 2019

Uh oh!

SparkQA commented Apr 18, 2019

Uh oh!

SparkQA commented May 21, 2019

Uh oh!

SparkQA commented Jun 12, 2019

Uh oh!

SparkQA commented Jun 12, 2019

Uh oh!

SparkQA commented Jun 12, 2019

Uh oh!

kiszk Jun 17, 2019

Choose a reason for hiding this comment

Uh oh!

kiszk Jun 17, 2019

Choose a reason for hiding this comment

Uh oh!

kiszk Jun 17, 2019

Choose a reason for hiding this comment

Uh oh!

kiszk Jun 17, 2019

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Oct 14, 2019

Uh oh!

SparkQA commented Oct 14, 2019

Uh oh!

SparkQA commented Oct 15, 2019

Uh oh!

SparkQA commented Oct 15, 2019

Uh oh!

SparkQA commented Oct 15, 2019

Uh oh!

SparkQA commented Oct 15, 2019

Uh oh!

SparkQA commented Oct 16, 2019

Uh oh!

SparkQA commented Oct 17, 2019

Uh oh!

s1ck commented Apr 4, 2019 •

edited

Loading

dongjoon-hyun commented Apr 8, 2019 •

edited

Loading

dongjoon-hyun commented Apr 8, 2019 •

edited

Loading