Skip to content

[SPARK-27299][GRAPH][WIP] Spark Graph API design proposal #24297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
s1ck wants to merge 124 commits into from
Closed

[SPARK-27299][GRAPH][WIP] Spark Graph API design proposal #24297

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
124 commits
Select commit Hold shift + click to select a range
c93f1a1
Add initial set of classes to get Cypher engine running
s1ck Feb 19, 2019
3d5f699
Add initial test infrastructure
s1ck Feb 19, 2019
3c58c2f
Setup TCK testing infrastructure
s1ck Feb 19, 2019
9091548
Get all TCK scenarios to pass
s1ck Feb 19, 2019
7abaddf
Remove unnecessary cast
s1ck Feb 19, 2019
3f0acbf
Remove unnecessary Neo4j specific dependencies
s1ck Feb 20, 2019
9541508
Exclude Antlr from spark-sql deps
s1ck Feb 20, 2019
14bdada
Add spark-graph-api
s1ck Feb 20, 2019
bfed6b5
Add adapters for spark-graph-cypher and example
s1ck Feb 20, 2019
3f54f6c
Use named arguments in example for better showcasing
s1ck Feb 20, 2019
852e990
Adapt to latest OKAPI changes
s1ck Feb 22, 2019
fdef716
Support for turning Cypher results into nodes/rels for property graph
s1ck Feb 22, 2019
542a3e7
Fix round trip example by replacing dots when importing
pstutz Feb 25, 2019
7bed1f9
Adapt to changes in okapi-relational
pstutz Feb 25, 2019
3a66462
Clearer semantics for column access via header
pstutz Feb 25, 2019
f347238
Adapt to entity mapping changes in okapi
s1ck Feb 26, 2019
417dc8e
Enable round trip example
s1ck Feb 26, 2019
ee968ef
Rename Node/RelationshipDataFrame to Node/RelationshipFrame
s1ck Feb 26, 2019
fd8d968
Adapt CypherResult to GraphElementFrame name changes
s1ck Feb 26, 2019
9c47864
Add constructor for GraphElementFrames that infers property mappings
pstutz Feb 26, 2019
7d7be32
Adapt branch-2.2 graph work to changes in branch-2.4
pstutz Feb 26, 2019
a171e4d
Adapt blacklist due to changes in Spark 2.4
pstutz Feb 27, 2019
ed7d8cc
Add documentation for spark-graph-api
s1ck Feb 27, 2019
c1d72f6
Add default parameter for relationships in `createGraph`
pstutz Feb 27, 2019
5797400
Use Spark 2.4 features where possible and remove/inline helpers
s1ck Feb 27, 2019
52c3d5c
Use CypherResult to create new Property Graph
s1ck Feb 27, 2019
cb3b5b2
Rename cypherEngine to cypherSession in tests
s1ck Mar 1, 2019
27abd67
Adapt versions to migrate graph work to Spark 3.0 branch
pstutz Mar 4, 2019
402e888
Fix to get Spark master branch to compile
pstutz Mar 4, 2019
0dddb42
Add methods to read/write property graphs
pstutz Mar 4, 2019
763e99c
Use collect instead of filter + map
pstutz Mar 4, 2019
cd2f730
Simplify filesystem resource handling
pstutz Mar 4, 2019
1884a7d
Improve code reuse in SparkCypherSession
pstutz Mar 4, 2019
1bfb540
Throw `UnsupportedOperationException` when `entityTables` is called
pstutz Mar 4, 2019
42f7896
Hide internals of `SparkCypherSession` outside of `spark` package
pstutz Mar 4, 2019
d7fa40e
Automatically convert ID columns that don't have BinaryType
pstutz Mar 5, 2019
af840de
Improve documentation
pstutz Mar 5, 2019
77b7be0
Remove workaround for `distinct` bug
pstutz Mar 6, 2019
b563423
Extend Cast expression with support for casting integer types to binary
s1ck Mar 5, 2019
4ed8bf8
Use new cast(BinaryType) feature for Byte|Short|Integer|Long conversion
pstutz Mar 6, 2019
0b4c90a
Remove id function in test
pstutz Mar 8, 2019
46537bc
Update to latest OKAPI release
DarthMax Mar 12, 2019
bfbd023
Expose schema information directly on PropertyGraph
s1ck Mar 14, 2019
e283ab9
Fix Scala warning
pstutz Mar 14, 2019
b3f5685
Minor cleanups
pstutz Mar 14, 2019
c21fbb1
Use column name conventions for node/relationship DataFrame
s1ck Mar 22, 2019
415a577
Update documentation for PropertyGraph.[nodes/relationships]
s1ck Mar 22, 2019
33fe7fd
Introduce CypherResult.graph to allow for round trip examples
s1ck Mar 25, 2019
80a456d
Add PoC for graph algorithm support using GraphX PageRank
s1ck Mar 25, 2019
119cf08
Bring back column naming convention example
s1ck Mar 25, 2019
77ab7b4
Add example for retaining user ids
s1ck Mar 25, 2019
0731b0c
Simplify save/load with default parquet format
s1ck Mar 25, 2019
d6f574f
Remove result -> graph conversion
s1ck Mar 29, 2019
200d957
Adapt implementation to okapi-relational 0.3.1-SNAPSHOT
s1ck Mar 29, 2019
59c1ac9
Move SparkCypherFunctions to impl root package
s1ck Mar 29, 2019
f80d489
Update TCK blacklist
s1ck Mar 29, 2019
db4498b
Move nullSafeConversion to ExprConversions
s1ck Mar 29, 2019
cd9d712
Remove references to CAPS
s1ck Mar 29, 2019
fcb3aeb
Close Source in TCK tests
s1ck Mar 29, 2019
54e7014
Extract schema methods into separate PropertyGraphType trait
s1ck Apr 3, 2019
34b595f
Update documentation
s1ck Apr 3, 2019
3af683a
Rename spark-graph-cypher to spark-cypher
s1ck Apr 3, 2019
13e8889
Rename org.apache.spark.graph.cypher to org.apache.spark.cypher
s1ck Apr 3, 2019
cc9c617
Document and check node/rel frame uniqueness on graph creation
s1ck Apr 3, 2019
0ca9c00
Add default implementation for graph creation from two DFs to API
s1ck Apr 3, 2019
73156e8
Add TODOs
s1ck Apr 3, 2019
08d81ea
Add empty label set to wide table extraction
s1ck Apr 4, 2019
ba75f55
Add examples aligned with design document
s1ck Apr 4, 2019
5ee6d04
Add example for custom column mapping
s1ck Apr 4, 2019
b2e9e93
Clean up examples
s1ck Apr 4, 2019
557715e
Add license headers
s1ck Apr 7, 2019
4abd883
Bump okapi dependencies to 0.3.2
s1ck Apr 15, 2019
b69a416
Make CypherSession.save public
s1ck Apr 18, 2019
687980d
Bump okapi version to 0.4.0
s1ck May 21, 2019
8e7ef83
Add PropertyGraphTest
s1ck Jun 12, 2019
c7608fa
Bump okapi-shade 0.4.2
s1ck Jun 12, 2019
9cb99d6
Adapt GraphElementFrame API to latest API changes
s1ck Jun 12, 2019
f985978
Add test for reading/writing Property Graphs
s1ck Jun 24, 2019
52edea5
Adapt to changes in SPIP PR
s1ck Jun 28, 2019
23242cd
Adapt to changes in https://github.com/apache/spark/pull/24851
s1ck Jul 24, 2019
bee92a3
Warn if label columns > 5 and throw if label columns > 10
s1ck Jul 25, 2019
5b7e5e1
Add note and fix indentation
s1ck Jul 25, 2019
7cfe827
Add inline doc that explains reasoning behind having CypherResult
s1ck Jul 25, 2019
777c652
Add Java test for PropertyGraph creation
s1ck Jul 25, 2019
b368686
Add doc to CypherSession companion object
s1ck Jul 25, 2019
a1200c2
Update documentation for CypherSession trait
s1ck Jul 25, 2019
ca950a2
Add reference to Neo4j Cypher manual on CypherSession::cypher methods
s1ck Jul 25, 2019
82e5ad7
Add reference to Neo4j Cypher parameter docs on CypherSession::cypher
s1ck Jul 25, 2019
eb6985b
Add parameter example to CypherSession::cypher methods
s1ck Jul 25, 2019
671e680
Accept Arrays instead of Seq of Node-/RelationshipFrames
s1ck Jul 25, 2019
8fa15c4
Change from case to if statement to improve readability
s1ck Jul 25, 2019
c77f399
Validate that label columns have BooleanType
s1ck Jul 25, 2019
4444260
Make GraphElementFrame sealed
s1ck Jul 25, 2019
e63c0cd
Add link to Property Graph Model to the class doc
s1ck Jul 26, 2019
0a938d4
Change PropertyGraph::labelSet to accept Array instead of Set
s1ck Jul 26, 2019
ea2bf1b
Sync docs with API PR
s1ck Jul 26, 2019
714d149
Fix modules in parent pom
s1ck Oct 8, 2019
94d7355
Restructure save/load API to match DataFrameWriter/Reader
Mats-SX Oct 8, 2019
8ac4c3f
Replace NodeFrame object with NodeFrameBuilder
s1ck Oct 9, 2019
5b2d08a
Replace RelationshipFrame object with RelationshipFrameBuilder
s1ck Oct 9, 2019
ccb5da4
Address review comments
s1ck Oct 10, 2019
68c058a
Extract ElementFrame classes and builders into separate files
s1ck Oct 10, 2019
f7adb94
Use Dataset[Row] over DataFrame
s1ck Oct 11, 2019
0c18e7e
Change indentation to 2 in JavaPropertyGraphSuite
s1ck Oct 11, 2019
1792d6c
Add clarifying comment on abstract methods in test suites
s1ck Oct 11, 2019
624a77a
Refactor ElementFrame conversion methods into CypherSession object
s1ck Oct 12, 2019
6615018
Use Dataset[Row] over DataFrame
Mats-SX Oct 11, 2019
fa1c76e
Renamings
dongjoon-hyun Oct 13, 2019
9114585
Address review comments
s1ck Oct 14, 2019
c337f6c
Address review comments
s1ck Oct 14, 2019
47948cd
Add a simple spark-graph example
FlorentinD Oct 2, 2019
17b6214
Add CompleteGraphAPIExample
FlorentinD Oct 4, 2019
a7be704
Merge pull request #1 from knutwalker/spark_graph_examples
FlorentinD Oct 11, 2019
b8018bb
Add pauses in complete graph API example
FlorentinD Oct 8, 2019
ae4d457
Rework complete graph API example to movies dataset
FlorentinD Oct 8, 2019
793086b
Add example with more advanced queries
FlorentinD Oct 8, 2019
5f59d28
Get examples to work from the IDE
neo-walker Oct 11, 2019
f656a08
Merge pull request #2 from FlorentinD/graph-examples
Mats-SX Oct 15, 2019
31ce35d
Adapt examples post rebase
Mats-SX Oct 15, 2019
80f9ce6
Rename and document type column validation
s1ck Oct 15, 2019
cb8ccdc
Inline TODO for relationship validation
s1ck Oct 15, 2019
314ea23
Refactor GraphElementDataset builders to element objects
s1ck Oct 15, 2019
3420b6c
Beautify example code
Mats-SX Oct 16, 2019
3786c73
Beautify example code
Mats-SX Oct 17, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions dev/.rat-excludes
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,4 @@ announce.tmpl
vote.tmpl
SessionManager.java
SessionHandler.java
.*blacklist
177 changes: 163 additions & 14 deletions examples/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,48 +51,72 @@
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-graphx_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-10_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-catalyst_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql-kafka-0-10_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-graph-api_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-cypher_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-math3</artifactId>
<scope>provided</scope>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.scalacheck</groupId>
Expand All @@ -102,17 +126,142 @@
<dependency>
<groupId>org.scala-lang</groupId>
<artifactId>scala-library</artifactId>
<scope>provided</scope>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.github.scopt</groupId>
<artifactId>scopt_${scala.binary.version}</artifactId>
<version>3.7.1</version>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>${hadoop.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-http</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-continuation</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlet</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-servlets</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-proxy</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-client</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-util</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-security</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-plus</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.eclipse.jetty</groupId>
<artifactId>jetty-server</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.jpmml</groupId>
<artifactId>pmml-model</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-column</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-hadoop</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.parquet</groupId>
<artifactId>parquet-avro</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>${hive.parquet.group}</groupId>
<artifactId>parquet-hadoop-bundle</artifactId>
<scope>provided</scope>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-annotations</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>com.github.scopt</groupId>
<artifactId>scopt_${scala.binary.version}</artifactId>
<version>3.7.0</version>
</dependency>
</dependencies>

Expand Down Expand Up @@ -151,7 +300,7 @@
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kinesis-asl_${scala.binary.version}</artifactId>
<version>${project.version}</version>
<scope>provided</scope>
<scope>compile</scope>
</dependency>
</dependencies>
</profile>
Expand Down
Loading