Skip to content

Latest commit

 

History

History
60 lines (43 loc) · 1.46 KB

connectors.md

File metadata and controls

60 lines (43 loc) · 1.46 KB

Infra Connectors Design

Interfaces

Refer to src/infra/pi.rs for details:

graph
 Sourceable --get--> Source;
 Source --get--> Reader
 Sinkable --get--> Sink;
 Sink --get--> Writer;
 StorageConnector --impl--> Sourceable;
 StorageConnector --impl--> Sinkable;
 TransformationConnector --impl--> Sourceable;
 TransformationConnector --impl--> Sinkable;
 TransformationConnector --impl--> Executable;
 Executable --get--> executor;
Loading

Sink: sink is a file location/handler/… where you can get a Writer from it.

Source: source is a handler where you can get a Reader from it.

StorageConnector: Sinkable + Sourceable

TransformationConnector: Sinkable + Sourceable + Executable

Connectors

kv

RedisConnector: Sinkable + Sourceable

gdb

Neo4jConnector: TransformationConnector

Demonstration workflow using page_rank (Neo4j -> Row -> Redis)

graph
 Neo4j .- Neo4jConnector;

 Neo4jConnector --get--> PageRankExecutor;
 PageRankExecutor --exec--> Neo4jQuerySource;
 PageRankExecutor --impl--> GraphComputationExecutor;
 GraphComputationExecutor .-> Neo4jQuerySource;
 Neo4jQuerySource --get--> Reader["Reader<Row>"];
 Neo4jQuerySource --impl--> Source["Source<Row>"];
 Source .-> Reader;

 Redis .- RedisConnector;
 RedisConnector --get--> RedisRowSink;
 RedisRowSink --impl--> Sink["Sink<Row>"]
 RedisRowSink --get--> Writer["Writer<Row>"];
 Sink .-> Writer;

 Reader .-data.-> Writer;
Loading