Skip to content

Latest commit

 

History

History
33 lines (24 loc) · 1.5 KB

README.md

File metadata and controls

33 lines (24 loc) · 1.5 KB
Raw

Apache Beam pipelines for Scylla-RDF

Here you can find pipelines that manipulate RDF data in Scylla-RDF.

Bulk loading

For testing purpose or small files, you can run the pipeline with the embedded Apache Flink. To do so:

  1. set SCYLLA_RDF_STORAGE_HOSTS in docker-compose.yml to hostnames/IPs of the ScyllaDB separated by comma,
  2. put your RDF files in upload folder near the docker-compose.yml,
  3. run the pipeline 
    docker-compose up && docker-compose rm -f

For large files, you can deploy pipeline on Google Dataflow or Apache Flink. The other runners could be supported as well.

If you want to use Apache Flink, then:

  1. build the project with mvn build -DskipTests,
  2. and upload to the cluster, more about it read in the Deployment & Operations / Clusters & Deployment section in the Flink's docs,
  3. don't forget to use the same parameters as in run_bulkload_flink.sh.

If you wan to use Google Dataflow, then:

  1. also build the project with mvn build -DskipTests,
  2. change parameters in run_bulkload_dataflow.sh, so it'd run in your GCP project,
  3. run the script: 
    ./run_bulkload_dataflow.sh --source=gs://<your bucket>/folder1/*,gs://<your bucket>/folder2/*