🎉 New destination: ScyllaDB

airbyte-config/init/src/main/resources/config/STANDARD_DESTINATION_DEFINITION/9255b7de-f778-4087-8973-e17d7898c43d.json

@@ -0,0 +1,7 @@
+{
+  "destinationDefinitionId": "9255b7de-f778-4087-8973-e17d7898c43d",
+  "name": "Scylla",
+  "dockerRepository": "airbyte/destination-scylla",
+  "dockerImageTag": "0.1.0",
+  "documentationUrl": "https://docs.airbyte.io/integrations/destinations/scylla"
+}

airbyte-integrations/connectors/destination-scylla/.dockerignore

@@ -0,0 +1,3 @@
+*
+!Dockerfile
+!build

airbyte-integrations/connectors/destination-scylla/Dockerfile

@@ -0,0 +1,11 @@
+FROM airbyte/integration-base-java:dev
+
+WORKDIR /airbyte
+ENV APPLICATION destination-scylla
+
+COPY build/distributions/${APPLICATION}*.tar ${APPLICATION}.tar
+
+RUN tar xf ${APPLICATION}.tar --strip-components=1
+
+LABEL io.airbyte.version=0.1.0
+LABEL io.airbyte.name=airbyte/destination-scylla

airbyte-integrations/connectors/destination-scylla/README.md

@@ -0,0 +1,68 @@
+# Destination Scylla
+
+This is the repository for the Scylla destination connector in Java.
+For information about how to use this connector within Airbyte, see [the User Documentation](https://docs.airbyte.io/integrations/destinations/scylla).
+
+## Local development
+
+#### Building via Gradle
+From the Airbyte repository root, run:
+```
+./gradlew :airbyte-integrations:connectors:destination-scylla:build
+```
+
+#### Create credentials
+**If you are a community contributor**, generate the necessary credentials and place them in `secrets/config.json` conforming to the spec file in `src/main/resources/spec.json`.
+Note that the `secrets` directory is git-ignored by default, so there is no danger of accidentally checking in sensitive information.
+
+**If you are an Airbyte core member**, follow the [instructions](https://docs.airbyte.io/connector-development#using-credentials-in-ci) to set up the credentials.
+
+### Locally running the connector docker image
+
+#### Build
+Build the connector image via Gradle:
+```
+./gradlew :airbyte-integrations:connectors:destination-scylla:airbyteDocker
+```
+When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
+the Dockerfile.
+
+#### Run
+Then run any of the connector commands as follows:
+```
+docker run --rm airbyte/destination-scylla:dev spec
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-scylla:dev check --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-scylla:dev discover --config /secrets/config.json
+docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/destination-scylla:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
+```
+
+## Testing
+We use `JUnit` for Java tests.
+
+### Unit and Integration Tests
+Place unit tests under `src/test/io/airbyte/integrations/destinations/scylla`.
+
+#### Acceptance Tests
+Airbyte has a standard test suite that all destination connectors must pass. Implement the `TODO`s in
+`src/test-integration/java/io/airbyte/integrations/destinations/scyllaDestinationAcceptanceTest.java`.
+
+### Using gradle to run tests
+All commands should be run from airbyte project root.
+To run unit tests:
+```
+./gradlew :airbyte-integrations:connectors:destination-scylla:unitTest
+```
+To run acceptance and custom integration tests:
+```
+./gradlew :airbyte-integrations:connectors:destination-scylla:integrationTest
+```
+
+## Dependency Management
+
+### Publishing a new version of the connector
+You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
+1. Make sure your changes are passing unit and integration tests.
+1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
+1. Create a Pull Request.
+1. Pat yourself on the back for being an awesome contributor.
+1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.

airbyte-integrations/connectors/destination-scylla/bootstrap.md

@@ -0,0 +1,32 @@
+# Scylla Destination
+
+Scylla is an open-source distributed NoSQL wide-column data store designed to handle large amounts of data across many
+commodity servers, providing high availability with no single point of failure. It is designed to be compatible with
+Apache Cassandra while achieving significantly higher throughputs and lower latencies. It supports the same protocols as
+Cassandra (CQL and Thrift) and the same file formats (SSTable)
+
+The data is structured in keyspaces and tables and is partitioned and replicated across different nodes in the
+cluster.  
+[Read more about Scylla](https://www.scylladb.com/)
+
+This connector maps an incoming `stream` to a Scylla `table` and a `namespace` to a Scylla`keyspace`.  
+When using destination sync mode `append` and `append_dedup`, an `insert` operation is performed against an existing
+Scylla table.  
+When using `overwrite`, the records are first placed in a temp table. When all the messages have been received the data
+is copied to the final table which is first truncated and the temp table is deleted.
+
+The Implementation uses the [Scylla](https://github.com/scylladb/java-driver/) driver in order to access
+Scylla. [ScyllaCqlProvider](./src/main/java/io/airbyte/integrations/destination/scylla/ScyllaCqlProvider.java)
+handles the communication with the Scylla cluster and internally it uses
+the [ScyllaSessionPool](./src/main/java/io/airbyte/integrations/destination/scylla/ScyllaSessionPool.java) to retrieve a
+session to the cluster.
+
+The [ScyllaMessageConsumer](./src/main/java/io/airbyte/integrations/destination/scylla/ScyllaMessageConsumer.java)
+class contains the logic for handling airbyte messages, events and copying data between tables.
+
+## Development
+
+See the [ScyllaCqlProvider](./src/main/java/io/airbyte/integrations/destination/scylla/ScyllaCqlProvider.java)
+class on how to use the Scylla driver.
+
+[Scylla driver docs.](https://docs.scylladb.com/using-scylla/drivers/cql-drivers/scylla-java-driver/)

airbyte-integrations/connectors/destination-scylla/build.gradle

@@ -0,0 +1,32 @@
+plugins {
+    id 'application'
+    id 'airbyte-docker'
+    id 'airbyte-integration-test-java'
+}
+
+application {
+    mainClass = 'io.airbyte.integrations.destination.scylla.ScyllaDestination'
+}
+
+def scyllaDriver = '3.10.2-scylla-1'
+def assertVersion = '3.21.0'
+def testContainersVersion = '1.16.2'
+
+dependencies {
+    implementation project(':airbyte-config:models')
+    implementation project(':airbyte-protocol:models')
+    implementation project(':airbyte-integrations:bases:base-java')
+    implementation files(project(':airbyte-integrations:bases:base-java').airbyteDocker.outputs)
+
+    implementation "com.scylladb:scylla-driver-core:${scyllaDriver}"
+
+    // https://mvnrepository.com/artifact/org.assertj/assertj-core
+    testImplementation "org.assertj:assertj-core:${assertVersion}"
+    // https://mvnrepository.com/artifact/org.testcontainers/testcontainers
+    testImplementation "org.testcontainers:testcontainers:${testContainersVersion}"
+
+
+
+    integrationTestJavaImplementation project(':airbyte-integrations:bases:standard-destination-test')
+    integrationTestJavaImplementation project(':airbyte-integrations:connectors:destination-scylla')
+}

airbyte-integrations/connectors/destination-scylla/docker-compose.yml

@@ -0,0 +1,20 @@
+version: '3'
+
+services:
+  scylla1:
+    image: scylladb/scylla
+    ports:
+      - "9042:9042"
+    container_name: scylla1
+    command: --smp 1
+
+#  uncomment if you want to run a cluster of scylladb nodes
+#  scylla2:
+#    image: scylladb/scylla
+#    container_name: scylla2
+#    command: --seeds=scylla1
+#
+#  scylla3:
+#    image: scylladb/scylla
+#    container_name: scylla3
+#    command: --seeds=scylla1

airbyte-integrations/connectors/destination-scylla/src/main/java/io/airbyte/integrations/destination/scylla/ScyllaConfig.java

@@ -0,0 +1,96 @@
+/*
+ * Copyright (c) 2021 Airbyte, Inc., all rights reserved.
+ */
+
+package io.airbyte.integrations.destination.scylla;
+
+import com.fasterxml.jackson.databind.JsonNode;
+import java.util.Objects;
+
+public class ScyllaConfig {
+
+  private final String keyspace;
+
+  private final String username;
+
+  private final String password;
+
+  private final String address;
+
+  private final int port;
+
+  private final int replication;
+
+  public ScyllaConfig(String keyspace, String username, String password, String address, int port, int replication) {
+    this.keyspace = keyspace;
+    this.username = username;
+    this.password = password;
+    this.address = address;
+    this.port = port;
+    this.replication = replication;
+  }
+
+  public ScyllaConfig(JsonNode jsonNode) {
+    this.keyspace = jsonNode.get("keyspace").asText();
+    this.username = jsonNode.get("username").asText();
+    this.password = jsonNode.get("password").asText();
+    this.address = jsonNode.get("address").asText();
+    this.port = jsonNode.get("port").asInt(9042);
+    this.replication = jsonNode.get("replication").asInt(1);
+  }
+
+  public String getKeyspace() {
+    return keyspace;
+  }
+
+  public String getUsername() {
+    return username;
+  }
+
+  public String getPassword() {
+    return password;
+  }
+
+  public String getAddress() {
+    return address;
+  }
+
+  public int getPort() {
+    return port;
+  }
+
+  public int getReplication() {
+    return replication;
+  }
+
+  @Override
+  public String toString() {
+    return "ScyllaConfig{" +
+        "keyspace='" + keyspace + '\'' +
+        ", username='" + username + '\'' +
+        ", password='" + password + '\'' +
+        ", address='" + address + '\'' +
+        ", port=" + port +
+        ", replication=" + replication +
+        '}';
+  }
+
+  @Override
+  public boolean equals(Object o) {
+    if (this == o) {
+      return true;
+    }
+    if (o == null || getClass() != o.getClass()) {
+      return false;
+    }
+    ScyllaConfig that = (ScyllaConfig) o;
+    return port == that.port && username.equals(that.username) && password.equals(that.password) &&
+        address.equals(that.address);
+  }
+
+  @Override
+  public int hashCode() {
+    return Objects.hash(username, password, address, port);
+  }
+
+}