Skip to content

Commit

Permalink
Publish Kafka Destination (#4900)
Browse files Browse the repository at this point in the history
Co-authored-by: Mario Molina <mmolimar@gmail.com>
  • Loading branch information
sherifnada and mmolimar authored Jul 22, 2021
1 parent fc3c692 commit a7b5085
Show file tree
Hide file tree
Showing 5 changed files with 78 additions and 0 deletions.
2 changes: 2 additions & 0 deletions airbyte-integrations/builds.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,8 @@
Google Cloud Storage (GCS) [![destination-gcs](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fdestination-s3%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/destination-gcs)

Google PubSub [![destination-pubsub](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fdestination-pubsub%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/destination-pubsub)

Kafka [![destination-kafka](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fdestination-kafka%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/destination-kafka)

Local CSV [![destination-csv](https://img.shields.io/endpoint?url=https%3A%2F%2Fdnsgjos7lj2fu.cloudfront.net%2Ftests%2Fsummary%2Fdestination-csv%2Fbadge.json)](https://dnsgjos7lj2fu.cloudfront.net/tests/summary/destination-csv)

Expand Down
68 changes: 68 additions & 0 deletions airbyte-integrations/connectors/destination-kafka/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Destination Kafka

This is the repository for the Kafka destination connector in Java.
For information about how to use this connector within Airbyte, see [the User Documentation](https://docs.airbyte.io/integrations/destinations/kafka).

## Local development

#### Building via Gradle
From the Airbyte repository root, run:
```
./gradlew :airbyte-integrations:connectors:destination-kafka:build
```

#### Create credentials
**If you are a community contributor**, generate the necessary credentials and place them in `secrets/config.json` conforming to the spec file in `src/main/resources/spec.json`.
Note that the `secrets` directory is git-ignored by default, so there is no danger of accidentally checking in sensitive information.

**If you are an Airbyte core member**, follow the [instructions](https://docs.airbyte.io/contributing-to-airbyte/building-new-connector#using-credentials-in-ci) to set up the credentials.

### Locally running the connector docker image

#### Build
Build the connector image via Gradle:
```
./gradlew :airbyte-integrations:connectors:destination-kafka:airbyteDocker
```
When building via Gradle, the docker image name and tag, respectively, are the values of the `io.airbyte.name` and `io.airbyte.version` `LABEL`s in
the Dockerfile.

#### Run
Then run any of the connector commands as follows:
```
docker run --rm airbyte/destination-kafka:dev spec
docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-kafka:dev check --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets airbyte/destination-kafka:dev discover --config /secrets/config.json
docker run --rm -v $(pwd)/secrets:/secrets -v $(pwd)/integration_tests:/integration_tests airbyte/destination-kafka:dev read --config /secrets/config.json --catalog /integration_tests/configured_catalog.json
```

## Testing
We use `JUnit` for Java tests.

### Unit and Integration Tests
Place unit tests under `src/test/io/airbyte/integrations/destinations/kafka`.

#### Acceptance Tests
Airbyte has a standard test suite that all destination connectors must pass. Implement the `TODO`s in
`src/test-integration/java/io/airbyte/integrations/destinations/kafkaDestinationAcceptanceTest.java`.

### Using gradle to run tests
All commands should be run from airbyte project root.
To run unit tests:
```
./gradlew :airbyte-integrations:connectors:destination-kafka:unitTest
```
To run acceptance and custom integration tests:
```
./gradlew :airbyte-integrations:connectors:destination-kafka:integrationTest
```

## Dependency Management

### Publishing a new version of the connector
You've checked out the repo, implemented a million dollar feature, and you're ready to share your changes with the world. Now what?
1. Make sure your changes are passing unit and integration tests.
1. Bump the connector version in `Dockerfile` -- just increment the value of the `LABEL io.airbyte.version` appropriately (we use [SemVer](https://semver.org/)).
1. Create a Pull Request.
1. Pat yourself on the back for being an awesome contributor.
1. Someone from Airbyte will take a look at your PR and iterate with you to merge it into master.
1 change: 1 addition & 0 deletions docs/SUMMARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -102,6 +102,7 @@
* [BigQuery](integrations/destinations/bigquery.md)
* [Google Cloud Storage (GCS)](integrations/destinations/gcs.md)
* [Google PubSub](integrations/destinations/pubsub.md)
* [Kafka](integrations/destinations/kafka.md)
* [Local CSV](integrations/destinations/local-csv.md)
* [Local JSON](integrations/destinations/local-json.md)
* [MeiliSearch](integrations/destinations/meilisearch.md)
Expand Down
1 change: 1 addition & 0 deletions docs/integrations/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ Airbyte uses a grading system for connectors to help users understand what to ex
|[BigQuery](./destinations/bigquery.md)| Certified |
|[Google Cloud Storage (GCS)](./destinations/s3.md)| Alpha |
|[Google Pubsub](./destinations/pubsub.md)| Alpha |
|[Kafka](./destinations/kafka.md)| Alpha |
|[Local CSV](./destinations/local-csv.md)| Certified |
|[Local JSON](./destinations/local-json.md)| Certified |
|[MeiliSearch](./destinations/meilisearch.md)| Beta |
Expand Down
6 changes: 6 additions & 0 deletions docs/integrations/destinations/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,3 +109,9 @@ following parameters on the Kafka destination (though many of these are optional
More info about this can be found in the [Kafka producer configs documentation site](https://kafka.apache.org/documentation/#producerconfigs).

*NOTE*: Some configurations for SSL are not available yet.

## Changelog

| Version | Date | Pull Request | Subject |
| :------ | :-------- | :----- | :------ |
| 0.1.0 | 2021-07-21 | [3746](https://github.com/airbytehq/airbyte/pull/3746) | Initial Release |

0 comments on commit a7b5085

Please sign in to comment.