Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add mongodb connector #61

Merged
merged 5 commits into from
Sep 7, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,13 @@ testbin/*
target/
# dep
cdk-go

# jetbrains
.idea

# go test resources
**/test/resources/*

# generated proto file
**/src/proto
**/*.pb.go
69 changes: 69 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# The Specification of Connector developing

## Project Layout
Each connector SHOULD follow the [templates](templates) directory's structure to make sure the necessary files will be added.

For the consistent developer's experience, each connector's README.md SHOULD be created according to [README-template.md](templates/README.md)

## Developer Experience

This section gives some restrictions in order to make all connectors have a consistent style whatever any aspect, which can
ensure developers have better experience in learning, understanding, and developing.

### Language
Because the connectors aim to be serverless application, So, each connector's programming SHOULD prefer to Golang in
order to meet the minimum package size and maximum speed of starting application, unless there is the special consideration,
like library and eco-system.


### Naming

#### Java
- package: `com.linkall.connector.{connector_name}.*`

#### Golang
- module: `github.com/linkall-labs/connector/{connector_name}`

### Error
TODO

### Log
TODO

### Testing
Because connectors really are stateless application, so the unit testing is pretty important to connectors. Thus, each
connector's ut coverage should be greater than 80%.

### Observability
TODO


## Configuration
each connector will have 2 config files:
- **config.json**: including all properties of connector needs, except the secret information.
- **secret.json**: any sensitive property。

## Deploy
each connector should provide 3 methods to run:
- **docker**: how to run connector in a docker engine.
- **k8s**: how to run connector in k8s cluster.

These already included in [templates](templates/README.md), whose 'how to use' section has been displayed it.

## How to create a new connector

### RDD

If you want to create a new connector, you MUST finish the `README.md` firstly. We call this the **RDD(README Drive Development)**.

The reason is that `README.md` is the first thing that the users will know what this connector is and how to use it.
There are many sections that users will care about in [README-template.md](templates/README.md), so we can think as a real
user and pay attention in details by writing readme doc.

When the readme doc is finished, the connector's design also almost be finished.

### Proposal

When you finished the `README.md`, you can create a PR to submit your file. if the README would be accepted, follow the
[developer instruction](#) to start developing if you want. If you don't want to implement it by yourself, the vance community
will make it implemented.
34 changes: 34 additions & 0 deletions build/go/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
FROM golang:1.18.5 as builder

ARG connector

COPY ./vance /build/vance
COPY ./cdk-go /build/cdk-go

RUN cd /build/vance/connectors/${connector} && \
go build -v -o /build/vance/bin/${connector} ./cmd/main.go

FROM centos:8.4.2105

ARG connector

WORKDIR /vance

COPY --from=builder /build/vance/bin/${connector} /vance/bin/${connector}
COPY --from=builder /build/vance/connectors/${connector}/run.sh /vance/run.sh

ENV CONNECTOR=${connector}
ENV EXECUTABLE_FILE=/vance/bin/${connector}
ENV CONNECTOR_HOME=/vance
ENV CONNECTOR_CONFIG=/vance/config/config.yml
ENV CONNECTOR_SECRET=/vance/secret/secert.yml
ENV CONNECTOR_SECRET_ENABLE=false

RUN echo '#!/bin/sh' >> /vance/run.sh
RUN echo $EXECUTABLE_FILE >> /vance/run.sh
RUN chmod a+x /vance/bin/${connector}
RUN chmod a+x /vance/run.sh

EXPOSE 8080

ENTRYPOINT ["/vance/run.sh"]
33 changes: 33 additions & 0 deletions build/java/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
FROM maven:3.8.6-jdk-11 as builder

ARG connector

COPY ./vance /build/vance
COPY ./cdk-java /build/cdk-java

WORKDIR /build/vance/connectors/${connector}

RUN apt-get -qq update
RUN apt-get -qq install libatomic1
RUN mvn clean package
RUN ls -alh target/*jar-with-dependencies.jar | awk '{system("cp " $9 " /build/executable.jar") }'

FROM openjdk:11

ARG connector
ARG version

COPY --from=builder /build/executable.jar /vance/${connector}/${version}.jar

ENV CONNECTOR=${connector}
ENV CONNECTOR_VERSION=${version}
ENV CONNECTOR_HOME=/vance
ENV CONNECTOR_CONFIG=/vance/config/config.yml
ENV CONNECTOR_SECRET=/vance/secret/secert.yml
ENV CONNECTOR_SECRET_ENABLE=false

RUN echo '#!/bin/sh' >> /vance/run.sh
RUN echo 'java -jar /vance/${CONNECTOR}/${CONNECTOR_VERSION}.jar' >> /vance/run.sh
RUN chmod a+x /vance/run.sh

ENTRYPOINT ["/vance/run.sh"]
9 changes: 9 additions & 0 deletions connectors/mongodb-sink/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
WORKDIR=$(shell pwd)
PROTO_ROOT=${WORKDIR}/../../schemas

generate-pb:
mkdir -p proto
protoc -I=${PROTO_ROOT} \
-I=${PROTO_ROOT}/thirds \
--go_out=paths=source_relative:proto \
${PROTO_ROOT}/database/mongodb.proto
Loading