Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎉 Containerize connector code generator #3763

Merged
merged 16 commits into from
Jun 8, 2021
Merged

🎉 Containerize connector code generator #3763

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
3cc3775
Wrap npm generate script with docker and create generate.sh script
TymoshokDmytro May 28, 2021
fa80b86
replace npm run generate -> ./generate.sh
TymoshokDmytro May 31, 2021
55736f4
Merge branch 'master' of https://github.com/airbytehq/airbyte into dt…
TymoshokDmytro May 31, 2021
9591835
Merge branch 'master' into dtymoshok/#3558-containerize-connector-cod…
subodh1810 May 31, 2021
d5d0e2a
Merge branch 'master' of github.com:airbytehq/airbyte into dtymoshok/…
TymoshokDmytro May 31, 2021
f582c85
Make generator works from both npm run generate or from generate.sh …
TymoshokDmytro May 31, 2021
5afa02b
Merge branch 'dtymoshok/#3558-containerize-connector-code-generator' …
TymoshokDmytro May 31, 2021
f2809d1
Update generate.sh to check the connectors folder before coping the o…
TymoshokDmytro Jun 1, 2021
0739681
Add some comments to generate.sh script
TymoshokDmytro Jun 1, 2021
18cbedc
Move generate.sh script to the airbyte root and update READMEs
TymoshokDmytro Jun 2, 2021
0ef00c5
Add newlines to the end of the dockerfile and script
TymoshokDmytro Jun 3, 2021
8479592
Fix path outputs in generator for better user experience
TymoshokDmytro Jun 3, 2021
cae9c49
Minor adding
TymoshokDmytro Jun 3, 2021
1787680
Merge branch 'master' of github.com:airbytehq/airbyte into dtymoshok/…
TymoshokDmytro Jun 7, 2021
dbfc657
Updates to have possibility to generate and modify files all over air…
TymoshokDmytro Jun 7, 2021
7228708
Move generator.sh script to airbyte-integrations/connector-templates/…
TymoshokDmytro Jun 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions airbyte-cdk/python/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,7 @@ This document is a general introduction to the CDK. Readers should have basic fa
Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run
```
cd airbyte-integrations/connector-templates/generator
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make it run from the project root instead?

./airbyte-integrations/connector-templates/generator/generate.sh

We've had this convention for almost all our scripts.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I move the generate.sh to the root of the project, update the script paths, and READMEs. So now it can be started from the airbyte root.

npm install
npm run generate
./generate.sh
```

then follow the interactive prompt. Next, find all `TODO`s in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.
Expand Down
3 changes: 1 addition & 2 deletions airbyte-integrations/bases/base-python/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,7 @@ This document is a general introduction to the CDK. Readers should have basic fa
Generate an empty connector using the code generator. First clone the Airbyte repository then from the repository root run
```
cd airbyte-integrations/connector-templates/generator
npm install
npm run generate
./generate.sh
```

then follow the interactive prompt. Next, find all `TODO`s in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.
Expand Down
12 changes: 12 additions & 0 deletions airbyte-integrations/connector-templates/generator/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
FROM node:14-alpine

ARG UID
ARG GID
ENV ENV_UID $UID
ENV ENV_GID $GID

RUN mkdir -p /airbyte
WORKDIR /airbyte/airbyte-integrations/connector-templates/generator

RUN npm install npm@7 --silent --no-update-notifier
CMD npm run generate && chown -R $ENV_UID:$ENV_GID /airbyte/*
17 changes: 17 additions & 0 deletions airbyte-integrations/connector-templates/generator/generate.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/usr/bin/env sh

# Remove container if already exist
docker container rm -f airbyte-connector-bootstrap >/dev/null 2>&1

# Build image for container from Dockerfile
# Specify the host system user UID and GID to chown the generated files to host system user.
# This is done because all generated files in container with mounted folders has root owner
docker build --build-arg UID="$(id -u)" --build-arg GID="$(id -g)" . -t airbyte/connector-bootstrap

# Run the container and mount the airbyte folder
docker run -it --name airbyte-connector-bootstrap -v "$(pwd)/../../../.":/airbyte airbyte/connector-bootstrap

# Remove container after coping files
docker container rm -f airbyte-connector-bootstrap >/dev/null 2>&1

exit 0
2 changes: 1 addition & 1 deletion airbyte-integrations/connector-templates/generator/plopfile.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ const getSuccessMessage = function(connectorName, outputPath, additionalMessage)

Success!

Your ${connectorName} connector has been created at ${path.resolve(outputPath)}.
Your ${connectorName} connector has been created at .${path.resolve(outputPath)}.

Follow the TODOs in the generated module to implement your connector.

Expand Down
3 changes: 1 addition & 2 deletions docs/contributing-to-airbyte/building-new-connector/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,8 +43,7 @@ Run the interactive generator:

```text
cd airbyte-integrations/connector-templates/generator
npm install
npm run generate
./generate.sh
```

and choose the relevant template. This will generate a new connector in the `airbyte-integrations/connectors/<your-connector>` directory.
Expand Down
3 changes: 1 addition & 2 deletions docs/contributing-to-airbyte/python/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,7 @@ Generate an empty connector using the code generator. First clone the Airbyte re

```text
cd airbyte-integrations/connector-templates/generator
npm install
npm run generate
./generate.sh
```

then follow the interactive prompt. Next, find all `TODO`s in the generated project directory -- they're accompanied by lots of comments explaining what you'll need to do in order to implement your connector. Upon completing all TODOs properly, you should have a functioning connector.
Expand Down