🎉 New Source: Hellobaton #8461

dluftspring · 2021-12-02T15:37:29Z

What

Creating a new python HTTP source for hellobaton based on this API spec

How

Using the python CDK to implement full refresh streams for every available endpoint listed in the redoc

🚨 User Impact 🚨

None

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

CLAassistant · 2021-12-02T15:37:39Z

All committers have signed the CLA.

marcosmarxm · 2021-12-02T21:13:10Z

thanks for the contribution @dluftspring we're going to review soon. I had create one issue to get credentials to run our integration tests #8476, is it possible for you share yours so we can anticipate the review?

dluftspring · 2021-12-02T21:53:58Z

@marcosmarxm let me ask for a set of dummy credentials from the baton folks. Mine are attached to real company data and i'd prefer not to expose them unless it's absolutely necessary. I'll reply to this conversation with more details

marcosmarxm · 2021-12-06T18:11:42Z

@marcosmarxm let me ask for a set of dummy credentials from the baton folks. Mine are attached to real company data and i'd prefer not to expose them unless it's absolutely necessary. I'll reply to this conversation with more details

Not a problem! So we need to wait the team to have a sandbox account.

marcosmarxm · 2021-12-09T18:21:56Z

@dluftspring can you share the unit test and integration test output?

dluftspring · 2021-12-11T15:19:35Z

@marcosmarxm which commands are used to run the unit and integration tests? Was searching through the docs but couldn't seem to find the right ones.

marcosmarxm · 2021-12-15T16:38:20Z

you can run ./gradlew airbyte-integrations:connectors:soruce-hellobaton:integrationTest

…est-pr-8461

marcosmarxm · 2021-12-15T17:15:31Z

@dluftspring make sure this commands works in your local machine, right now there are some failures building the docker container because of Flake/Black formatting files and the integration tests are not working.

dluftspring · 2021-12-16T13:29:47Z

Okay running the integration tests I can see two task failures. Here are the two failures

2021-12-16T08:28:15.961-0500 [DEBUG] [org.gradle.internal.operations.DefaultBuildOperationRunner] Build operation 'Run main tasks' completed
2021-12-16T08:28:15.965-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
2021-12-16T08:28:15.967-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build completed with 2 failures.
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 1: Task failed with an exception.
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] -----------
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Execution failed for task ':airbyte-integrations:connectors:source-hellobaton:flakeCheck'.
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Python call failed: .venv/bin/python -m flake8 . --config /Users/dluftspring/repos/airbyte/tools/python/.flake8
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Try:
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Run with --stacktrace option to get the stack trace.  Run with --scan to get full insights.
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] ==============================================================================
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] 2: Task failed with an exception.
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] -----------
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Execution failed for task ':tools:code-generator:airbyteDocker'.
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] > Process 'command '/Users/dluftspring/repos/airbyte/tools/bin/build_image.sh'' finished with non-zero exit value 1
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Try:
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Run with --stacktrace option to get the stack trace.  Run with --scan to get full insights.
2021-12-16T08:28:15.968-0500 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] ==============================================================================

Do you need the full stacktrace? Any places I should start looking for linting errors?

=========== Update =================

Found these let me push a commit and see if this at least gets rid of the linting failures

> Task :airbyte-integrations:connectors:source-hellobaton:flakeCheck FAILED
[python] .venv/bin/python -m flake8 . --config /Users/dluftspring/repos/airbyte/tools/python/.flake8
         ./source_hellobaton/streams.py:7:1: F401 'typing.List' imported but unused
         ./source_hellobaton/streams.py:7:1: F401 'typing.Tuple' imported but unused
         ./source_hellobaton/streams.py:11:1: F401 'airbyte_cdk.AirbyteLogger' imported but unused
         ./source_hellobaton/streams.py:12:1: F401 'airbyte_cdk.sources.streams.Stream' imported but unused
         ./source_hellobaton/streams.py:14:1: F401 'airbyte_cdk.sources.streams.http.auth.TokenAuthenticator' imported but unused
         ./source_hellobaton/source.py:6:1: F401 'abc.ABC' imported but unused
         ./source_hellobaton/source.py:7:1: F401 'typing.Iterable' imported but unused
         ./source_hellobaton/source.py:7:1: F401 'typing.MutableMapping' imported but unused
         ./source_hellobaton/source.py:7:1: F401 'typing.Optional' imported but unused
         ./source_hellobaton/source.py:14:1: F401 '.streams.HellobatonStream' imported but unused
         ./source_hellobaton/source.py:32:1: E302 expected 2 blank lines, found 1

dluftspring · 2021-12-16T14:11:24Z

Might need a little help chasing down the issues in build_image.sh

I think this is the relevant stacktrace

> Task :tools:code-generator:airbyteDocker FAILED
#5 8.038     Running setup.py install for typed-ast: started
#5 8.161     Running setup.py install for typed-ast: finished with status 'error'
#5 8.166     ERROR: Command errored out with exit status 1:
#5 8.166      command: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-t7ha8wv7/typed-ast_57d9f08187904d3c90acceded4abb219/setup.py'"'"'; __file__='"'"'/tmp/pip-install-t7ha8wv7/typed-ast_57d9f08187904d3c90acceded4abb219/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-rah6u5dw/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7m/typed-ast
#5 8.166          cwd: /tmp/pip-install-t7ha8wv7/typed-ast_57d9f08187904d3c90acceded4abb219/
#5 8.166     Complete output (23 lines):
#5 8.166     running install
#5 8.166     running build
#5 8.166     running build_py
#5 8.166     creating build
#5 8.166     creating build/lib.linux-aarch64-3.7
#5 8.166     creating build/lib.linux-aarch64-3.7/typed_ast
#5 8.166     copying typed_ast/ast3.py -> build/lib.linux-aarch64-3.7/typed_ast
#5 8.166     copying typed_ast/__init__.py -> build/lib.linux-aarch64-3.7/typed_ast
#5 8.166     copying typed_ast/conversions.py -> build/lib.linux-aarch64-3.7/typed_ast
#5 8.166     copying typed_ast/ast27.py -> build/lib.linux-aarch64-3.7/typed_ast
#5 8.166     package init file 'ast3/tests/__init__.py' not found (or not a regular file)
#5 8.166     creating build/lib.linux-aarch64-3.7/typed_ast/tests
#5 8.166     copying ast3/tests/test_basics.py -> build/lib.linux-aarch64-3.7/typed_ast/tests
#5 8.166     running build_ext
#5 8.166     building '_ast27' extension
#5 8.166     creating build/temp.linux-aarch64-3.7
#5 8.166     creating build/temp.linux-aarch64-3.7/ast27
#5 8.166     creating build/temp.linux-aarch64-3.7/ast27/Parser
#5 8.166     creating build/temp.linux-aarch64-3.7/ast27/Python
#5 8.166     creating build/temp.linux-aarch64-3.7/ast27/Custom
#5 8.166     gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -Iast27/Include -I/usr/local/include/python3.7m -c ast27/Parser/acceler.c -o build/temp.linux-aarch64-3.7/ast27/Parser/acceler.o
#5 8.166     unable to execute 'gcc': No such file or directory
#5 8.166     error: command 'gcc' failed with exit status 1
#5 8.166     ----------------------------------------
#5 8.166 ERROR: Command errored out with exit status 1: /usr/local/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-t7ha8wv7/typed-ast_57d9f08187904d3c90acceded4abb219/setup.py'"'"'; __file__='"'"'/tmp/pip-install-t7ha8wv7/typed-ast_57d9f08187904d3c90acceded4abb219/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-rah6u5dw/install-record.txt --single-version-externally-managed --compile --install-headers /usr/local/include/python3.7m/typed-ast Check the logs for full command output.
#5 8.168 WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
#5 8.168 You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
#5 ERROR: executor failed running [/bin/sh -c pip install datamodel-code-generator==0.10.1]: exit code: 1

marcosmarxm · 2021-12-21T21:54:10Z

@dluftspring are you still having problems? You can use the command I sent to you or the ones in the README.md file created with the generator.

dluftspring · 2021-12-22T15:53:07Z

@marcosmarxm I fixed the linter errors with this commit but I don't really have a good idea of why build_image.sh is failing. It just says theres a nonzero exit code but no other logging that I can find easily. Any help would be very appreciated :)

marcosmarxm · 2021-12-22T17:46:00Z

@dluftspring what command are you executing?

dluftspring · 2021-12-22T18:53:31Z

From the repo root im running ./gradlew airbyte-integrations:connectors:source-hellobaton:integrationTest

…est-pr-8461

marcosmarxm

awesome work @dluftspring 📟

* Connector working through read method for hellobaton source * Check connection method fixes * Cleaning up directories and files +activity stream * Changing webapp port for testing * More testing ports * Ports ports ports * porto * Reverting port changes * Updating activity stream schema and config catalog * Fixing flake lint failures * fix schemas * correct schemas and acceptance tests * run format * bump version in config files * Docs page for hellobaton * add doc * run format Co-authored-by: Marcos Marx <marcosmarxm@gmail.com>

Daniel Luftspring and others added 9 commits November 15, 2021 17:20

Connector working through read method for hellobaton source

90a55a6

Check connection method fixes

2478cee

Cleaning up directories and files +activity stream

c63c490

Changing webapp port for testing

3b4ae8e

More testing ports

eedef37

Ports ports ports

3983c8e

porto

2c673c0

Reverting port changes

56b478a

Merge branch 'airbytehq:master' into master

f38ef58

github-actions bot added the area/connectors Connector related issues label Dec 2, 2021

octavia-squidington-iii added the community label Dec 2, 2021

marcosmarxm mentioned this pull request Dec 2, 2021

CI Sandbox for Source Hellobaton #8476

Closed

Updating activity stream schema and config catalog

19235f7

Merge branch 'master' of github.com:dluftspring/airbyte into marcos/t…

0bfe7c7

…est-pr-8461

Fixing flake lint failures

ce0ad62

marcosmarxm self-assigned this Dec 17, 2021

github-actions bot added area/scheduler area/server area/worker Related to worker CDK Connector Development Kit kubernetes normalization labels Jan 14, 2022

marcosmarxm added 3 commits January 13, 2022 22:41

correct schemas and acceptance tests

ff1218a

run format

305fe2a

Merge branch 'master' into marcos/test-pr-8461

1f754c5

github-actions bot removed area/frontend area/worker Related to worker area/platform issues related to the platform kubernetes area/scheduler area/server normalization CDK Connector Development Kit area/documentation Improvements or additions to documentation labels Jan 14, 2022

marcosmarxm and others added 2 commits January 13, 2022 23:06

bump version in config files

fc1ef65

Docs page for hellobaton

d62f488

github-actions bot added the area/documentation Improvements or additions to documentation label Jan 14, 2022

marcosmarxm added 3 commits January 14, 2022 00:20

Merge branch 'master' of github.com:dluftspring/airbyte into marcos/t…

74988c1

…est-pr-8461

add doc

7523844

solve conflict

3dcef4b

marcosmarxm temporarily deployed to more-secrets January 14, 2022 17:34 Inactive

run format

d7baaa8

marcosmarxm approved these changes Jan 14, 2022

View reviewed changes

marcosmarxm merged commit e331387 into airbytehq:master Jan 14, 2022

octavia-squidington-iii mentioned this pull request Jan 20, 2022

Bump Airbyte version from 0.35.5-alpha to 0.35.6-alpha #9630

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Hellobaton #8461

🎉 New Source: Hellobaton #8461

dluftspring commented Dec 2, 2021 •

edited

Loading

CLAassistant commented Dec 2, 2021 •

edited

Loading

marcosmarxm commented Dec 2, 2021

dluftspring commented Dec 2, 2021

marcosmarxm commented Dec 6, 2021

marcosmarxm commented Dec 9, 2021

dluftspring commented Dec 11, 2021

marcosmarxm commented Dec 15, 2021

marcosmarxm commented Dec 15, 2021

dluftspring commented Dec 16, 2021 •

edited

Loading

dluftspring commented Dec 16, 2021 •

edited

Loading

marcosmarxm commented Dec 21, 2021

dluftspring commented Dec 22, 2021

marcosmarxm commented Dec 22, 2021

dluftspring commented Dec 22, 2021 •

edited

Loading

marcosmarxm left a comment

🎉 New Source: Hellobaton #8461

🎉 New Source: Hellobaton #8461

Conversation

dluftspring commented Dec 2, 2021 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

CLAassistant commented Dec 2, 2021 • edited Loading

marcosmarxm commented Dec 2, 2021

dluftspring commented Dec 2, 2021

marcosmarxm commented Dec 6, 2021

marcosmarxm commented Dec 9, 2021

dluftspring commented Dec 11, 2021

marcosmarxm commented Dec 15, 2021

marcosmarxm commented Dec 15, 2021

dluftspring commented Dec 16, 2021 • edited Loading

dluftspring commented Dec 16, 2021 • edited Loading

marcosmarxm commented Dec 21, 2021

dluftspring commented Dec 22, 2021

marcosmarxm commented Dec 22, 2021

dluftspring commented Dec 22, 2021 • edited Loading

marcosmarxm left a comment

Choose a reason for hiding this comment

dluftspring commented Dec 2, 2021 •

edited

Loading

CLAassistant commented Dec 2, 2021 •

edited

Loading

dluftspring commented Dec 16, 2021 •

edited

Loading

dluftspring commented Dec 16, 2021 •

edited

Loading

dluftspring commented Dec 22, 2021 •

edited

Loading