🎉 New Source: SurveyCTO #19371

siddhant3030 · 2022-11-14T06:42:09Z

What

Describe what the change is solving

SurveyCTO is a mobile data collection platform for researchers and professionals working in offline settings. For organizations using SurveyCTO having a native Airbyte connector to import data will allow for ingestion into their data warehouses and use of the data for analysis. Current processes involve working with SurveyCTO APIs and ingestion through scripts.

It helps to add screenshots if it affects the frontend. - doesn't effect the frontend

How

Describe the solution

The current version of the connector provides for ingesting data through Airbyte from multiple forms in an organization's SurveyCTO account. The implemented version of the connector will allow for all sync modes - full refresh and incremental including de-duped mode. A best-guess normalization is also provided to ease the normalization process using a couple of python libraries to guess the schema from the data fetched back through the API

Pre-merge Checklist

Expand the relevant checklist and delete the others.

SurveyCTO

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

unit_tests/test_source.py::test_spec PASSED
unit_tests/test_source.py::test_check_connection PASSED
unit_tests/test_source.py::test_streams {"type": "LOG", "log": {"level": "INFO", "message": "Processed 0 lines"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Processed 0 lines"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Processed 0 lines"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Processed 0 lines"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Processed 0 lines"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Processed 0 lines"}}
{"type": "LOG", "log": {"level": "INFO", "message": "Processed 0 lines"}}
PASSED
unit_tests/test_streams.py::test_get_json_schema PASSED

Integration

Test session starts (platform: darwin, Python 3.10.6, pytest 6.2.5, pytest-sugar 0.9.6)
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/Users/arun/Documents/airbyte/airbyte-integrations/connectors/source-surveycto/.hypothesis/examples')
rootdir: /Users/arun/Documents/airbyte, configfile: pytest.ini
plugins: hypothesis-6.54.6, requests-mock-1.9.3, mock-3.6.1, sugar-0.9.6, timeout-1.4.2, cov-3.0.0
collecting ...

Results (0.35s):

Acceptance

sajarin · 2022-11-14T18:12:45Z

Thanks @siddhant3030, this will be reviewed as part of Airbyte's Community Maintainer program. Please remain patient as someone is assigned to review your PR and ping me if you don't get a response in a few days.

andresbravog · 2022-11-14T18:28:11Z

Thanks @siddhant3030 , here Andres, part of the Community Maintainer program. I'll be assisting with the review of this PR.

andresbravog · 2022-11-14T18:59:54Z

@siddhant3030 seems you are still working on the PR, please ask for my review when you are ready. Thanks!

siddhant3030 · 2022-11-14T19:02:41Z

@andresbravog Hey. It's ready for review. I just did some cleanups. Please let me know what things to add more here

andresbravog · 2022-11-14T19:20:12Z

Thanks, @siddhant3030 there is a Community member or Airbyter pre-merge checklist on this PR description. would you mind t go over it and checking things out?

siddhant3030 · 2022-11-14T19:30:36Z

@andresbravog Can you help me with this? Getting this while running the acceptance test

"source_acceptance_test.plugin"
ImportError: Error importing plugin "source_acceptance_test.plugin": No module named 'source_acceptance_test'

andresbravog · 2022-11-14T19:49:54Z

@siddhant3030 in your case as this is a Python CDK connector you can:

Prepare

cd airbyte-integrations/connectors/source-surveycto
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
docker build . -t airbyte/source-surveycto:dev

Integration tests

python -m pytest -s integration_tests

or just acceptance

python -m pytest integration_tests -p integration_tests.acceptance

Unit tests

python -m pytest -s unit_tests

siddhant3030 · 2022-11-18T18:43:33Z

@andresbravog Just fixed it all. Please review

andresbravog · 2022-11-18T19:02:52Z

@siddhant3030 well done you are clean now!

@sajarin can we retest?

marcosmarxm · 2022-11-22T20:25:29Z

Hello 👋, first thank you for this amazing contribution.

We really appreciate the effort you've made to improve the project.
We ask you patience for the code review. Last month our team was focused on Hacktoberfest event and that probably left some PR without the proper feedback. And this week, due to the Thanksgiving US Holiday, most our team is out of office with their families. Another important piece of information why code won't be merge this week is: as a safety measure the core team has decided to freeze merging code to main branch to keep the release stable. Next week we'll return to you with the proper code review and update the status of your contribution.

If you have any questions feel free to send me a message in Slack!
Thanks!

sajarin · 2022-11-28T19:06:51Z

/publish connector=connectors/source-surveycto run-tests=false

🕑 Publishing the following connectors:
connectors/source-surveycto
https://github.com/airbytehq/airbyte/actions/runs/3567856588

Connector	Did it publish?	Were definitions generated?
connectors/source-surveycto	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

andresbravog · 2022-11-30T09:51:19Z

@sajarin this seems to be completed and published who takes care about conflicts?

siddhant3030 · 2022-11-30T18:25:11Z

@sajarin Is this ready to be merged?

sajarin · 2022-11-30T18:26:28Z

@siddhant3030 thanks for the PR and thanks @andresbravog for the review!

siddhant3030 · 2022-11-30T18:35:22Z

@sajarin @andresbravog Thankyou guys.

survey cto connector

f992673

octavia-squidington-iv added bounty community connectors/source/surveycto area/connectors Connector related issues labels Nov 14, 2022

siddhant3030 changed the title ~~survey cto connector~~ contributing-to-airbyte/survey-cto-connector Nov 14, 2022

siddhant3030 added 2 commits November 14, 2022 12:12

Merge branch 'master' into feature/survey-cto_connector

1990cd7

Merge branch 'master' into feature/survey-cto_connector

d6cac7d

siddhant3030 changed the title ~~contributing-to-airbyte/survey-cto-connector~~ New Source: SurveyCTO Nov 14, 2022

siddhant3030 and others added 3 commits November 14, 2022 18:27

Merge branch 'master' into feature/survey-cto_connector

abe80bc

Merge branch 'master' into feature/survey-cto_connector

f84d99a

Merge branch 'master' into feature/survey-cto_connector

53763d2

sajarin changed the title ~~New Source: SurveyCTO~~ 🎉 New Source: SurveyCTO Nov 14, 2022

sajarin added the bounty-XL Maintainer program: claimable extra large bounty PR label Nov 14, 2022

datetime fix

2ee0161

sajarin assigned andresbravog Nov 14, 2022

siddhant3030 added 2 commits November 15, 2022 00:01

Merge branch 'master' into feature/survey-cto_connector

adbf1e8

removed some unused tests

4bf3af8

Merge branch 'master' into feature/survey-cto_connector

36e30aa

intergration tests

2c86b4d

siddhant3030 added 3 commits November 15, 2022 01:41

acceptance test is passing

275a578

Merge branch 'master' into feature/survey-cto_connector

ed3b031

airbyte_secret

b9f7558

siddhant3030 added 3 commits November 18, 2022 23:39

refactor

084b271

python format fix

2b2815d

Merge branch 'master' into feature/survey-cto_connector

91e7247

siddhant3030 added 4 commits November 19, 2022 00:36

Merge branch 'master' into feature/survey-cto_connector

745d8af

Merge branch 'master' into feature/survey-cto_connector

3e10acb

Merge branch 'master' into feature/survey-cto_connector

972be18

Merge branch 'master' into feature/survey-cto_connector

c4283d4

siddhant3030 and others added 5 commits November 23, 2022 09:12

Merge branch 'master' into feature/survey-cto_connector

5262a96

Merge branch 'master' into feature/survey-cto_connector

76b0229

Merge branch 'master' into feature/survey-cto_connector

212b8ea

Merge branch 'master' into feature/survey-cto_connector

1b9cb19

Merge branch 'master' into feature/survey-cto_connector

ff522c9

sajarin requested a review from a team as a code owner November 28, 2022 18:54

sajarin requested review from sajarin and removed request for a team November 28, 2022 19:04

sajarin approved these changes Nov 28, 2022

View reviewed changes

auto-bump connector version

13ca688

siddhant3030 and others added 2 commits November 30, 2022 21:46

docs update

30a9977

Merge branch 'master' into feature/survey-cto_connector

6e454ce

sajarin merged commit 4c0ffb6 into airbytehq:master Nov 30, 2022

octavia-squidington-iii mentioned this pull request Nov 30, 2022

Bump Airbyte version from 0.40.22 to 0.40.23 #19941

Merged

siddhant3030 deleted the feature/survey-cto_connector branch December 8, 2022 05:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: SurveyCTO #19371

🎉 New Source: SurveyCTO #19371

siddhant3030 commented Nov 14, 2022 •

edited

Loading

sajarin commented Nov 14, 2022

andresbravog commented Nov 14, 2022

andresbravog commented Nov 14, 2022

siddhant3030 commented Nov 14, 2022

andresbravog commented Nov 14, 2022

siddhant3030 commented Nov 14, 2022 •

edited

Loading

andresbravog commented Nov 14, 2022 •

edited

Loading

siddhant3030 commented Nov 18, 2022

andresbravog commented Nov 18, 2022

marcosmarxm commented Nov 22, 2022

sajarin commented Nov 28, 2022 •

edited by github-actions bot

Loading

andresbravog commented Nov 30, 2022

siddhant3030 commented Nov 30, 2022 •

edited

Loading

sajarin commented Nov 30, 2022

siddhant3030 commented Nov 30, 2022

🎉 New Source: SurveyCTO #19371

🎉 New Source: SurveyCTO #19371

Conversation

siddhant3030 commented Nov 14, 2022 • edited Loading

What

How

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

sajarin commented Nov 14, 2022

andresbravog commented Nov 14, 2022

andresbravog commented Nov 14, 2022

siddhant3030 commented Nov 14, 2022

andresbravog commented Nov 14, 2022

siddhant3030 commented Nov 14, 2022 • edited Loading

andresbravog commented Nov 14, 2022 • edited Loading

Prepare

Integration tests

Unit tests

siddhant3030 commented Nov 18, 2022

andresbravog commented Nov 18, 2022

marcosmarxm commented Nov 22, 2022

sajarin commented Nov 28, 2022 • edited by github-actions bot Loading

andresbravog commented Nov 30, 2022

siddhant3030 commented Nov 30, 2022 • edited Loading

sajarin commented Nov 30, 2022

siddhant3030 commented Nov 30, 2022

siddhant3030 commented Nov 14, 2022 •

edited

Loading

siddhant3030 commented Nov 14, 2022 •

edited

Loading

andresbravog commented Nov 14, 2022 •

edited

Loading

sajarin commented Nov 28, 2022 •

edited by github-actions bot

Loading

siddhant3030 commented Nov 30, 2022 •

edited

Loading