🐛 Source Qualaroo: Fix start_date & custom survey_ids #13121

danieldiamond · 2022-05-24T11:09:53Z

What

Fix issue with incorrect survey_ids schema
Fix issue when start_date is not properly formatted to epoch time
Add stream_slices
Seperate streams and sources

How

Convert start_date to epoch time
Convert survey_ids to strings for comparison

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit


➜ python -m pytest unit_tests
==================================================== test session starts =====================================================
platform darwin -- Python 3.9.12, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /Users/danieldiamond/.virtualenvs/airbyte/bin/python
cachedir: .pytest_cache
rootdir: /Users/danieldiamond/repos/airbyte, configfile: pytest.ini
plugins: requests-mock-1.9.3, mock-3.7.0
collected 4 items

unit_tests/test_streams.py::test_request_params PASSED
unit_tests/test_streams.py::test_next_page_token PASSED
unit_tests/test_streams.py::test_surveys_stream PASSED
unit_tests/test_streams.py::test_responses_stream PASSED

====================================================== warnings summary ======================================================
airbyte-integrations/connectors/source-qualaroo/unit_tests/test_streams.py: 11 warnings
  /Users/danieldiamond/.virtualenvs/airbyte/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:42: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator: HttpAuthenticator = NoAuth()

airbyte-integrations/connectors/source-qualaroo/unit_tests/test_streams.py: 11 warnings
  /Users/danieldiamond/.virtualenvs/airbyte/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
=============================================== 4 passed, 22 warnings in 0.43s ===============================================

Integration

NA

Acceptance

NA

danieldiamond · 2022-05-24T11:23:58Z

@marcosmarxm pagination is a pain with qualaroo API. this handles that and fixes survey_ids and start_date formatting issues

marcosmarxm · 2022-05-24T11:25:47Z

thanks @danieldiamond will look later today the contribution

marcosmarxm · 2022-05-24T11:26:16Z

/test connector=connectors/source-qualaroo

🕑 connectors/source-qualaroo https://github.com/airbytehq/airbyte/actions/runs/2377472518
❌ connectors/source-qualaroo https://github.com/airbytehq/airbyte/actions/runs/2377472518
🐛 https://gradle.com/s/bc7jcssx7yzru

Build Failed

Test summary info:

Could not find result summary

danieldiamond · 2022-05-24T13:25:28Z

@marcosmarxm can't seem to figure out why the above failed
Could not find result summary is not very informative. did the integration test just fail randomly in github actions? Can we retry?

marcosmarxm · 2022-05-25T00:07:40Z

@danieldiamond looks a flake issue:

[python] .venv/bin/python -m pflake8 --config /actions-runner/_work/airbyte/airbyte/pyproject.toml ./
	 ./source_qualaroo/source.py:6:1: F401 'abc.ABC' imported but unused
	 ./source_qualaroo/source.py:8:1: F401 'typing.Iterable' imported but unused
	 ./source_qualaroo/source.py:8:1: F401 'typing.MutableMapping' imported but unused
	 ./source_qualaroo/source.py:8:1: F401 'typing.Optional' imported but unused
	 ./source_qualaroo/source.py:14:1: F401 'airbyte_cdk.sources.streams.http.HttpStream' imported but unused

can you run format for those files

danieldiamond · 2022-05-25T00:15:51Z

@marcosmarxm i did that in my last commit, can you rerun the test?

marcosmarxm · 2022-05-25T00:20:06Z

/test connector=connectors/source-qualaroo

🕑 connectors/source-qualaroo https://github.com/airbytehq/airbyte/actions/runs/2381354189
✅ connectors/source-qualaroo https://github.com/airbytehq/airbyte/actions/runs/2381354189
Python tests coverage:

Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
source_acceptance_test/utils/__init__.py                 6      0   100%
source_acceptance_test/tests/__init__.py                 4      0   100%
source_acceptance_test/__init__.py                       2      0   100%
source_acceptance_test/tests/test_full_refresh.py       52      2    96%
source_acceptance_test/utils/asserts.py                 37      2    95%
source_acceptance_test/config.py                        77      6    92%
source_acceptance_test/utils/json_schema_helper.py     105     13    88%
source_acceptance_test/tests/test_incremental.py       121     25    79%
source_acceptance_test/utils/common.py                  80     17    79%
source_acceptance_test/tests/test_core.py              294    106    64%
source_acceptance_test/utils/compare.py                 62     23    63%
source_acceptance_test/base.py                          10      4    60%
source_acceptance_test/utils/connector_runner.py       110     48    56%
------------------------------------------------------------------------
TOTAL                                                  960    246    74%
Name                          Stmts   Miss  Cover
-------------------------------------------------
source_qualaroo/__init__.py       2      0   100%
source_qualaroo/streams.py       61      6    90%
source_qualaroo/source.py        41     25    39%
-------------------------------------------------
TOTAL                           104     31    70%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config
======================== 21 passed, 1 skipped in 16.66s ========================

harshithmullapudi · 2022-05-25T09:15:45Z

@danieldiamond is it possible to add incremental tests to acceptance-test-config.yml

danieldiamond · 2022-05-25T09:36:49Z

@harshithmullapudi this isnt an incremental connector.
why is it trying to run an incremental test?

danieldiamond · 2022-05-25T09:43:56Z

@harshithmullapudi why would I add incremental tests if this isn't an incremental connector? Is it not meant to skip the test_two_sequential_reads as shown? It appears in the previous PRs associated with qualaroo and they were still published

harshithmullapudi · 2022-05-25T09:48:25Z

My bad that's not because are failing got confused with different PR 😢

harshithmullapudi · 2022-05-25T09:58:00Z

/publish connector=connectors/source-qualaroo auto-bump-version=false

🕑 connectors/source-qualaroo https://github.com/airbytehq/airbyte/actions/runs/2383638271
🚀 Successfully published connectors/source-qualaroo
✅ connectors/source-qualaroo https://github.com/airbytehq/airbyte/actions/runs/2383638271

* Fix survey_ids schema, fix epoch time start_date, add stream_slices * Formatting * Bump version * Update tests * Formatting * Remove unused imports * chore: bump version in source definitions * chore: update seed file Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com>

danieldiamond added 3 commits May 24, 2022 21:00

Fix survey_ids schema, fix epoch time start_date, add stream_slices

c5e23b4

Formatting

80ae058

Bump version

649ca0f

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels May 24, 2022

octavia-squidington-iii added the community label May 24, 2022

danieldiamond added 2 commits May 24, 2022 21:18

Update tests

683fee3

Formatting

b87a3e1

Remove unused imports

d1dc9ea

harshithmullapudi self-requested a review May 25, 2022 09:55

harshithmullapudi self-assigned this May 25, 2022

chore: bump version in source definitions

8cbbf9a

chore: update seed file

3dffb5b

harshithmullapudi merged commit 4bfd329 into airbytehq:master May 25, 2022

octavia-squidington-iii mentioned this pull request May 25, 2022

Bump Airbyte version from 0.39.1-alpha to 0.39.2-alpha #13188

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Qualaroo: Fix start_date & custom survey_ids #13121

🐛 Source Qualaroo: Fix start_date & custom survey_ids #13121

danieldiamond commented May 24, 2022 •

edited

Loading

danieldiamond commented May 24, 2022

marcosmarxm commented May 24, 2022

marcosmarxm commented May 24, 2022 •

edited by github-actions bot

Loading

danieldiamond commented May 24, 2022

marcosmarxm commented May 25, 2022

danieldiamond commented May 25, 2022

marcosmarxm commented May 25, 2022 •

edited by github-actions bot

Loading

harshithmullapudi commented May 25, 2022

danieldiamond commented May 25, 2022 •

edited

Loading

danieldiamond commented May 25, 2022

harshithmullapudi commented May 25, 2022

harshithmullapudi commented May 25, 2022 •

edited by github-actions bot

Loading

🐛 Source Qualaroo: Fix start_date & custom survey_ids #13121

🐛 Source Qualaroo: Fix start_date & custom survey_ids #13121

Conversation

danieldiamond commented May 24, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

danieldiamond commented May 24, 2022

marcosmarxm commented May 24, 2022

marcosmarxm commented May 24, 2022 • edited by github-actions bot Loading

Build Failed

danieldiamond commented May 24, 2022

marcosmarxm commented May 25, 2022

danieldiamond commented May 25, 2022

marcosmarxm commented May 25, 2022 • edited by github-actions bot Loading

Build Passed

harshithmullapudi commented May 25, 2022

danieldiamond commented May 25, 2022 • edited Loading

danieldiamond commented May 25, 2022

harshithmullapudi commented May 25, 2022

harshithmullapudi commented May 25, 2022 • edited by github-actions bot Loading

danieldiamond commented May 24, 2022 •

edited

Loading

marcosmarxm commented May 24, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented May 25, 2022 •

edited by github-actions bot

Loading

danieldiamond commented May 25, 2022 •

edited

Loading

harshithmullapudi commented May 25, 2022 •

edited by github-actions bot

Loading