Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🎉 New Source: AppFollow Service #14418

Merged
merged 28 commits into from
Aug 16, 2022
Merged

Conversation

dtt101
Copy link
Contributor

@dtt101 dtt101 commented Jul 5, 2022

What

This introduces a new source connector for the Appfollow service.

It allows users to gather mobile application data from multiple app stores.

This initial version contains support for one API endpoint: ratings as documented here

How

The connector exposes a number of config params, api secret, client ID, app store (i,e play / apple) ID and country and reads ratings data from the source API

🚨 User Impact 🚨

There are no breaking changes - the PR should provide a new connector option for airbyte users

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

  • Community member? Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
    • docs/integrations/README.md
    • airbyte-integrations/builds.md
  • PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub by running the /publish command described here
  • After the connector is published, connector added to connector index as described here
  • Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • If new credentials are required for use in CI, add them to GSM. Instructions.
  • /test connector=connectors/<name> command is passing
  • New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit
Test session starts (platform: darwin, Python 3.9.12, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/vs/projects/data/airbyte, configfile: pytest.ini
plugins: requests-mock-1.9.3, sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...
 airbyte-integrations/connectors/source-appfollow/unit_tests/test_source.py::test_check_connection ✓  9% ▉
 airbyte-integrations/connectors/source-appfollow/unit_tests/test_source.py::test_streams ✓          18% █▊
 airbyte-integrations/connectors/source-appfollow/unit_tests/test_streams.py::test_request_params ✓  27% ██▊
 airbyte-integrations/connectors/source-appfollow/unit_tests/test_streams.py::test_parse_response ✓  36% ███▋
 airbyte-integrations/connectors/source-appfollow/unit_tests/test_streams.py::test_request_headers ✓ 45% ████▋
 airbyte-integrations/connectors/source-appfollow/unit_tests/test_streams.py::test_http_method ✓     55% █████▌
 airbyte-integrations/connectors/source-appfollow/unit_tests/test_streams.py::test_should_retry[HTTPStatus.OK-False] ✓64% ██████▍
 airbyte-integrations/connectors/source-appfollow/unit_tests/test_streams.py::test_should_retry[HTTPStatus.BAD_REQUEST-False] ✓73% ███████▍
 airbyte-integrations/connectors/source-appfollow/unit_tests/test_streams.py::test_should_retry[HTTPStatus.TOO_MANY_REQUESTS-True] ✓82% ████████▎
 airbyte-integrations/connectors/source-appfollow/unit_tests/test_streams.py::test_should_retry[HTTPStatus.INTERNAL_SERVER_ERROR-True] ✓91% █████████▏
 airbyte-integrations/connectors/source-appfollow/unit_tests/test_streams.py::test_backoff_time ✓   100% ██████████
================================================ warnings summary =================================================
airbyte-integrations/connectors/source-appfollow/unit_tests/test_source.py::test_check_connection
airbyte-integrations/connectors/source-appfollow/unit_tests/test_source.py::test_check_connection
  /Users/vs/projects/data/airbyte/airbyte-integrations/connectors/source-appfollow/.venv/lib/python3.9/site-packages/requests/auth.py:36: DeprecationWarning: Non-string usernames will no longer be supported in Requests 3.0.0. Please convert the object you've passed in (<MagicMock name='mock.__getitem__()' id='4425779328'>) to a string or bytes object in the near future to avoid problems.
    warnings.warn(

airbyte-integrations/connectors/source-appfollow/unit_tests/test_source.py::test_check_connection
airbyte-integrations/connectors/source-appfollow/unit_tests/test_source.py::test_check_connection
  /Users/vs/projects/data/airbyte/airbyte-integrations/connectors/source-appfollow/.venv/lib/python3.9/site-packages/requests/auth.py:46: DeprecationWarning: Non-string passwords will no longer be supported in Requests 3.0.0. Please convert the object you've passed in (<class 'unittest.mock.MagicMock'>) to a string or bytes object in the near future to avoid problems.
    warnings.warn(

airbyte-integrations/connectors/source-appfollow/unit_tests/test_source.py::test_streams
  /Users/vs/projects/data/airbyte/airbyte-integrations/connectors/source-appfollow/source_appfollow/source.py:111: DeprecationWarning: Call to deprecated class TokenAuthenticator. (Use airbyte_cdk.sources.streams.http.requests_native_auth.TokenAuthenticator instead) -- Deprecated since version 0.1.20.
    auth = BasicHttpAuthenticator(username=config["api_secret"], password=config["api_secret"])

airbyte-integrations/connectors/source-appfollow/unit_tests/test_source.py::test_streams
airbyte-integrations/connectors/source-appfollow/unit_tests/test_source.py::test_streams
  /Users/vs/projects/data/airbyte/airbyte-integrations/connectors/source-appfollow/.venv/lib/python3.9/site-packages/deprecated/classic.py:173: DeprecationWarning: Call to deprecated class HttpAuthenticator. (Use requests.auth.AuthBase instead) -- Deprecated since version 0.1.20.
    return old_new1(cls, *args, **kwargs)

airbyte-integrations/connectors/source-appfollow/unit_tests/test_source.py::test_streams
  /Users/vs/projects/data/airbyte/airbyte-integrations/connectors/source-appfollow/.venv/lib/python3.9/site-packages/airbyte_cdk/sources/streams/http/http.py:41: DeprecationWarning: Call to deprecated class NoAuth. (Set `authenticator=None` instead) -- Deprecated since version 0.1.20.
    self._authenticator: HttpAuthenticator = NoAuth()

-- Docs: https://docs.pytest.org/en/stable/warnings.html

Results (0.37s):
      11 passed
Integration
Test session starts (platform: darwin, Python 3.9.12, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/vs/projects/data/airbyte, configfile: pytest.ini
plugins: requests-mock-1.9.3, sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...

Results (0.01s):
Acceptance
Test session starts (platform: darwin, Python 3.9.12, pytest 6.2.5, pytest-sugar 0.9.4)
cachedir: .pytest_cache
rootdir: /Users/vs/projects/data/airbyte, configfile: pytest.ini
plugins: requests-mock-1.9.3, sugar-0.9.4, mock-3.6.1, timeout-1.4.2
collecting ...
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_config_match_spec[inputs0] ✓5% ▌
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_match_expected[inputs0] ✓10% ▉
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_docker_env[inputs0] ✓14% █▌
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oneof_usage[inputs0] ✓19% █▉
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_required[inputs0] ✓24% ██▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_optional[inputs0] ✓29% ██▉
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_has_secret[inputs0] ✓33% ███▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_secret_never_in_the_output[inputs0] ✓38% ███▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_defined_refs_exist_in_json_spec_file[inputs0] ✓43% ████▍
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestSpec.test_oauth_flow_parameters[inputs0] ✓48% ████▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs0] ✓52% █████▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestConnection.test_check[inputs1] ✓57% █████▊
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_discover[inputs0] ✓62% ██████▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_cursors_exist_in_schema[inputs0] ✓67% ██████▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_refs_exist_in_schema[inputs0] ✓71% ███████▎
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-allOf] ✓76% ███████▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_defined_keyword_exist_in_schema[inputs0-not] ✓81% ████████▏
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestDiscovery.test_primary_keys_exist_in_schema[inputs0] ✓86% ████████▋
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_read[inputs0] ✓90% █████████▏{"type": "LOG", "log": {"level": "ERROR", "message": "Docker container was failed, code 1, error:\n{\"type\": \"TRACE\", \"trace\": {\"type\": \"ERROR\", \"emitted_at\": 1657035716483.1519, \"error\": {\"message\": \"Something went wrong in the connector. See the logs for more details.\", \"internal_message\": \"2 validation errors for ConfiguredAirbyteCatalog\\nstreams -> 0 -> sync_mode\\n  value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\\nstreams -> 0 -> destination_sync_mode\\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\", \"stack_trace\": \"Traceback (most recent call last):\\n  File \\\"/airbyte/integration_code/main.py\\\", line 13, in <module>\\n    launch(source, sys.argv[1:])\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\\\", line 118, in launch\\n    for message in source_entrypoint.run(parsed_args):\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\\\", line 106, in run\\n    config_catalog = self.source.read_catalog(parsed_args.catalog)\\n  File \\\"/usr/local/lib/python3.9/site-packages/airbyte_cdk/sources/source.py\\\", line 28, in read_catalog\\n    return ConfiguredAirbyteCatalog.parse_obj(self.read_config(catalog_path))\\n  File \\\"/usr/local/lib/python3.9/site-packages/pydantic/main.py\\\", line 521, in parse_obj\\n    return cls(**obj)\\n  File \\\"/usr/local/lib/python3.9/site-packages/pydantic/main.py\\\", line 341, in __init__\\n    raise validation_error\\npydantic.error_wrappers.ValidationError: 2 validation errors for ConfiguredAirbyteCatalog\\nstreams -> 0 -> sync_mode\\n  value is not a valid enumeration member; permitted: 'full_refresh', 'incremental' (type=type_error.enum; enum_values=[<SyncMode.full_refresh: 'full_refresh'>, <SyncMode.incremental: 'incremental'>])\\nstreams -> 0 -> destination_sync_mode\\n  value is not a valid enumeration member; permitted: 'append', 'overwrite', 'append_dedup' (type=type_error.enum; enum_values=[<DestinationSyncMode.append: 'append'>, <DestinationSyncMode.overwrite: 'overwrite'>, <DestinationSyncMode.append_dedup: 'append_dedup'>])\\n\", \"failure_type\": \"system_error\"}}}\n"}}

 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py::TestBasicRead.test_airbyte_trace_message_on_failure[inputs0] ✓95% █████████▌
 airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py::TestFullRefresh.test_sequential_reads[inputs0] ✓100% ██████████
{"type": "LOG", "log": {"level": "INFO", "message": "/Users/vs/projects/data/airbyte/airbyte-integrations/connectors/source-appfollow - SAT run - a8760e28aefc5352d7d6b21038ba0dee097d5983 - PASSED"}}

================================================ warnings summary =================================================
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py: 17 warnings
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py: 1 warning
  /Users/vs/projects/data/airbyte/airbyte-integrations/connectors/source-appfollow/.venv/lib/python3.9/site-packages/docker/utils/utils.py:52: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    s1 = StrictVersion(v1)

airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_core.py: 17 warnings
airbyte-integrations/bases/source-acceptance-test/source_acceptance_test/tests/test_full_refresh.py: 1 warning
  /Users/vs/projects/data/airbyte/airbyte-integrations/connectors/source-appfollow/.venv/lib/python3.9/site-packages/docker/utils/utils.py:53: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    s2 = StrictVersion(v2)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================================= short test summary info =============================================
SKIPPED [1] ../../bases/source-acceptance-test/source_acceptance_test/plugin.py:56: Skipping TestIncremental.test_two_sequential_reads because not found in the config

Results (19.74s):
      21 passed

@CLAassistant
Copy link

CLAassistant commented Jul 5, 2022

CLA assistant check
All committers have signed the CLA.

@github-actions github-actions bot added the area/connectors Connector related issues label Jul 5, 2022
@dtt101 dtt101 marked this pull request as ready for review July 5, 2022 15:50
@marcosmarxm marcosmarxm changed the title Adding source connector for AppFollow Service 🎉 New SourcE: AppFollow Service Jul 5, 2022
@marcosmarxm
Copy link
Member

Thanks for the contributin @dtt101 ! Soon someone from the team will review it

@marcosmarxm marcosmarxm changed the title 🎉 New SourcE: AppFollow Service 🎉 New Source: AppFollow Service Jul 5, 2022
@dtt101
Copy link
Contributor Author

dtt101 commented Jul 5, 2022

Thanks @marcosmarxm - appreciated - the first one I have written so advice is definitely welcome!

@igrankova
Copy link
Contributor

@dtt101
David, thank you very much.
Use this email olga@appfollow.io
I've informed Olga about your email.

@dtt101
Copy link
Contributor Author

dtt101 commented Jul 23, 2022

@igrankova - email sent with confirmation we are a customer

@igrankova
Copy link
Contributor

@dtt101 Thank you very much!
We received the sandbox with API access

@marcosmarxm
Copy link
Member

@harshithmullapudi you can review and publish without the sandbox account.

@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Aug 11, 2022
@harshithmullapudi
Copy link
Contributor

harshithmullapudi commented Aug 11, 2022

/publish connector=connectors/source-appfollow

🕑 Publishing the following connectors:
connectors/source-appfollow
https://github.com/airbytehq/airbyte/actions/runs/2839218380


Connector Did it publish? Were definitions generated?
connectors/source-appfollow

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@dtt101
Copy link
Contributor Author

dtt101 commented Aug 12, 2022

@harshithmullapudi - thanks for the commits!
For the failed publish - is there a way I can run this process locally to debug?
Do you have any suggestions on how I could address the failed test in the publish action workflow?

@dtt101
Copy link
Contributor Author

dtt101 commented Aug 12, 2022

/publish connector=connectors/source-appfollow

@harshithmullapudi
Copy link
Contributor

harshithmullapudi commented Aug 13, 2022

/publish connector=connectors/source-appfollow

🕑 Publishing the following connectors:
connectors/source-appfollow
https://github.com/airbytehq/airbyte/actions/runs/2850665116


Connector Did it publish? Were definitions generated?
connectors/source-appfollow

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@dtt101
Copy link
Contributor Author

dtt101 commented Aug 13, 2022

@harshithmullapudi - thanks for running the publish - bumping the version has resolved the failing test.
The exception raised by the publish workflow now is INTERNALERROR> Config expected dict not NoneType (type=type_error)

This looks like because the acceptance test config is missing - is this because you commented out, or is there some other config I need to add?

@harshithmullapudi
Copy link
Contributor

harshithmullapudi commented Aug 15, 2022

/test connector=connectors/source-appfollow

🕑 connectors/source-appfollow https://github.com/airbytehq/airbyte/actions/runs/2860520459
❌ connectors/source-appfollow https://github.com/airbytehq/airbyte/actions/runs/2860520459
🐛 https://gradle.com/s/d4ramwej6jqmw

Build Failed

Test summary info:

=========================== short test summary info ============================
ERROR test_core.py::TestSpec::test_config_match_spec[inputs0] - FileNotFoundE...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestConnection.test_check because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestDiscovery.test_discover because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestBasicRead.test_read because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
==================== 10 passed, 6 skipped, 1 error in 8.97s ====================

@harshithmullapudi
Copy link
Contributor

harshithmullapudi commented Aug 15, 2022

/publish connector=connectors/source-appfollow

🕑 Publishing the following connectors:
connectors/source-appfollow
https://github.com/airbytehq/airbyte/actions/runs/2860617168


Connector Did it publish? Were definitions generated?
connectors/source-appfollow

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@harshithmullapudi
Copy link
Contributor

harshithmullapudi commented Aug 15, 2022

/publish connector=connectors/source-appfollow run-tests=false

🕑 Publishing the following connectors:
connectors/source-appfollow
https://github.com/airbytehq/airbyte/actions/runs/2860885518


Connector Did it publish? Were definitions generated?
connectors/source-appfollow

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@dtt101
Copy link
Contributor Author

dtt101 commented Aug 15, 2022

Ok - great it is building and publishing! I am not sure of the next step (apols if I am being dense).

Would welcome any suggestions - thanks for the time so far cc @harshithmullapudi @marcosmarxm

@harshithmullapudi
Copy link
Contributor

Hey @dtt101 you can go ahead and use the connector

@harshithmullapudi harshithmullapudi merged commit 92aeac3 into airbytehq:master Aug 16, 2022
rodireich pushed a commit that referenced this pull request Aug 20, 2022
* Generated template

Using https://docs.airbyte.com/connector-development/tutorials/cdk-speedrun

* Added spec.yaml

Ref https://appfollow.docs.apiary.io/
Does not currently include authentication

* Schema and working discover for ratings API

* Add catalog

* Add cid as required parameter

* Working ratings call

BASIC AUTH REALLY

* Use basic auth in check connection

* Updated comment for request params

* Update README for local run

* Moved with common params to stream superclass

* Add support for country parameter

* WIP: acceptance

90% pass

* Passing acceptance tests for ratings in docker

* Remove incremental unit tests

We have no incremental streams

* Match number of streams to test

* Passing streams unit tests

* Connection check test

* Update unit test type check

* Remove TODO

* fix: tests are failing

* docs: added documentation for appfollow and logo

* docs: added documentation for appfollow and logo

* Bump io.airbyte.version

* fix: tests are failing

* auto-bump connector version [ci skip]

Co-authored-by: Harshith Mullapudi <harshithmullapudi@gmail.com>
Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>
@wallies
Copy link
Contributor

wallies commented Dec 11, 2022

@dtt101 first thanks for building this connector. Did you manage to get this working. All i get is json in the ratings column no matter what destination I try.

@dtt101
Copy link
Contributor Author

dtt101 commented Dec 12, 2022

@wallies - yes - rating is the only endpoint it supports (or do you mean you get an error in the expected response)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation blocked bounty community connectors/source/appfollow waiting-for-cred Waiting for CI credentials
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

7 participants