Source Google Ads: fix schema for "campaigns" stream #19700

grubberr · 2022-11-22T10:03:06Z

Signed-off-by: Sergey Chvalyuk grubberr@gmail.com

What

Try to fix this oncall https://github.com/airbytehq/oncall/issues/947

https://developers.google.com/google-ads/api/fields/v12/metrics#metrics.conversions - DOUBLE
https://developers.google.com/google-ads/api/fields/v12/metrics#metrics.conversions_value - DOUBLE

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-11-22T10:07:37Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3522359518
❌ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3522359518
🐛 https://gradle.com/s/eimgvfqzwhglc

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - so...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
======= 1 failed, 29 passed, 1 skipped, 29 warnings in 888.03s (0:14:48) =======

> Task :airbyte-integrations:connectors:source-google-ads:sourceAcceptanceTest FAILED

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings

Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
50 actionable tasks: 37 executed, 13 up-to-date

Publishing build scan...
https://gradle.com/s/eimgvfqzwhglc

The remote build cache was disabled during the build due to errors.
S3 cache 634ms wasted on misses, reads: 1, elapsed: 634ms
S3 cache writes: 1, elapsed: 52ms, sent to cache: 90 KiB

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-11-22T18:04:00Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3525933181
❌ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3525933181
🐛 https://gradle.com/s/vwusjzzirhpzw

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - so...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: Backward compatibility tests are disabled for version 0.2.4.
======= 1 failed, 28 passed, 2 skipped, 29 warnings in 847.90s (0:14:07) =======

> Task :airbyte-integrations:connectors:source-google-ads:sourceAcceptanceTest FAILED

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.4/userguide/command_line_interface.html#sec:command_line_warnings

Execution optimizations have been disabled for 1 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
50 actionable tasks: 37 executed, 13 up-to-date

Publishing build scan...
https://gradle.com/s/vwusjzzirhpzw

The remote build cache was disabled during the build due to errors.
S3 cache 563ms wasted on misses, reads: 1, elapsed: 563ms
S3 cache writes: 1, elapsed: 59ms, sent to cache: 90 KiB

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-11-22T18:41:29Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3526179278

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-11-22T18:44:11Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3526195751
❌ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3526195751
🐛 https://gradle.com/s/k2nedrvmwlxmw

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Timeout >600.0s
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:370: Backward compatibility tests are disabled for version 0.2.4.
============= 1 failed, 28 passed, 2 skipped in 1163.07s (0:19:23) =============

grubberr · 2022-11-22T19:24:57Z

/test connector=connectors/source-google-ads

🕑 connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3526460245
✅ connectors/source-google-ads https://github.com/airbytehq/airbyte/actions/runs/3526460245
Python tests coverage:

Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/models.py                   18      0   100%
source_google_ads/__init__.py                  2      0   100%
source_google_ads/google_ads.py               68     10    85%
source_google_ads/streams.py                 165     26    84%
source_google_ads/source.py                   86     24    72%
source_google_ads/custom_query_stream.py      75     46    39%
--------------------------------------------------------------
TOTAL                                        414    106    74%
Name                                       Stmts   Miss  Cover
--------------------------------------------------------------
source_google_ads/models.py                   18      0   100%
source_google_ads/__init__.py                  2      0   100%
source_google_ads/streams.py                 165      8    95%
source_google_ads/source.py                   86      5    94%
source_google_ads/custom_query_stream.py      75      6    92%
source_google_ads/google_ads.py               68     12    82%
--------------------------------------------------------------
TOTAL                                        414     31    93%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1563    349    78%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:370: Backward compatibility tests are disabled for version 0.2.4.
================== 29 passed, 2 skipped in 1033.41s (0:17:13) ==================

grubberr · 2022-11-22T22:01:50Z

/publish connector=connectors/source-google-ads

🕑 Publishing the following connectors:
connectors/source-google-ads
https://github.com/airbytehq/airbyte/actions/runs/3527422091

Connector	Did it publish?	Were definitions generated?
connectors/source-google-ads	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

alafanechere · 2022-11-23T19:46:28Z

@grubberr I reverted this change on cloud after multiple normalization failures.

alafanechere · 2022-11-23T21:37:27Z

@grubberr could you please share why you disabled backward compatibility tests?
I think the original on-call issue only impacted a subset of users that received floats on the conversion metrics.
Changing the field type would solve the problem for this subset of users after a schema refresh, but the other users with successful syncs with integers on conversion metrics will fail.
I suggest you ask for multiple reviews when you want to bypass backward incompatibility tests so we'd have the opportunity for a deeper analysis of the potential side effects.

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr · 2022-11-27T12:40:13Z

@alafanechere I have answered here https://github.com/airbytehq/oncall/issues/947#issuecomment-1325767925
I have put link here for better navigation

* master: (74 commits) Fix support icon in sidebar fix: add BuildPulse report for helm ac tests (#19785) fix: yaml syntax (#19775) 🪟 🐛 Fix custom connection creation endpoint (#19702) Source facebook marketing: check "breakdowns" combinations (#19645) fix typo: notify instead of sync (#19737) find_non_rate_limited_PAT (#19736) Source Google Ads: fix schema for "campaigns" stream (#19700) 🎉 Source Asana: migrate to new SAT, added base HTTP errors handling (#19561) fix order not to randomly fail backward compatibility check (#19377) Bump helm chart version reference to 0.42.0 (#19706) fix: add extraEnv block (#19703) Bump Airbyte version from 0.40.21 to 0.40.22 (#19687) Bump helm chart version reference to 0.41.3 (#19685) Add connector builder server to airbyte proxy, kube overlays, and helm charts (#19554) dbt Cloud integration doc (#19619) 🪟 🎉 Display service token validation errors in the UI (#19578) 17644 Update Destination data type test to use the new data types (#19281) Docs: fix broken connector builder UI docs links (#19631) Bump Airbyte version from 0.40.20 to 0.40.21 (#19634) ...

fix schema for campaigns

faa33af

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr self-assigned this Nov 22, 2022

octavia-squidington-iv added area/connectors Connector related issues connectors/source/google-ads labels Nov 22, 2022

google-ads.md updated

ed42cd4

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

octavia-squidington-iv added the area/documentation Improvements or additions to documentation label Nov 22, 2022

grubberr added 2 commits November 22, 2022 19:41

Merge branch 'master' into grubberr/oncall-947-source-google-ads

aa7eb9e

0.1.45 -> 0.2.4

83dd42a

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

add backward_compatibility_tests_config for discovery

b0d26cb

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

upgrade acceptance-test-config.yml

96ae969

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

grubberr requested review from davydov-d, arsenlosenko, lazebnyi, bazarnov, darynaishchenko and artem1205 November 22, 2022 19:59

artem1205 approved these changes Nov 22, 2022

View reviewed changes

auto-bump connector version

74c8ef3

octavia-squidington-iii temporarily deployed to more-secrets November 22, 2022 22:40 Inactive

grubberr merged commit cc6facf into master Nov 22, 2022

grubberr deleted the grubberr/oncall-947-source-google-ads branch November 22, 2022 23:01

octavia-squidington-iii mentioned this pull request Nov 23, 2022

Bump Airbyte version from 0.40.22 to 0.40.23 #19762

Closed

SofiiaZaitseva pushed a commit that referenced this pull request Nov 24, 2022

Source Google Ads: fix schema for "campaigns" stream (#19700)

514a3ae

Signed-off-by: Sergey Chvalyuk <grubberr@gmail.com>

github-actions bot mentioned this pull request Nov 28, 2022

Bump helm chart version reference to 0.42.2 #19839

Closed

octavia-squidington-iii mentioned this pull request Nov 30, 2022

Bump Airbyte version from 0.40.22 to 0.40.23 #19941

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Google Ads: fix schema for "campaigns" stream #19700

Source Google Ads: fix schema for "campaigns" stream #19700

grubberr commented Nov 22, 2022 •

edited

Loading

grubberr commented Nov 22, 2022 •

edited by github-actions bot

Loading

grubberr commented Nov 22, 2022 •

edited by github-actions bot

Loading

grubberr commented Nov 22, 2022 •

edited by github-actions bot

Loading

grubberr commented Nov 22, 2022 •

edited by github-actions bot

Loading

grubberr commented Nov 22, 2022 •

edited by github-actions bot

Loading

grubberr commented Nov 22, 2022 •

edited by github-actions bot

Loading

alafanechere commented Nov 23, 2022

alafanechere commented Nov 23, 2022

grubberr commented Nov 27, 2022

Source Google Ads: fix schema for "campaigns" stream #19700

Source Google Ads: fix schema for "campaigns" stream #19700

Conversation

grubberr commented Nov 22, 2022 • edited Loading

What

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Tests

grubberr commented Nov 22, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented Nov 22, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented Nov 22, 2022 • edited by github-actions bot Loading

grubberr commented Nov 22, 2022 • edited by github-actions bot Loading

Build Failed

grubberr commented Nov 22, 2022 • edited by github-actions bot Loading

Build Passed

grubberr commented Nov 22, 2022 • edited by github-actions bot Loading

alafanechere commented Nov 23, 2022

alafanechere commented Nov 23, 2022

grubberr commented Nov 27, 2022

grubberr commented Nov 22, 2022 •

edited

Loading

grubberr commented Nov 22, 2022 •

edited by github-actions bot

Loading

grubberr commented Nov 22, 2022 •

edited by github-actions bot

Loading

grubberr commented Nov 22, 2022 •

edited by github-actions bot

Loading

grubberr commented Nov 22, 2022 •

edited by github-actions bot

Loading

grubberr commented Nov 22, 2022 •

edited by github-actions bot

Loading

grubberr commented Nov 22, 2022 •

edited by github-actions bot

Loading