Google Analytics v4: Added support for segments and filters

[krisjan-oldekamp]

What

Also see this issue: #6305
Since I needed these options for a client project pretty soon, I updated the connector with the options described below (although someone else already claimed the issue in April).

Added the "segments" request parameter to custom report options

So we can fetch segmented data from the Google Analytics API.

There are two ways to add a segment:

1. Using the segment-ID (of prebuild segments in the GA interface) https://developers.google.com/analytics/devguides/reporting/core/v4/basics#segment_id or the (backwards compatible) segment syntax of the v3 reporing api: https://developers.google.com/analytics/devguides/reporting/core/v3/segments. Both can be inserted in the same request field.
2. Using dynamicSegment definition (https://developers.google.com/analytics/devguides/reporting/core/v4/samples#segments)

Since method 1 is way simpler and can achieve most use cases (and most analylist are familier with these segment definitions), I've chosen for option 1.

When using segments, the dimension ga:segments has to be added, because when adding more than 1 segment, data gets multiplied (e.g. for every segment configured). The ga:segments dimension will make sure you can differentiate on segment.

Example custom report definition

Using a segment-ID (3) of an existing segment in the GA interface (tip: update documentation with these examples):

[{"name": "sessions_segment", "dimensions": ["ga:date", "ga:segment"], "metrics": ["ga:pageviews","ga:sessions","ga:users","ga:transactions"], "segments": ["gaid::-3"]}]

Using the (backwards compatible) segment syntax of the V3 api (see https://developers.google.com/analytics/devguides/reporting/core/v3/reference#segment)

[{"name": "sessions_segment", "dimensions": ["ga:date", "ga:segment"], "metrics": ["ga:pageviews","ga:sessions","ga:users","ga:transactions"], "segments": ["sessions::condition::ga:browser==Chrome"]}]

Added the filter option to custom report options

Again two options available, the more complicated v4 filter syntax, or the simplified (most known) v3 format: https://developers.google.com/analytics/devguides/reporting/core/v3/reference#filters

Selected the v3 format.

Usage (tip: update documentation with these examples):

See filter expression list: https://developers.google.com/analytics/devguides/reporting/core/v3/reference#filters

[{"name": "sessions_filter", "dimensions": ["ga:date", "ga:deviceCategory"], "metrics": ["ga:pageviews","ga:sessions","ga:users","ga:transactions"], "filter": "ga:deviceCategory==desktop}]

Testing

I tested my connector using a separate integration test catalog:

python main.py read --config secrets/service_config.json --catalog integration_tests/configured_catalog_segment_filters.json

Where I used these configuration values for the segments / filters -> it will only return the data for browser = Chrome (segment) and only for 2022-09-01 (put a filter on the date dimension)

"custom_reports": "[{\"name\": \"sessions_segment\", \"dimensions\": [\"ga:date\", \"ga:segment\"], \"metrics\": [\"ga:pageviews\",\"ga:sessions\",\"ga:users\",\"ga:transactions\",\"ga:itemQuantity\"], \"segments\": [\"sessions::condition::ga:browser==Chrome\"], \"filter\": \"ga:date==20220901\"}]"

Recommended reading order

1. x.java
2. y.python

🚨 User Impact 🚨

No breaking changes, only extra options in the custom report fields.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

• Community member? Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • docs/integrations/README.md
  • airbyte-integrations/builds.md
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub by running the /publish command described here
• After the connector is published, connector added to connector index as described here
• Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

• Grant edit access to maintainers (instructions)
• Secrets in the connector's spec are annotated with airbyte_secret
• Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
• Code reviews completed
• Documentation updated
  • Connector's README.md
  • Connector's bootstrap.md. See description and examples
  • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
• PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

• Create a non-forked branch based on this PR and test the below items on it
• Build is successful
• If new credentials are required for use in CI, add them to GSM. Instructions.
• /test connector=connectors/<name> command is passing
• New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

• Issue acceptance criteria met
• PR name follows PR naming conventions
• If adding a new generator, add it to the list of scaffold modules being tested
• The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
• Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

[sajarin]

Thanks @krisjan-oldekamp for making this PR. This PR will be reviewed by a member of the community as part of Airbyte's Community Maintainer program. Someone will be assigned to review and help get your PR merged onto our main branch. Thanks for being patient!

[itaseskii]

@sajarin I can pick this one for reviewing since it was initially assigned to me :)

[sajarin]

@itaseskii, awesome thanks for picking this up!

[krisjan-oldekamp]

@itaseskii Updated the docs and bumped the version in the Dockerfile

[itaseskii]

@krisjan-oldekamp at first look this PR seems fine and inline with how the API works as explained in https://developers.google.com/analytics/devguides/reporting/core/v4/rest/v4/reports/batchGet

I have one question related to filtering. Is there any semantic or capability based difference between a "filtersExpression" property and a "dimensionFilterClauses"? can you filter using a "filtersExpression" on regex values, range values, case sensitivity as in the GA4 query explorer https://ga-dev-tools.web.app/ga4/query-explorer/?

[krisjan-oldekamp]

@itaseskii The tool you're referring to is related to Google Analytics 4. This connector is related to the old version of GA, Universal Analytics Core Reporting API version 4, so it's a bit confusing. So the query explorer tool for Google Universal Analytics is using the exact same methods as used in this connector (the v3 style filter expressions and dynamic segment definitions). You can use RegEx in these filters (which you could also use for ranges), but Regex in filtersExpression is case insensitive. However the v3 style definitions are pretty much industry standard for years (almost every other tool that uses the GA Core Reporting API is using the v3 style declarations).

[itaseskii]

@sajarin Can you please run the acceptance test suite? :)

[krisjan-oldekamp]

Screenshots:

Running with segment + filter (filter on date = 20220901 / succesfully only returns this data):

Running with segment

Last lines of running configured_catalog.yaml

[krisjan-oldekamp]

@sajarin / @itaseskii added a fix, because the connection test didn't pass, so good to go for the acceptance test.

[YowanR]

@sajarin could you take a look at this, please?

[sajarin]

/test connector=connectors/source-google-analytics-v4

🕑 connectors/source-google-analytics-v4 https://github.com/airbytehq/airbyte/actions/runs/3236418511
✅ connectors/source-google-analytics-v4 https://github.com/airbytehq/airbyte/actions/runs/3236418511
Python tests coverage:

Name                                     Stmts   Miss  Cover
------------------------------------------------------------
source_google_analytics_v4/__init__.py       2      0   100%
source_google_analytics_v4/source.py       290     26    91%
------------------------------------------------------------
TOTAL                                      292     26    91%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          10      4    60%   15-18
	 source_acceptance_test/config.py                        83      6    93%   78-80, 84-86
	 source_acceptance_test/conftest.py                     164    164     0%   6-282
	 source_acceptance_test/plugin.py                        48     48     0%   6-104
	 source_acceptance_test/tests/test_core.py              329    111    66%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 374-376, 379, 439-448, 477-478, 484, 487, 520-530, 543-568, 573-577
	 source_acceptance_test/tests/test_full_refresh.py       52      2    96%   34, 65
	 source_acceptance_test/tests/test_incremental.py       152     26    83%   21-23, 29-31, 36-43, 48-61, 239, 250-258
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     17    78%   15-16, 24-30, 47-54, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-67, 70-72, 75-77, 80-82, 85-87, 90-92, 95-113, 147-149
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1358    466    66%

Build Passed

Test summary info:

All Passed

[JonasBolin]

@krisjan-oldekamp huge thanks for taking on this one!

[YowanR]

@sajarin can you confirm that we can close this given this is merged: #6305?

[DaiZack]

I run into errors when using Google Analytics (Universal Analytics)

It seems the model is missing segments in the property:
airbyte-integrations/connectors/source-google-analytics-v4/source_google_analytics_v4/custom_reports_validator.py
row 19

I am just starting to use airbyte, don't know how to test from my side. Could you verify it?

[DaiZack]

I have just created a new connector with the line changed above, it works for me.

Also, it is annoying that the deleting source is missing from the GUI.

[ViniciusARZ]

@DaiZack do you have any recommendations on how to filter the data based on the column ga:isConversionEvent?
I tried multiple variations according to the documentation here, such as "filter": "ga:isConversionEvent==true", tried with segments too, using "segments": "ga:isConversionEvent==true" and also tried with conditions, such as "segments": ["eventCount::condition::ga:isConversionEvent==true"].

None of them worked. They pass through the testing on Airbyte, but the data returned comes unfiltered.

[JonasBolin]

@ViniciusARZ you might be confusing the Google Analytics Core Reporting API v4, with the Google Analytics 4 Data API.

I believe Google Analytics Core Reporting API v4 does not have the ga:isConversionEvent dimension, it is only present in the GA4 Data API.

[ViniciusARZ]

@JonasBolin It actually does work. The extraction works perfectly and the isConversionEvent dimension comes with normal data, as expected. The only thing I'm not being able to achieve is filtering by it, since I wanted it to return only records with true as the value in the dimension.

[DaiZack]

1. There is a bug in custom_reports_validator.py as I mentioned above. I tried to submit a merge request but tucked into CI/CD process, don't know how to make the merge, but it worked on my server well. Source Google Analytics: fix custom report with segments #22223
2. When using segmentation, you need also include the segmentent into the dimension, otherwise google will reject it.

[ViniciusARZ]

Do you have any recommendations on how to filter by this column? I also tried adding the segments dimension, and it still got the data unfiltered.