adding support for gaarf workflow installation

gaarf-wf/README.md

@@ -0,0 +1,149 @@
+# Ads Policy Monitor
+
+_Providing a 1-stop-shop, centralized view of all your Google Ads policy
+reports._
+
+Ads Policy Monitor provides a data pipeline for pulling ad and asset approval
+data from Google Ads, enabling you to monitor trends and spikes with policy
+approvals.
+
+## Solution Overview
+
+To keep ads safe and appropriate for everyone, ads are reviewed to make sure
+they comply with Google Ads policies,
+[read more here](https://support.google.com/google-ads/answer/1722120?sjid=13030684844768437853-EU).
+
+For large advertisers and agencies, it can be challenging to monitor the state
+of your ad copy across all of your Google Ads accounts. This can be amplified
+when automation is used to create ad copy, as it's easy to introduce mistakes
+that result in ads being disapproved, for example, the incorrect use of
+capitalisation.
+
+This solution pulls a daily snapshot of your ad and asset policy approvals,
+stores them in BigQuery, and provides a Looker Studio dashboard template to
+monitor this.
+
+When disapprovals happen you can take action by following the deeplink into
+Google Ads to get more information and take action.
+
+In the dashboard you can instantly monitor the overall status of your accounts:
+![Dashboard overview](./docs/images/looker-studio-overview.png)
+
+And how this changes over time. This allows you to identify any changes in the
+trend, to investigate further.
+![Dashboard time series](./docs/images/looker-studio-time-series.png)
+
+You can drilldown into the disapprovals to try to get more information about why
+this has happened:
+![Dashboard disapprovals](./docs/images/looker-studio-disapprovals.png)
+
+You're also able to see insights like when an ad group has no approved ads:
+![Dashboard no approved ads](./docs/images/looker-studio-no-approved-ads.png)
+
+And analyze asset disapprovals:
+![Dashboard assets](./docs/images/looker-studio-assets.png)
+
+## How does it work?
+
+![Architecture diagram](./docs/images/architecture-diagram.png)
+
+1.  A daily cron job runs that makes a HTTPS request to trigger the
+    `ads_policy_monitor` Cloud Function.
+2.  The Cloud Function uses
+    [Google Ads Query Language (GAQL)](https://developers.google.com/google-ads/api/docs/query/overview)
+    to run a number of reports in Google Ads via the API.
+3.  The output of these reports is written to a dataset in BigQuery called
+    `ads_policy_monitor`.
+4.  We provide a template Looker Studio dashboard to visualise this data.
+    However, there is nothing to stop you using your own dashboard solution. For
+    example, it is possible to connect this data with the Looker Platform for
+    more advanced BI reporting, alerting, data drilldown etc.
+
+## Requirements
+
+-   A Google Cloud Project
+-   Access to the Google Ads API (e.g. developer token, account access),
+    [see getting started](https://developers.google.com/google-ads/api/docs/get-started/introduction).
+-   Access to Looker Studio or a dashboard solution that can connect to
+    BigQuery.
+    -   If you wish to have access to the
+        [Looker Studio template provided by Google](https://lookerstudio.google.com/c/u/0/reporting/13995d1f-741c-40f0-934c-9517e2ffc361/),
+        please join this group
+        [ads-policy-monitor-template-readers](https://groups.google.com/g/ads-policy-monitor-template-readers).
+
+## Deployment
+
+To install it, click on the blue Deploy button and follow the instructions in Google Cloud:
+
+[![Click to deploy ads-policy-monitor](https://encrypted-tbn0.gstatic.com/images?q=tbn:ANd9GcSCDIyJjIDWlJHd_x6RAaKczT5_9yc_IC3voZoSUgPwZ9Qn2gQRI3-e_Ra9UR2zEgMVMBM&usqp=CAU)](https://console.cloud.google.com/?cloudshell=true&cloudshell_git_repo=https://github.com/google-marketing-solutions/ads-policy-monitor&cloudshell_tutorial=walkthrough.md)
+
+## FAQ
+
+### Which Google Cloud APIs will be enabled to my project?
+
+- Big Query API `bigquery.googleapis.com`
+- Cloud Build `cloudbuild.googleapis.com`
+- Cloud Funtions `cloudfunctions.googleapis.com`
+- Cloud Resource Manager API `cloudresourcemanager.googleapis.com`
+- Google Ads API `googleads.googleapis.com`
+- Identity and Access Management (IAM) `iam.googleapis.com`
+- Secret Manager `secretmanager.googleapis.com`
+- Cloud Run `run.googleapis.com`
+- Cloud Scheduler `cloudscheduler.googleapis.com`
+
+You can also check this on the deployment file [ads-policy-monitor/init.sh](https://github.com/google-marketing-solutions/ads-policy-monitor/blob/main/init.sh#L78-L87).
+
+### Which GAQL queries are executed?
+Please refer to the folder google_ads_queries [cloud_functions/ads_policy_monitor/gaql](https://github.com/google-marketing-solutions/ads-policy-monitor/tree/main/cloud_functions/ads_policy_monitor/gaql).
+
+
+### Can I deploy it in an existing Cloud Project or do I need to create a new one just for this solution?
+You can use an existing Project if you want to. However, please remember that the best practice for clients is to create a new project dedicated to this solution (or any new solution).
+
+### Can this solution refresh the data more often than once a day?
+You can make the Cloud Scheduler run more often than once a day, however you'd need to also make some customization in the code as the code and templates provided here would show duplicated records.
+
+The reason this solution was designed to run once a day was based on learnings from hundreds of deployments, where the tradeoff between solution costs and Policy Reviews turn-around time was taken into consideration.
+
+If youy still wish to make the refresh rate higher, these are some of the adjustments you'd have to perform:
+1. Introduce time into the [`event_date`](https://github.com/google-marketing-solutions/ads-policy-monitor/blob/main/cloud_functions/ads_policy_monitor/gaql/ad_policy_data.sql#L15) parameter of the queries.
+2. Update the [cron schedule](https://github.com/google-marketing-solutions/ads-policy-monitor/blob/main/terraform/main.tf#L264) to run more frequently.
+3. Update the [partitioning on the BigQuery table](https://github.com/google-marketing-solutions/ads-policy-monitor/blob/main/terraform/main.tf#L48) to hourly.
+4. Update the [latestadpolicy data query](https://github.com/google-marketing-solutions/ads-policy-monitor/blob/main/bigquery/views/latest_ad_policy_data.sql#L37) to filter on only the latest data.
+5. Refresh your Looker Studio dashboard data source to mke sure the `event_date` is now a datetime and instead of just date.
+
+
+## Contributing
+
+### Code formatting
+
+The code is formatted using [yapf](https://github.com/google/yapf). After making
+a change, before submitting the code please do the following:
+
+Install the dev requirements:
+```
+pip install -r requirements_dev.txt
+```
+
+Then run yapf:
+
+```
+yapf --style google -r -i .
+```
+
+## Disclaimers
+__This is not an officially supported Google product.__
+
+Copyright 2024 Google LLC. This solution, including any related sample code or
+data, is made available on an “as is,” “as available,” and “with all faults”
+basis, solely for illustrative purposes, and without warranty or representation
+of any kind. This solution is experimental, unsupported and provided solely for
+your convenience. Your use of it is subject to your agreements with Google, as
+applicable, and may constitute a beta feature as defined under those agreements.
+To the extent that you make any data available to Google in connection with your
+use of the solution, you represent and warrant that you have all necessary and
+appropriate rights, consents and permissions to permit Google to use and process
+that data. By using any portion of this solution, you acknowledge, assume and
+accept all risks, known and unknown, associated with its usage, including with
+respect to your deployment of any portion of this solution in your systems, or
+usage in connection with your business, if at all.

gaarf-wf/answers.json

@@ -0,0 +1,38 @@
+{
+  "use_current_project": true,
+  "name": "policy",
+  "gcs_bucket": "",
+  "service_account": "",
+  "path_to_ads_queries": "google_ads_queries",
+  "path_to_bq_queries": "bq_queries",
+  "custom_ids_query_path": "get-accounts.sql",
+  "path_to_googleads_config": "google-ads.yaml",
+  "cf_memory": "2048MB",
+  "deploy_scripts": true,
+  "deploy_wf": true,
+  "output_dataset": "policy_ads",
+  "ads_macro": {
+    "start_date": ":YYYYMMDD-90",
+    "end_date": ":YYYYMMDD-1",
+    "today": ":YYYYMMDD",
+    "yesterday": ":YYYYMMDD-1"
+  },
+  "bq_macro": {
+    "bq_dataset": "policy_ads"
+  },
+  "schedule_wf": true,
+  "schedule_time": "09:00",
+  "run_job": false,
+  "run_wf": true,
+  "clone_dashboard": true,
+  "dashboard_id": "ef1f136c-e008-403a-83db-1f0fd5a152bf",
+  "dashboard_name": "Ads Policy Monitor",
+  "dashboard_dataset": "policy_ads_bq",
+  "dashboard_datasources": {
+    "LatestAssetPolicyData": "latest_asset_policy_data",
+    "LatestAdPolicyData": "latest_ad_policy_data",
+    "AdPolicyDataTimeSeries": "ad_snapshots_*",
+    "NoApprovedAdsAdGroup": "no_approved_ads_ad_group",
+    "AssetPolicyDataTimeSeries": "asset_snapshots_*"
+  }
+}

gaarf-wf/bq_queries/01-latest_ad_policy_data.sql

@@ -0,0 +1,37 @@
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+/**
+ * Pull the latest Ad Policy Data combined with deep links to Google Ads.
+ */
+
+CREATE SCHEMA IF NOT EXISTS `{bq_dataset}_bq`;
+CREATE OR REPLACE TABLE `{bq_dataset}_bq.latest_ad_policy_data` AS
+SELECT
+  Ocid.ocid,
+  STRUCT(
+    CONCAT("https://ads.google.com/aw/overview?ocid=", Ocid.ocid) AS home,
+    CONCAT(
+    "https://ads.google.com/aw/ads?campaignId=", AdPolicyData.campaign_id,
+    "&adGroupId=", AdPolicyData.ad_group_id,
+    "&ocid=", Ocid.ocid) AS ads
+   ) AS gads_links,
+  AdPolicyData.*,
+  SPLIT(ARRAY_TO_STRING(AdPolicyData.ad_group_ad_policy_summary_policy_topic_entries, "|"),"|") AS ad_policy_topics,
+FROM
+  `{bq_dataset}.ad_policy_data` AS AdPolicyData
+LEFT JOIN
+  (
+    SELECT DISTINCT customer_id, ocid FROM `{bq_dataset}.ocid_mapping`
+  ) AS Ocid ON Ocid.customer_id = AdPolicyData.customer_id
+WHERE Date(event_date) = CURRENT_DATE()-1

gaarf-wf/bq_queries/02-latest_asset_policy_data.sql

@@ -0,0 +1,125 @@
+# Copyright 2024 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+/**
+ * Pull the latest Asset Policy Data combined with deep links to Google Ads.
+ */
+
+CREATE SCHEMA IF NOT EXISTS `{bq_dataset}_bq`;
+CREATE OR REPLACE TABLE `{bq_dataset}_bq.latest_asset_policy_data` AS
+SELECT
+  CURRENT_DATE()-1 AS event_date,
+  Ocid.ocid,
+  STRUCT(
+    CONCAT("https://ads.google.com/aw/overview?ocid=", Ocid.ocid) AS home,
+    CONCAT("https://ads.google.com/aw/assetreport/associations/allupgraded",
+    "?ocid=", Ocid.ocid,
+    IF(AssetPolicyData.campaign_id is Null, "", "&campaignId=" || AssetPolicyData.campaign_id),
+    IF(AssetPolicyData.ad_group_id is Null, "", "&adGroupId=" || AssetPolicyData.ad_group_id)
+    ) AS assets
+   ) AS gads_links,
+  AssetPolicyData.customer_id,
+  AssetPolicyData.customer_descriptive_name,
+  AssetPolicyData.asset_id,
+  AssetPolicyData.asset_type,
+  AssetPolicyData.asset_source,
+  "Ad Group" AS asset_level,
+  AssetPolicyData.asset_policy_summary_approval_status AS asset_policy_summary_approval_status,
+  AssetPolicyData.asset_policy_summary_review_status AS asset_policy_summary_review_status,
+  SPLIT(ARRAY_TO_STRING(AssetPolicyData.asset_policy_summary_policy_topic_entries_topics, "|"),"|") AS asset_policy_summary_policy_topic_entries_topics
+FROM
+  `{bq_dataset}.ad_group_asset` AS AssetPolicyData
+LEFT JOIN
+  (
+    SELECT DISTINCT customer_id, ocid FROM `{bq_dataset}.ocid_mapping`
+  ) AS Ocid ON Ocid.customer_id = AssetPolicyData.customer_id
+WHERE Date(event_date) = CURRENT_DATE()-1
+UNION ALL
+SELECT
+  CURRENT_DATE()-1 AS event_date,
+  Ocid.ocid,
+  STRUCT(
+    CONCAT("https://ads.google.com/aw/overview?ocid=", Ocid.ocid) AS home,
+    CONCAT("https://ads.google.com/aw/assetreport/associations/allupgraded",
+    "?ocid=", Ocid.ocid,
+    IF(CampaignAssetPolicyData.campaign_id is Null, "", "&campaignId=" || CampaignAssetPolicyData.campaign_id)
+    ) AS assets
+   ) AS gads_links,
+  CampaignAssetPolicyData.customer_id,
+  CampaignAssetPolicyData.customer_descriptive_name,
+  CampaignAssetPolicyData.asset_id,
+  CampaignAssetPolicyData.asset_type,
+  CampaignAssetPolicyData.asset_source,
+  "Campaign" AS asset_level,
+  CampaignAssetPolicyData.asset_policy_summary_approval_status AS asset_policy_summary_approval_status,
+  CampaignAssetPolicyData.asset_policy_summary_review_status AS asset_policy_summary_review_status,
+  SPLIT(ARRAY_TO_STRING(CampaignAssetPolicyData.asset_policy_summary_policy_topic_entries_topics, "|"),"|") AS asset_policy_summary_policy_topic_entries_topics
+FROM
+  `{bq_dataset}.campaign_asset` AS CampaignAssetPolicyData
+LEFT JOIN
+  (
+    SELECT DISTINCT customer_id, ocid FROM `{bq_dataset}.ocid_mapping`
+  ) AS Ocid ON Ocid.customer_id = CampaignAssetPolicyData.customer_id
+WHERE Date(event_date) = CURRENT_DATE()-1
+UNION ALL
+SELECT
+  CURRENT_DATE()-1 AS event_date,
+  Ocid.ocid,
+  STRUCT(
+    CONCAT("https://ads.google.com/aw/overview?ocid=", Ocid.ocid) AS home,
+    CONCAT("https://ads.google.com/aw/assetreport/associations/allupgraded",
+    "?ocid=", Ocid.ocid
+    ) AS assets
+   ) AS gads_links,
+  CustomerAssetPolicyData.customer_id,
+  CustomerAssetPolicyData.customer_descriptive_name,
+  CustomerAssetPolicyData.asset_id,
+  CustomerAssetPolicyData.asset_type,
+  CustomerAssetPolicyData.asset_source,
+  "Account" AS asset_level,
+  CustomerAssetPolicyData.asset_policy_summary_approval_status AS asset_policy_summary_approval_status,
+  CustomerAssetPolicyData.asset_policy_summary_review_status AS asset_policy_summary_review_status,
+  SPLIT(ARRAY_TO_STRING(CustomerAssetPolicyData.asset_policy_summary_policy_topic_entries_topics, "|"),"|") AS asset_policy_summary_policy_topic_entries_topics
+FROM
+  `{bq_dataset}.customer_asset` AS CustomerAssetPolicyData
+LEFT JOIN
+  (
+    SELECT DISTINCT customer_id, ocid FROM `{bq_dataset}.ocid_mapping`
+  ) AS Ocid ON Ocid.customer_id = CustomerAssetPolicyData.customer_id
+WHERE Date(event_date) = CURRENT_DATE()-1
+UNION ALL
+SELECT
+  CURRENT_DATE()-1 AS event_date,
+  Ocid.ocid,
+  STRUCT(
+    CONCAT("https://ads.google.com/aw/overview?ocid=", Ocid.ocid) AS home,
+    CONCAT("https://ads.google.com/aw/assetreport/associations/allupgraded",
+    "?ocid=", Ocid.ocid,
+    IF(AssetGroupAssetPolicyData.asset_group_id is Null, "", "&asset_group_id=" || AssetGroupAssetPolicyData.asset_group_id)
+    ) AS assets
+   ) AS gads_links,
+  AssetGroupAssetPolicyData.customer_id,
+  AssetGroupAssetPolicyData.customer_descriptive_name,
+  AssetGroupAssetPolicyData.asset_id,
+  AssetGroupAssetPolicyData.asset_type,
+  AssetGroupAssetPolicyData.source AS asset_source,
+  "Asset Group" AS asset_level,
+  AssetGroupAssetPolicyData.asset_policy_summary_approval_status AS asset_policy_summary_approval_status,
+  AssetGroupAssetPolicyData.asset_policy_summary_review_status AS asset_policy_summary_review_status,
+  SPLIT(ARRAY_TO_STRING(AssetGroupAssetPolicyData.asset_policy_summary_policy_topic_entries_topics, "|"),"|") AS asset_policy_summary_policy_topic_entries_topics
+FROM
+  `{bq_dataset}.asset_group_asset` AS AssetGroupAssetPolicyData
+LEFT JOIN
+  (
+    SELECT DISTINCT customer_id, ocid FROM `{bq_dataset}.ocid_mapping`
+  ) AS Ocid ON Ocid.customer_id = AssetGroupAssetPolicyData.customer_id

gaarf-wf/bq_queries/03-ad_snapshot.sql

@@ -0,0 +1,22 @@
+# Copyright 2022 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+CREATE SCHEMA IF NOT EXISTS `{bq_dataset}_bq`;
+CREATE OR REPLACE TABLE `{bq_dataset}_bq.ad_snapshots_${format(today(),'yyyyMMdd')}` AS
+SELECT
+    DATE(event_date) AS event_date,
+    ad_group_ad_policy_summary_approval_status,
+    COUNT(DISTINCT ad_group_ad_ad_id) AS ad_count
+FROM `{bq_dataset}.ad_policy_data`
+GROUP BY 1,2