Support symlinks

[sean-rose]

Checklist for reviewer:

• Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title).
• If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up
  in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
• If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated.
• When adding a new derived dataset, ensure that data is not available already (fully or partially) and recommend extending an existing dataset in favor of creating new ones. Data can be available in the bigquery-etl repository, looker-hub or in looker-spoke-default.

For modifications to schemas in restricted namespaces (see CODEOWNERS):

• Follow the change control procedure

┆Issue is synchronized with this Jira Task

[sean-rose]

Path.glob() and Path.rglob() will be getting a new follow_symlinks argument in Python 3.13, so once we eventually upgrade to that we should be able to revert workarounds like this and specify follow_symlinks=True instead.

[dataops-ci-bot]

Integration report for "Merge branch 'main' into support-symlinks"

sql.diff

Click to expand!

Only in /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/hubs_derived/active_subscription_ids_live: schema.yaml
Only in /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/relay_derived/active_subscription_ids_live: schema.yaml
Only in /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest: schema.yaml
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/hubs_derived/active_subscription_ids_live/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/hubs_derived/active_subscription_ids_live/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/hubs_derived/active_subscription_ids_live/schema.yaml	2024-01-24 20:57:49.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/hubs_derived/active_subscription_ids_live/schema.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,7 +0,0 @@
-fields:
-- name: active_date
-  type: DATE
-  mode: NULLABLE
-- name: subscription_id
-  type: STRING
-  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_etl_scheduled_queries_cost_v1/query.py /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_etl_scheduled_queries_cost_v1/query.py
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_etl_scheduled_queries_cost_v1/query.py	2024-01-24 20:57:49.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_etl_scheduled_queries_cost_v1/query.py	2024-01-24 20:45:33.000000000 +0000
@@ -4,6 +4,7 @@
 
 from argparse import ArgumentParser
 from fnmatch import fnmatchcase
+from glob import glob
 from pathlib import Path
 
 from google.cloud import bigquery
@@ -43,9 +44,13 @@
     args = parser.parse_args()
     client = bigquery.Client(args.project)
 
-    sql_queries = list(Path(args.sql_dir).rglob("query.sql"))
-    python_queries = list(Path(args.sql_dir).rglob("query.py"))
-    multipart_queries = list(Path(args.sql_dir).rglob("part1.sql"))
+    sql_queries = list(map(Path, glob(f"{args.sql_dir}/**/query.sql", recursive=True)))
+    python_queries = list(
+        map(Path, glob(f"{args.sql_dir}/**/query.py", recursive=True))
+    )
+    multipart_queries = list(
+        map(Path, glob(f"{args.sql_dir}/**/part1.sql", recursive=True))
+    )
     query_paths = sql_queries + python_queries + multipart_queries
 
     query = create_query(query_paths, args.date)
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_etl_scheduled_query_usage_v1/query.py /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_etl_scheduled_query_usage_v1/query.py
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_etl_scheduled_query_usage_v1/query.py	2024-01-24 20:57:49.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring_derived/bigquery_etl_scheduled_query_usage_v1/query.py	2024-01-24 20:45:33.000000000 +0000
@@ -3,6 +3,7 @@
 """Determine cost of previously scheduled bigquery-etl queries."""
 
 from argparse import ArgumentParser
+from glob import glob
 from pathlib import Path
 
 from google.cloud import bigquery
@@ -57,9 +58,13 @@
 def main():
     args = parser.parse_args()
 
-    sql_queries = list(Path(args.sql_dir).rglob("query.sql"))
-    python_queries = list(Path(args.sql_dir).rglob("query.py"))
-    multipart_queries = list(Path(args.sql_dir).rglob("part1.sql"))
+    sql_queries = list(map(Path, glob(f"{args.sql_dir}/**/query.sql", recursive=True)))
+    python_queries = list(
+        map(Path, glob(f"{args.sql_dir}/**/query.py", recursive=True))
+    )
+    multipart_queries = list(
+        map(Path, glob(f"{args.sql_dir}/**/part1.sql", recursive=True))
+    )
     query_paths = sql_queries + python_queries + multipart_queries
     partition = args.date.replace("-", "")
     destination_table = f"{args.project}.{args.destination_dataset}.{args.destination_table}${partition}"
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/relay_derived/active_subscription_ids_live/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/relay_derived/active_subscription_ids_live/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/relay_derived/active_subscription_ids_live/schema.yaml	2024-01-24 20:57:49.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/relay_derived/active_subscription_ids_live/schema.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,7 +0,0 @@
-fields:
-- name: active_date
-  type: DATE
-  mode: NULLABLE
-- name: subscription_id
-  type: STRING
-  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest/schema.yaml	2024-01-24 20:57:49.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/releases_latest/schema.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,39 +0,0 @@
-fields:
-- name: date
-  type: DATE
-  mode: NULLABLE
-  description: null
-- name: product
-  type: STRING
-  mode: NULLABLE
-  description: null
-- name: category
-  type: STRING
-  mode: NULLABLE
-  description: null
-- name: channel
-  type: STRING
-  mode: NULLABLE
-- name: build_number
-  type: INTEGER
-  mode: NULLABLE
-  description: null
-- name: release_date
-  type: DATE
-  mode: NULLABLE
-- name: version
-  type: STRING
-  mode: NULLABLE
-  description: null
-- name: major_version
-  type: NUMERIC
-  mode: NULLABLE
-- name: minor_version
-  type: NUMERIC
-  mode: NULLABLE
-- name: patch_version
-  type: NUMERIC
-  mode: NULLABLE
-- name: beta_version
-  type: NUMERIC
-  mode: NULLABLE

Link to full diff