feat(DENG-7899): fx_accounts_active_daily_clients_v1

[kwindau]

Description

This PR creates the new table and associated view:

• moz-fx-data-shared-prod.telemetry_derived.fx_accounts_active_daily_clients_v1
• moz-fx-data-shared-prod.telemetry.fx_accounts_active_daily_clients

Related Tickets & Documents

• DENG-7899

Reviewer, please follow this checklist

[florezita]

Hi--

The only thing I am curious about is if we can a) in normalized_os_version differentiate between Windows 10 and Windows 11; or b) bring out the windows_build_number out of the nested field into its own column as well.

This is because for stakeholders, understanding OS and OS versions will be critical in understanding account profiles.

[kwindau]

Hi--

The only thing I am curious about is if we can a) in normalized_os_version differentiate between Windows 10 and Windows 11; or b) bring out the windows_build_number out of the nested field into its own column as well.

This is because for stakeholders, understanding OS and OS versions will be critical in understanding account profiles.

Good idea - I pulled windows_build_number out of the nested field into its own column. I also added a column to the view called "os_version_build" which basically helps distinguish Windows 10 vs 11 based on build #, and then for all other things, it's just the concatenation of normalized OS and normalized OS version. We can always adjust the view later on as needed in a subsequent PR to adjust this logic

[dataops-ci-bot]

Integration report for "Add column to view"

sql.diff

Click to expand!

Only in /tmp/workspace/generated-sql/dags/: bqetl_client_attributes.py
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_client_attributes.py /tmp/workspace/generated-sql/dags/bqetl_client_attributes.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_client_attributes.py	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_client_attributes.py	2025-02-21 18:15:55.000000000 +0000
@@ -0,0 +1,96 @@
+# Generated via https://github.com/mozilla/bigquery-etl/blob/main/bigquery_etl/query_scheduling/generate_airflow_dags.py
+
+from airflow import DAG
+from airflow.sensors.external_task import ExternalTaskMarker
+from airflow.sensors.external_task import ExternalTaskSensor
+from airflow.utils.task_group import TaskGroup
+import datetime
+from operators.gcp_container_operator import GKEPodOperator
+from utils.constants import ALLOWED_STATES, FAILED_STATES
+from utils.gcp import bigquery_etl_query, bigquery_dq_check, bigquery_bigeye_check
+
+docs = """
+### bqetl_client_attributes
+
+Built from bigquery-etl repo, [`dags/bqetl_client_attributes.py`](https://github.com/mozilla/bigquery-etl/blob/generated-sql/dags/bqetl_client_attributes.py)
+
+#### Description
+
+Schedules daily level retention queries
+#### Owner
+
+kwindau@mozilla.com
+
+#### Tags
+
+* impact/tier_2
+* repo/bigquery-etl
+"""
+
+
+default_args = {
+    "owner": "kwindau@mozilla.com",
+    "start_date": datetime.datetime(2025, 2, 20, 0, 0),
+    "end_date": None,
+    "email": ["telemetry-alerts@mozilla.com", "kwindau@mozilla.com"],
+    "depends_on_past": False,
+    "retry_delay": datetime.timedelta(seconds=1800),
+    "email_on_failure": True,
+    "email_on_retry": False,
+    "retries": 2,
+}
+
+tags = ["impact/tier_2", "repo/bigquery-etl"]
+
+with DAG(
+    "bqetl_client_attributes",
+    default_args=default_args,
+    schedule_interval="40 19 * * *",
+    doc_md=docs,
+    tags=tags,
+) as dag:
+
+    wait_for_copy_deduplicate_all = ExternalTaskSensor(
+        task_id="wait_for_copy_deduplicate_all",
+        external_dag_id="copy_deduplicate",
+        external_task_id="copy_deduplicate_all",
+        execution_delta=datetime.timedelta(seconds=67200),
+        check_existence=True,
+        mode="reschedule",
+        poke_interval=datetime.timedelta(minutes=5),
+        allowed_states=ALLOWED_STATES,
+        failed_states=FAILED_STATES,
+        pool="DATA_ENG_EXTERNALTASKSENSOR",
+    )
+
+    checks__fail_telemetry_derived__fx_accounts_active_daily_clients__v1 = bigquery_dq_check(
+        task_id="checks__fail_telemetry_derived__fx_accounts_active_daily_clients__v1",
+        source_table="fx_accounts_active_daily_clients_v1",
+        dataset_id="telemetry_derived",
+        project_id="moz-fx-data-shared-prod",
+        is_dq_check_fail=True,
+        owner="kwindau@mozilla.com",
+        email=["kwindau@mozilla.com", "telemetry-alerts@mozilla.com"],
+        depends_on_past=False,
+        parameters=["submission_date:DATE:{{ds}}"],
+        retries=0,
+    )
+
+    telemetry_derived__fx_accounts_active_daily_clients__v1 = bigquery_etl_query(
+        task_id="telemetry_derived__fx_accounts_active_daily_clients__v1",
+        destination_table="fx_accounts_active_daily_clients_v1",
+        dataset_id="telemetry_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="kwindau@mozilla.com",
+        email=["kwindau@mozilla.com", "telemetry-alerts@mozilla.com"],
+        date_partition_parameter="submission_date",
+        depends_on_past=False,
+    )
+
+    checks__fail_telemetry_derived__fx_accounts_active_daily_clients__v1.set_upstream(
+        telemetry_derived__fx_accounts_active_daily_clients__v1
+    )
+
+    telemetry_derived__fx_accounts_active_daily_clients__v1.set_upstream(
+        wait_for_copy_deduplicate_all
+    )
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry: fx_accounts_active_daily_clients
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived: fx_accounts_active_daily_clients_v1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/fx_accounts_active_daily_clients/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/fx_accounts_active_daily_clients/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/fx_accounts_active_daily_clients/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/fx_accounts_active_daily_clients/metadata.yaml	2025-02-21 18:10:36.000000000 +0000
@@ -0,0 +1,15 @@
+friendly_name: Fx Accounts Active Daily Clients
+description: |-
+  Please provide a description for the query
+owners: []
+labels: {}
+bigquery: null
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:dataops-managed/taar
+  - workgroup:mozilla-confidential
+references:
+  view.sql:
+  - moz-fx-data-shared-prod.telemetry_derived.fx_accounts_active_daily_clients_v1
+require_column_descriptions: false
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/fx_accounts_active_daily_clients/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/fx_accounts_active_daily_clients/view.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry/fx_accounts_active_daily_clients/view.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry/fx_accounts_active_daily_clients/view.sql	2025-02-21 18:07:42.000000000 +0000
@@ -0,0 +1,18 @@
+CREATE OR REPLACE VIEW
+  `moz-fx-data-shared-prod.telemetry.fx_accounts_active_daily_clients`
+AS
+SELECT
+  *,
+  CASE
+    WHEN normalized_os = 'Windows'
+      AND normalized_os_version = '10.0'
+      AND windows_build_number < 22000
+      THEN 'Windows 10'
+    WHEN normalized_os = 'Windows'
+      AND normalized_os_version = '10.0'
+      AND windows_build_number >= 22000
+      THEN 'Windows 11'
+    ELSE normalized_os || ' ' || normalized_os_version
+  END AS os_version_build
+FROM
+  `moz-fx-data-shared-prod.telemetry_derived.fx_accounts_active_daily_clients_v1`
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/checks.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/checks.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/checks.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/checks.sql	2025-02-21 18:07:42.000000000 +0000
@@ -0,0 +1,2 @@
+#fail
+{{ is_unique(["client_id"], "submission_date = @submission_date")}}
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/metadata.yaml	2025-02-21 18:10:57.000000000 +0000
@@ -0,0 +1,30 @@
+friendly_name: Firefox Accounts Active Daily Clients
+description: |-
+  Table with 1 row per submission date and Glean client ID sending a Firefox Accounts ping that date
+  Intentionally does not contain Firefox Account ID
+  Primary key is submission date & Glean client_info.client_id
+owners:
+- kwindau@mozilla.com
+labels:
+  incremental: true
+  owner1: kwindau
+  table_type: client_level
+  dag: bqetl_client_attributes
+scheduling:
+  dag_name: bqetl_client_attributes
+bigquery:
+  time_partitioning:
+    type: day
+    field: submission_date
+    require_partition_filter: true
+    expiration_days: 775.0
+  range_partitioning: null
+  clustering:
+    fields:
+    - normalized_channel
+workgroup_access:
+- role: roles/bigquery.dataViewer
+  members:
+  - workgroup:mozilla-confidential
+references: {}
+require_column_descriptions: true
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/query.sql	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/query.sql	2025-02-21 18:07:42.000000000 +0000
@@ -0,0 +1,24 @@
+SELECT
+  DATE(submission_timestamp) AS submission_date,
+  fxa.metrics.uuid.client_association_legacy_client_id AS legacy_telemetry_client_id,
+  fxa.client_info.*,
+  fxa.metadata,
+  fxa.metrics.labeled_counter,
+  fxa.metrics.string.glean_client_annotation_experimentation_id,
+  fxa.ping_info.experiments,
+  fxa.normalized_channel,
+  fxa.normalized_app_name,
+  fxa.app_version_major,
+  fxa.app_version_minor,
+  fxa.app_version_patch,
+  fxa.normalized_os,
+  fxa.normalized_os_version,
+  fxa.normalized_country_code,
+  fxa.sample_id
+FROM
+  `moz-fx-data-shared-prod.firefox_desktop.fx_accounts` fxa
+WHERE
+  DATE(submission_timestamp) = @submission_date
+  AND fxa.client_info.client_id IS NOT NULL
+QUALIFY
+  ROW_NUMBER() OVER (PARTITION BY fxa.client_info.client_id ORDER BY submission_timestamp DESC) = 1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/schema.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/telemetry_derived/fx_accounts_active_daily_clients_v1/schema.yaml	2025-02-21 18:07:42.000000000 +0000
@@ -0,0 +1,339 @@
+fields:
+- mode: NULLABLE
+  name: submission_date
+  type: DATE
+  description: The submission date of the Firefox Account ping.
+- name: legacy_telemetry_client_id
+  type: STRING
+  mode: NULLABLE
+  description: A UUID uniquely identifying the client, from the Firefox Account ping
+- name: android_sdk_version
+  type: STRING
+  mode: NULLABLE
+  description: The optional Android specific SDK version of the software running on this hardware device.
+- name: app_build
+  type: STRING
+  mode: NULLABLE
+  description: The build identifier generated by the CI system (e.g. "1234/A").
+- name: app_channel
+  type: STRING
+  mode: NULLABLE
+  description: The channel the application is being distributed on.
+- name: app_display_version
+  type: STRING
+  mode: NULLABLE
+  description: The user visible version string (e.g. "1.0.3").
+- name: architecture
+  type: STRING
+  mode: NULLABLE
+  description: The architecture of the device, (e.g. "arm", "x86").
+- name: build_date
+  type: STRING
+  mode: NULLABLE
+  description: The date & time the application was built
+- name: client_id
+  type: STRING
+  mode: NULLABLE
+  description: A UUID uniquely identifying the client; also called Glean client ID
+- name: device_manufacturer
+  type: STRING
+  mode: NULLABLE
+  description: The manufacturer of the device the application is running on.
+- name: device_model
+  type: STRING
+  mode: NULLABLE
+  description: The model of the device the application is running on.
+- name: first_run_date
+  type: STRING
+  mode: NULLABLE
+  description: The date of the first run of the application.
+- name: locale
+  type: STRING
+  mode: NULLABLE
+  description: The locale of the application during initialization (e.g. "es-ES").
+- name: os
+  type: STRING
+  mode: NULLABLE
+  description: Operating System
+- name: os_version
+  type: STRING
+  mode: NULLABLE
+  description: Operating System Version
+- name: session_count
+  type: INTEGER
+  mode: NULLABLE
+  description: Session Count
+- name: session_id
+  type: STRING
+  mode: NULLABLE
+  description: An optional UUID uniquely identifying the client's current session.
+- name: telemetry_sdk_build
+  type: STRING
+  mode: NULLABLE
+  description: Telemetry SDK Build
+- name: windows_build_number
+  type: INTEGER
+  mode: NULLABLE
+  description: The optional Windows build number, reported by Windows (e.g. 22000) and not set for other platforms
+- name: metadata
+  type: RECORD
+  mode: NULLABLE
+  description: Metadata
+  fields:
+  - name: geo
+    type: RECORD
+    mode: NULLABLE
+    description: Results of a geographic lookup based on the client's IP address
+    fields:
+    - name: city
+      type: STRING
+      mode: NULLABLE
+      description: City
+    - name: country
+      type: STRING
+      mode: NULLABLE
+      description: An ISO 3166-1 alpha-2 country code
+    - name: db_version
+      type: STRING
+      mode: NULLABLE
+      description: The specific geo database version used for this lookup
+    - name: subdivision1
+      type: STRING
+      mode: NULLABLE
+      description: First major country subdivision, typically a state, province, or county
+    - name: subdivision2
+      type: STRING
+      mode: NULLABLE
+      description: Second major country subdivision; not applicable for most countries
+  - name: header
+    type: RECORD
+    mode: NULLABLE
+    description: Headers included in the client's HTTP request
+    fields:
+    - name: date
+      type: STRING
+      mode: NULLABLE
+      description: Date HTTP header
+    - name: dnt
+      type: STRING
+      mode: NULLABLE
+      description: DNT (Do Not Track) Header
+    - name: x_debug_id
+      type: STRING
+      mode: NULLABLE
+      description: X-Debug-ID HTTP Header
+    - name: x_foxsec_ip_reputation
+      type: STRING
+      mode: NULLABLE
+      description: X-Foxsec-IP-Reputation header
+    - name: x_lb_tags
+      type: STRING
+      mode: NULLABLE
+      description: X-LB-Tags HTTP Header
+    - name: x_pingsender_version
+      type: STRING
+      mode: NULLABLE
+      description: X-PingSender-Version HTTP Header
+    - name: x_source_tags
+      type: STRING
+      mode: NULLABLE
+      description: X-Source-Tags HTTP header
+    - name: x_telemetry_agent
+      type: STRING
+      mode: NULLABLE
+      description: X-Telemetry-Agent HTTP header
+    - name: parsed_date
+      type: TIMESTAMP
+      mode: NULLABLE
+      description: Parsed Date
+    - name: parsed_x_source_tags
+      type: STRING
+      mode: REPEATED
+      description: Parsed X Source Tags
+    - name: parsed_x_lb_tags
+      type: RECORD
+      mode: NULLABLE
+      description: Parsed X LB Tags
+      fields:
+      - name: tls_version
+        type: STRING
+        mode: NULLABLE
+        description: TLS Version
+      - name: tls_cipher_hex
+        type: STRING
+        mode: NULLABLE
+        description: TLS Cipher Hex
+  - name: isp
+    type: RECORD
+    mode: NULLABLE
+    description: Results of ISP lookup based on the client's IP address
+    fields:
+    - name: db_version
+      type: STRING
+      mode: NULLABLE
+      description: The specific geo ISP database version used for this lookup
+    - name: name
+      type: STRING
+      mode: NULLABLE
+      description: The name of the ISP associated with the client's IP address
+    - name: organization
+      type: STRING
+      mode: NULLABLE
+      description: |-
+        The name of a specific business entity associated with the client's IP address when available;
+        otherwise the ISP name
+  - name: user_agent
+    type: RECORD
+    mode: NULLABLE
+    description: User Agent
+    fields:
+    - name: browser
+      type: STRING
+      mode: NULLABLE
+      description: Browser
+    - name: os
+      type: STRING
+      mode: NULLABLE
+      description: Operating System
+    - name: version
+      type: STRING
+      mode: NULLABLE
+      description: Version
+- name: labeled_counter
+  type: RECORD
+  mode: NULLABLE
+  description: Labeled Counter
+  fields:
+  - name: glean_error_invalid_label
+    type: RECORD
+    mode: REPEATED
+    description: |-
+      Counts the number of times a metric was set with an invalid label.
+      The labels are the `category.name` identifier of the metric.
+    fields:
+    - name: key
+      type: STRING
+      mode: NULLABLE
+      description: Key
+    - name: value
+      type: INTEGER
+      mode: NULLABLE
+      description: Value
+  - name: glean_error_invalid_overflow
+    type: RECORD
+    mode: REPEATED
+    description: |-
+      Counts the number of times a metric was set a value that overflowed.
+      The labels are the `category.name` identifier of the metric.
+    fields:
+    - name: key
+      type: STRING
+      mode: NULLABLE
+      description: Key
+    - name: value
+      type: INTEGER
+      mode: NULLABLE
+      description: Value
+  - name: glean_error_invalid_state
+    type: RECORD
+    mode: REPEATED
+    description: |-
+      Counts the number of times a timing metric was used incorrectly.
+      The labels are the `category.name` identifier of the metric.
+    fields:
+    - name: key
+      type: STRING
+      mode: NULLABLE
+      description: Key
+    - name: value
+      type: INTEGER
+      mode: NULLABLE
+      description: Value
+  - name: glean_error_invalid_value
+    type: RECORD
+    mode: REPEATED
+    description: |-
+      Counts the number of times a metric was set with an invalid label.
+      The labels are the `category.name` identifier of the metric.
+    fields:
+    - name: key
+      type: STRING
+      mode: NULLABLE
+      description: Key
+    - name: value
+      type: INTEGER
+      mode: NULLABLE
+      description: Value
+- name: glean_client_annotation_experimentation_id
+  type: STRING
+  mode: NULLABLE
+  description: |-
+    An experimentation identifier derived and provided by the application for the purpose of experimentation
+    enrollment.
+- name: experiments
+  type: RECORD
+  mode: REPEATED
+  description: Experiments
+  fields:
+  - name: key
+    type: STRING
+    mode: NULLABLE
+    description: Key
+  - name: value
+    type: RECORD
+    mode: NULLABLE
+    description: Value
+    fields:
+    - name: branch
+      type: STRING
+      mode: NULLABLE
+      description: Branch
+    - name: extra
+      type: RECORD
+      mode: NULLABLE
+      description: Extra
+      fields:
+      - name: enrollment_id
+        type: STRING
+        mode: NULLABLE
+        description: Enrollment ID
+      - name: type
+        type: STRING
+        mode: NULLABLE
+        description: Type
+- name: normalized_channel
+  type: STRING
+  mode: NULLABLE
+  description: Normalized Channel
+- name: normalized_app_name
+  type: STRING
+  mode: NULLABLE
+  description: Normalized App Name
+- name: app_version_major
+  type: NUMERIC
+  mode: NULLABLE
+  description: App Version - Major Version Number
+- name: app_version_minor
+  type: NUMERIC
+  mode: NULLABLE
+  description: App Version - Minor Version Number
+- name: app_version_patch
+  type: NUMERIC
+  mode: NULLABLE
+  description: App Version - Patch Version Number
+- name: normalized_os
+  type: STRING
+  mode: NULLABLE
+  description: Normalized Operating System
+- name: normalized_os_version
+  type: STRING
+  mode: NULLABLE
+  description: Normalized Operating System Version
+- name: normalized_country_code
+  type: STRING
+  mode: NULLABLE
+  description: Normalized Country Code
+- name: sample_id
+  type: INTEGER
+  mode: NULLABLE
+  description: Hashed version of client_id (if present) useful for partitioning; ranges from 0 to 99

Link to full diff