Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DENG-1705 - Add startup_profile_selection_reason from first ping to clients_daily, clients_first_seen_v2 and downstream #4482

Merged
merged 5 commits into from
Nov 13, 2023
Merged

DENG-1705 - Add startup_profile_selection_reason from first ping to clients_daily, clients_first_seen_v2 and downstream #4482

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
721faff
DENG-1705 - Add startup_profile_selection_reason to clients_first_seen
ANich Oct 25, 2023
b5522c7
Add startup_profile_selection_reason_first_ping_only
ANich Nov 13, 2023
e26d8eb
Query typo
ANich Nov 13, 2023
6b0a3f6
Update test schema
ANich Nov 13, 2023
3cb4f48
Update sql/moz-fx-data-shared-prod/telemetry_derived/clients_first_se…
ANich Nov 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_joined_v1/schema.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2375,3 +2375,6 @@ fields:
- name: min_subsession_counter
type: INTEGER
mode: NULLABLE
- name: startup_profile_selection_first_ping_only
type: STRING
mode: NULLABLE
3 changes: 3 additions & 0 deletions sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_v6/query.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1453,6 +1453,9 @@ aggregates AS (
ARRAY_AGG(startup_profile_selection_reason ORDER BY submission_timestamp ASC)[
OFFSET(0)
] AS startup_profile_selection_reason_first,
mozfun.stats.mode_last(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 There should be very very few cases where we receive MULTIPLE pings with subsession_counter = 1 from a single client_id. But in those cases, if maybe one of them has the startup_profile_selection_reason we are looking for (while maybe 2+ others don't), we will miss it.

These would be weird clients that we possibly would want to ignore for other reasons, so I'm not advocating for any change here, just noting what the behavior will be for the record.

ARRAY_AGG(IF(subsession_counter = 1, startup_profile_selection_reason, NULL) ORDER BY submission_timestamp ASC)
) as startup_profile_selection_first_ping_only,
SUM(
scalar_parent_browser_ui_interaction_textrecognition_error
) AS scalar_parent_browser_ui_interaction_textrecognition_error_sum,
Expand Down
3 changes: 3 additions & 0 deletions sql/moz-fx-data-shared-prod/telemetry_derived/clients_daily_v6/schema.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2339,3 +2339,6 @@ fields:
- name: min_subsession_counter
type: INTEGER
mode: NULLABLE
- name: startup_profile_selection_first_ping_only
type: STRING
mode: NULLABLE
47 changes: 16 additions & 31 deletions sql/moz-fx-data-shared-prod/telemetry_derived/clients_first_seen_28_days_later_v1/query.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ WITH clients_first_seen_28_days_ago AS (
attribution_experiment,
attribution_dltoken,
attribution_dlsource,
-- startup_profile_selection_reason, when startup_profile_selection_reason is available
startup_profile_selection_reason,
first_seen_date,
FROM
telemetry_derived.clients_first_seen_v2
Expand All @@ -38,24 +38,20 @@ clients_first_seen_28_days_ago_with_days_seen AS (
AND cls.submission_date = @submission_date
)
SELECT
client_id,
sample_id,
first_seen_date,
@submission_date AS submission_date,
country_code,
channel,
build_id,
os,
os_version,
distribution_id,
attribution_source,
attribution_ua,
attribution_medium,
attribution_campaign,
attribution_content,
attribution_experiment,
attribution_dltoken,
attribution_dlsource,
* REPLACE (
COALESCE(
days_seen_bits,
mozfun.bits28.from_string('0000000000000000000000000000')
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you do it this way to make it clear that this is a zero-activity-days bit pattern? Over e.g. 0

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, for comprehension purposes

) AS days_seen_bits,
COALESCE(
days_visited_1_uri_bits,
mozfun.bits28.from_string('0000000000000000000000000000')
) AS days_visited_1_uri_bits,
COALESCE(
days_interacted_bits,
mozfun.bits28.from_string('0000000000000000000000000000')
) AS days_interacted_bits
),
COALESCE(
BIT_COUNT(mozfun.bits28.from_string('1111111000000000000000000000') & days_seen_bits) >= 5,
FALSE
Expand Down Expand Up @@ -84,17 +80,6 @@ SELECT
) > 0,
FALSE
) AS qualified_week4,
COALESCE(
days_seen_bits,
mozfun.bits28.from_string('0000000000000000000000000000')
) AS days_seen_bits,
COALESCE(
days_visited_1_uri_bits,
mozfun.bits28.from_string('0000000000000000000000000000')
) AS days_visited_1_uri_bits,
COALESCE(
days_interacted_bits,
mozfun.bits28.from_string('0000000000000000000000000000')
) AS days_interacted_bits,
@submission_date AS submission_date,
FROM
clients_first_seen_28_days_ago_with_days_seen
4 changes: 4 additions & 0 deletions ...moz-fx-data-shared-prod/telemetry_derived/clients_first_seen_28_days_later_v1/schema.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,3 +85,7 @@ fields:
- name: days_interacted_bits
type: INTEGER
mode: NULLABLE
- name: startup_profile_selection_reason
type: STRING
mode: NULLABLE
description: Client's first startup_profile_selection reason.
ANich marked this conversation as resolved.
Show resolved Hide resolved
3 changes: 3 additions & 0 deletions sql/moz-fx-data-shared-prod/telemetry_derived/clients_first_seen_v1/schema.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2341,3 +2341,6 @@ fields:
- name: min_subsession_counter
type: INTEGER
mode: NULLABLE
- name: startup_profile_selection_first_ping_only
type: STRING
mode: NULLABLE
6 changes: 3 additions & 3 deletions sql/moz-fx-data-shared-prod/telemetry_derived/clients_first_seen_v2/query.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -374,9 +374,9 @@ main_ping AS (
ARRAY_AGG(normalized_os_version RESPECT NULLS ORDER BY submission_date)[
SAFE_OFFSET(0)
] AS normalized_os_version,
CAST(
NULL AS STRING
) AS startup_profile_selection_reason, -- main_v5:payload.processes.parent.scalars.startup_profile_selection_reason
ARRAY_AGG(startup_profile_selection_reason_first_ping_only RESPECT NULLS ORDER BY submission_date)[
SAFE_OFFSET(0)
] AS startup_profile_selection_reason,
ANich marked this conversation as resolved.
Show resolved Hide resolved
ARRAY_AGG(attribution.dltoken RESPECT NULLS ORDER BY submission_date)[
SAFE_OFFSET(0)
] AS attribution_dltoken,
Expand Down
3 changes: 3 additions & 0 deletions sql/moz-fx-data-shared-prod/telemetry_derived/clients_last_seen_joined_v1/schema.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2110,6 +2110,9 @@ fields:
- name: startup_profile_selection_reason_first
type: STRING
mode: NULLABLE
- name: startup_profile_selection_reason_first_ping_only
type: STRING
mode: NULLABLE
- name: first_document_id
type: STRING
mode: NULLABLE
Expand Down
3 changes: 3 additions & 0 deletions sql/moz-fx-data-shared-prod/telemetry_derived/clients_last_seen_v1/schema.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2388,3 +2388,6 @@ fields:
- name: min_subsession_counter
type: INTEGER
mode: NULLABLE
- name: startup_profile_selection_first_ping_only
type: STRING
mode: NULLABLE
4 changes: 4 additions & 0 deletions ...ents_first_seen_v2/moz-fx-data-shared-prod.telemetry_derived.clients_daily_v6.schema.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -160,5 +160,9 @@
{
"type": "STRING",
"name": "normalized_os_version"
},
ANich marked this conversation as resolved.
Show resolved Hide resolved
{
"type": "STRING",
"name": "startup_profile_selection_reason_first_ping_only"
}
]